3195
Of your peers have already watched this video.
20:30 Minutes
The most insightful time you'll spend today!
Delivering analysis-ready financial data at scale
Refinitiv, is a leading provider of financial market data. It serves over 40,000 institutions and operates in 190 countries.
“We’re constantly looking at solving some of the most difficult problems our users face when accessing new data sets as well as onboarding new sources that will be relevant for workflows across the front, middle, and back office,” says Catalina Vazquez, Proposition Director, Tick History, Refinitiv.
For every dollar spent on financial market data, a further eight are spent in processing, storing, and transforming that data before it can be analyzed. That’s a problem Refinitiv wanted to get at.
Refinitiv partnered with Google Cloud to deliver instant access to 25 years of financial markets data. What was formerly delivered by hard disk then managed by hand is now provided automatically with BigQuery.
Watch Tom Salmon, Customer Engineer, Google Cloud and Catalina Vazquez and learn how Refinitiv addressed key challenges by designing their architecture with Google Cloud’s simplicity, security, scale, and speed in mind.
Uniting Data and Analytics in One Place: Apache Iceberg and BigLake
1139
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
When your data is siloed data across lakes and warehouses, it can be hard to transform outcomes with your data. Apache Iceberg is an open table format that provides data management capabilities for data hosted on object stores and enables organizations to run analytics and AI use cases over a single copy of data. A growing community of data engineers, customers, and industry partners are contributing, integrating, and deploying Iceberg, making it the standard for organizations building open-format lakehouses.
To help customers on this journey, we announced support for Iceberg through BigLake in October, 2022. Since its preview, many customers have started building lakehouse workloads using Apache Iceberg as their data management layer, and this support is now generally available.
Unify analytics, streaming and AI use cases over a single copy of data
You can use open-source engines to process and ingest data into Iceberg tables, and BigQuery can query those tables. Since the preview, customers have also used Spark, Trino and Flink to process Iceberg tables and make those tables available to their BigQuery users. Then, BigLake Metastore provides shared metadata for Iceberg tables across BigQuery and open-source engines, eliminating the need to maintain multiple table definitions. Further, you can provide BigQuery datasets and table properties when creating new Iceberg tables in Spark, and those tables become automatically available for the BigQuery user to query.
When implementing Iceberg lakehouse workloads, query performance is a top priority for data warehouse users. BigQuery natively integrates with the Iceberg transaction logs and leverages its rich metadata for efficient query planning. Query plans are designed to reduce BigQuery compute consumption by lowering the amount of scanned data, by optimizing joins, and by improving data-plane parallelism. The net result is that you get better query performance and lower slot usage when querying BigLake Iceberg tables.
This GA release also adds support to provide automatic synchronization of table schema in BigQuery when the table is modified through an open-source engine.
Engine-agnostic, industry-leading security and governance built-in
Customers have been telling us that building Iceberg lakehouses in a secure and governed manner is a top priority. BigLake support for Iceberg provides fine-grained access control, including row- and column-level security as well as data masking to simplify this. These features are designed to work independently of the query engine. During the preview, we expanded BigQuery to also support differential privacy for all tables including Iceberg.
You can also define security policies on a BigLake Iceberg table using BigQuery. Security policies are then enforced regardless of the query engine used — BigQuery natively enforces these policies at runtime, and open-source engines can securely access the data using the BigQuery Storage API. The BigQuery Storage API enforces the security policies at the data-plane layer, and is offered via pre-built connectors for Spark, Trino, Presto and TensorFlow. You can also use client libraries to build connectors for custom applications.
New use cases with multi-cloud Iceberg lakehouse
The open nature of Apache Iceberg lets you build multi-cloud lakehouses with uniform management of data. With this launch, you can now create BigLake Iceberg tables on Amazon S3, and query them using BigQuery Omni. BigLake’s performance and fine-grained access control features seamlessly extend to multi-cloud Iceberg tables, so you can securely perform cross-cloud analytics with BigQuery. We’ll extend similar support to Azure data lake Gen 2 in the coming weeks.
Apache Iceberg’s format uniformity across clouds also enables new data sharing use cases to help you share data with your customers, partners and suppliers, regardless of which cloud they are using. BigLake Iceberg tables on Cloud Storage or on Amazon S3 can be shared through Analytics Hub and consumed via BigQuery or OSS engines via BigQuery Storage Read API, providing an open sharing standard and the flexibility to use multiple query engines. A notable example of this is the recently announced Salesforce Data Cloud data sharing use case, which enables bi-directional cross-cloud data sharing between Salesforce and BigQuery and which is powered by Iceberg.
Getting started
To create and query your first Iceberg table, following the documentation. or run a quick proof of concept on our analytics lakehouse with the Iceberg jump start solution.
Anthos for Manufacturing: Tackle DevOps Complexities and Drive Digital Transformation
DOWNLOAD E-BOOK6243
Of your peers have already downloaded this article
2:00 Minutes
The most insightful time you'll spend today!
5427
Of your peers have already downloaded this article
14:30 Minutes
The most insightful time you'll spend today!
To make great products: do machine learning like the great engineer you are, not like the great machine learning expert you aren’t.
Most of the problems you will face are, in fact, engineering problems. Even with all the resources of a great machine learning expert, most of the gains come from great features, not great machine learning algorithms. So, the basic approach is:
- Make sure your pipeline is solid end to end.
- Start with a reasonable objective.
- Add common-sense features in a simple way.
- Make sure that your pipeline stays solid.
This approach will work well for a long period of time. Diverge from this approach only when there are no more simple tricks to get you any farther. Adding complexity slows future releases.
Once you’ve exhausted the simple tricks, cutting-edge machine learning might indeed be in your future. See the section on Phase III machine learning projects.
This document is arranged as follows:
- The first part should help you understand whether the time is right for building a machine learning system.
- The second part is about deploying your first pipeline.
- The third part is about launching and iterating while adding new features to your pipeline, how to evaluate models and training-serving skew.
- The final part is about what to do when you reach a plateau.
- Afterwards, there is a list of related work and an appendix with some background on the systems commonly used as examples in this document.
5672
Of your peers have already watched this video.
22:00 Minutes
The most insightful time you'll spend today!
Chronicle Security Analytics: Key to Address Security Data Overload
Google Cloud’s Chronicle is a security analytics platform built for modern use cases to combat modern threats. In today’s world, enterprises have undergone significant changes in all aspects, and must adapt to their security needs to counter threats and attacks. Watch the video to learn Google Cloud’s initiative to help businesses improve their security by 10x to catch up with the way the world is changing and how Chronicle Security Analytics is poised to help address data security challenges.
5356
Of your peers have already watched this video.
44:30 Minutes
The most insightful time you'll spend today!
How Go-Jek, Indonesia’s First Billion-dollar Startup, Improved the Productivity of Data Scientists
Go-Jek, Indonesia’s first billion-dollar startup, has seen an incredible amount of growth in both users and data over the past two years. Many of the ride-hailing company’s services are backed by machine learning models hosted on Google Cloud Platform.
Models range from driver
But senior executives at Go-Jek realized something: One of their most important and expensive resources, data scientists, were spending far too much time cleaning data. That wasn’t part of their remit and resulted in a waste of time and money.
As a COO, this a major concern for any company undertaking a machine learning initiative. Data scientists are hard to come by and their salaries have been on the rise for the last few years. Yet according to some reports data scientists spend upto 80% of their time just preparing data—not creating models.
Watch how operational teams at Go-Jek combined the right Google tools and processes to improve the productivity of their data scientists.
More Relevant Stories for Your Company
Everything You Need to Know About Google Cloud ML Engine 101
Machine learning is all around us today. But data scientists and IT teams tasked with creating models have a hard time bringing together the right mix of ingredients—from data, infrastructure, tools, and APIs—to do their jobs effectively. Google’s Cloud ML Engine eases many of the challenges data scientists and IT
How Kinguin Notched Up Shopping Experience with Google Recommendations AI
Over 2.14 billion people worldwide are expected to buy online this year, according to Statista. Online retail sales will account for 22% of all purchases by 2023. But in a competitive retail landscape, positive interactions can mean the difference between a sale and an abandoned shopping cart. One of the leading global
CCAI Insights: Answer Customers’ Queries & Understand Them Better with Conversation Data
With CCAI Insights, businesses can drive contact center efficiency, solve customer problems and leverage data from customer interactions to understand them better! CCAI Insights, a core piece of the Google Cloud's Contact Center AI product suite is built to help contact center management dive into data to adjust business needs,
Principles to Make Organizations Data Engineering Driven
In the “What type of data processing organisation” paper, we examined that you can build a data culture whether your organization consists mostly of data analysts, or data engineers, or data scientists. However, the path and technologies to become a data-driven innovator are different and success comes from implementing the right
