BigQuery Explainable AI for Demystifying the Inner Workings of ML Models. Now GA! - Build What's Next
Blog

BigQuery Explainable AI for Demystifying the Inner Workings of ML Models. Now GA!

6552

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

Google Cloud announces the general availability (GA) of BigQuery Explainable AI to interpret machine learning (ML) models. Read this blogpost to understand the applicability of BigQuery Explainable AI along with relevant examples.

Explainable AI (XAI) helps you understand and interpret how your machine learning models make decisions. We’re excited to announce that BigQuery Explainable AI is now generally available (GA). BigQuery is the data warehouse that supports explainable AI in a most comprehensive way w.r.t both XAI methodology and model types. It does this at BigQuery scale, enabling millions of explanations within seconds with a single SQL query.

Why is Explainable AI so important? To demystify the inner workings of machine learning models, Explainable AI is quickly becoming an essential and growing need for businesses as they continue to invest in AI and ML. With 76% of enterprises now prioritizing artificial intelligence (AI) and machine learning (ML) over other initiatives in 2021 IT budgets, the majority of CEOs (82%) believe that AI-based decisions must be explainable to be trusted according to a PwC survey.

While the focus of this blogpost is on BigQuery Explainable AI, Google Cloud provides a variety of tools and frameworks to help you interpret models outside of BigQuery, such as with Vertex Explainable AI, which includes AutoML Tables, AutoML Vision, and custom-trained models.

So how does Explainable AI in BigQuery work exactly? And how might you use it in practice? 

Two types of Explainable AI: global and local explainability

When it comes to Explainable AI, the first thing to note is that there are two main types of explainability as they relate to the features used to train the ML model: global explainability and local explainability.

Imagine that you have a ML model that predicts housing price (as a dollar amount), based on three features: (1) number of bedrooms, (2) distance to the nearest city center, and (3) construction date.

Global explainability (a.k.a. global feature importance) describes the features’ overall influence on the model and helps you understand if a feature had a greater influence than other features over the model’s predictions. For example, global explainability can reveal that the number of bedrooms and distance to city center typically has a much stronger influence than the construction date on predicting housing prices. Global explainability is especially useful if you have hundreds or thousands of features and you want to determine which features are the most important contributors to your model. You may also consider using global explainability as a way to identify and prune less important features to improve the generalizability of their models.

Local explainability (a.k.a. feature attributions) describes the breakdown of how each feature contributes towards a specific prediction. For example, if the model predicts that house ID#1001 has a predicted price of $230,000, local explainability would describe a baseline amount (e.g. $50,000) and how each of the features contributes on top of the baseline towards the predicted price. For example, the model may say that on top of the baseline of $50,000, having 3 bedrooms contributed an additional $50,000, close proximity to the city center added $100,000, and construction date of 2010 added $30,000, for a total predicted price of $230,000. In essence, understanding the exact contribution of each feature used by the model to make each prediction is the main purpose of local explainability.

What ML models does BigQuery Explainable AI apply to?

BigQuery Explainable AI applies to a variety of models, including supervised learning models for IID data and time series models. The documentation for BigQuery Explainable AI provides an overview of the different ways of applying explainability per model. Note that each explainability method has its own way of calculation (e.g. Shapley values), which are covered more in-depth in the documentation.

Explainable AI offerings in BigQuery ML
See: https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-syntax-xai-overview

Examples with BigQuery Explainable AI

In this next section, we will show three examples of how to use BigQuery Explainable AI in different ML applications: 

Regression models with BigQuery Explainable AI

Let’s use a boosted tree regression model to predict how much a taxi cab driver will receive in tips for a taxi ride, based on features such as number of passengers, payment type, total payment and trip distance. Then let’s use BigQuery Explainable AI to help us understand how the model made the predictions in terms of global explainability (which features were most important?) and local explainability (how did the model arrive at each prediction?).

The taxi trips dataset comes from the BigQuery public datasets and is publicly available in the table: bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2018

First, you can train a boosted tree regression model.

  CREATE OR REPLACE MODEL bqml_tutorial.taxi_tip_regression_model
OPTIONS (model_type='boosted_tree_regressor',
         input_label_cols=['tip_amount'],
         max_iterations = 50,
         tree_method = 'HIST',
         subsample = 0.85,
         enable_global_explain = TRUE
) AS
SELECT
  vendor_id,
  passenger_count,
  trip_distance,
  rate_code,
  payment_type,
  total_amount,
  tip_amount
FROM
  `bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2018`
WHERE tip_amount >= 0
LIMIT 1000000

Now let’s do a prediction using ML.PREDICT, which is the standard way in BigQuery ML to make predictions without explainability.

  SELECT *
FROM
ML.PREDICT(MODEL bqml_tutorial.taxi_tip_regression_model,
 (
 SELECT
   "0" AS vendor_id,
   1 AS passenger_count,
   CAST(5.85 AS NUMERIC) AS trip_distance,
   "0" AS rate_code,
   "0" AS payment_type,
   CAST(55.56 AS NUMERIC) AS total_amount))
Regression ML Predict

But you might wonder—how did the model generate this prediction of ~11.077?

BigQuery Explainable AI can help us answer this question. Instead of using ML.PREDICT, you use ML.EXPLAIN_PREDICT with an additional optional parameter top_k_features. ML.EXPLAIN_PREDICT extends the capabilities of ML.PREDICT by outputting several additional columns that explain how each feature contributes to the predicted value. In fact, since ML.EXPLAIN_PREDICT includes all the output from ML.PREDICT anyway, you may want to consider using ML.EXPLAIN_PREDICT every time instead.

  SELECT *
FROM
ML.EXPLAIN_PREDICT(MODEL bqml_tutorial.taxi_tip_regression_model,
 (
 SELECT
   "0" AS vendor_id,
   1 AS passenger_count,
   CAST(5.85 AS NUMERIC) AS trip_distance,
   "0" AS rate_code,
   "0" AS payment_type,
   CAST(55.56 AS NUMERIC) AS total_amount),
 STRUCT(6 AS top_k_features))
Regression ML Explain Predict

The way to interpret these columns is:

Σfeature_attributions + baseline_prediction_value = prediction_value

Let’s break this down. The prediction_value is ~11.077, which is simply the predicted_tip_amount. The baseline_prediction_value is ~6.184, which is the tip amount for an average instance. top_feature_attributions indicates how much each of the features contributes towards the prediction value. For example, total_amount contributes ~2.540 to the predicted_tip_amount

ML.EXPLAIN_PREDICT provides local feature explainability for regression models. For global feature importance, see the documentation for ML.GLOBAL_EXPLAIN.

Classification models with BigQuery Explainable AI

Let’s use a logistic regression model to show you an example of BigQuery Explainable AI with classification models. We can use the same public dataset as before: bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2018.

Train a logistic regression model to predict the bracket of the percentage of the tip amount out of the taxi bill.

  CREATE OR REPLACE MODEL bqml_tutorial.taxi_tip_classification_model
OPTIONS
 (model_type='logistic_reg',
  input_label_cols=['tip_bucket'],
  enable_global_explain=true
) AS
SELECT
  vendor_id,
  passenger_count,
  trip_distance,
  rate_code,
  payment_type,
  total_amount,
  CASE
    WHEN tip_amount > total_amount*0.20 THEN '20% or more'
    WHEN tip_amount > total_amount*0.15 THEN '15% to 20%'
    WHEN tip_amount > total_amount*0.10 THEN '10% to 15%'
  ELSE '10% or less'
  END AS tip_bucket
FROM
  `bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2018`
WHERE tip_amount >= 0
LIMIT 1000000

Next, you can run ML.EXPLAIN_PREDICT to get both the classification results and the additional information for local feature explainability. For global explainability, you can use ML.GLOBAL_EXPLAIN. Again, since ML.EXPLAIN_PREDICT includes all the output from ML.PREDICT anyway, you may want to consider using ML.EXPLAIN_PREDICT every time instead.

  SELECT *
FROM
ML.EXPLAIN_PREDICT(MODEL bqml_tutorial.taxi_tip_classification_model,
 (
 SELECT
   "0" AS vendor_id,
   1 AS passenger_count,
   CAST(5.85 AS NUMERIC) AS trip_distance,
   "0" AS rate_code,
   "0" AS payment_type,
   CAST(55.56 AS NUMERIC) AS total_amount),
 STRUCT(6 AS top_k_features))
Classification ML Explain Predict

Similar to the regression example earlier, the formula is used to derive the prediction_value:

Σfeature_attributions + baseline_prediction_value = prediction_value

As you can see in the screenshot above, the baseline_prediction_value is ~0.296. total_amount is the most important feature in making this specific prediction, contributing ~0.067 to the prediction_value, though followed by trip_distance. The feature passenger_count contributes negatively to prediction_value by -0.0015. The features vendor_idrate_code, and payment_type did not seem to contribute much to the prediction_value.

You may wonder why the prediction_value of ~0.389 doesn’t equal the probability value of  ~0.359. The reason is that unlike for regression models, for classification models, prediction_value is not a probability score. Instead, prediction_value is the logit value (i.e., log-odds) for the predicted class, which you could separately convert to probabilities by applying the softmax transformation to the logit values. For example, a three-class classification has a log-odds output of [2.446, -2.021, -2.190]. After applying the softmax transformation, the probability of these class predictions is [0.9905, 0.0056, 0.0038].

Time-series forecasting models with BigQuery Explainable AI

Plot of historical daily number of bike trips in NYC

Explainable AI for forecasting provides more interpretability into how the forecasting model came to its predictions. Let’s go through an example of forecasting the number of bike trips in NYC using the new_york.citibike_trips public data in BigQuery.

You can train a time-series model ARIMA_PLUS:

  CREATE OR REPLACE MODEL bqml_tutorial.nyc_citibike_arima_model
OPTIONS
  (model_type = 'ARIMA_PLUS',
   time_series_timestamp_col = 'date',
   time_series_data_col = 'num_trips',
   holiday_region = 'US'
  ) AS
SELECT
   EXTRACT(DATE from starttime) AS date,
   COUNT(*) AS num_trips
FROM
  `bigquery-public-data.new_york.citibike_trips`
GROUP BY date

Next, you can first try forecasting without explainability using ML.FORECAST:
SELECT
  *
FROM
  ML.FORECAST(MODEL bqml_tutorial.nyc_citibike_arima_model,
              STRUCT(365 AS horizon, 0.9 AS confidence_level))

This function outputs the forecasted values and the prediction interval. Plotting it in addition to the input time series gives the following figure.

Plot of historical daily number of bike trips with forecasts and prediction intervals using ML.FORECAST

But how does the forecasting model arrive at its predictions? Explainability is especially important if the model ever generates unexpected results.

With ML.EXPLAIN_FORECAST, BigQuery Explainable AI provides extra transparency into the seasonality, trend, holiday effects, level (step) changes, and spikes and dips outlier removal. In fact, since ML.EXPLAIN_FORECAST includes all the output from ML.FORECAST anyway, you may want to consider using ML.EXPLAIN_FORECAST every time instead.

  SELECT
  *
FROM
  ML.EXPLAIN_FORECAST(MODEL bqml_tutorial.nyc_citibike_arima_model,
                      STRUCT(365 AS horizon, 0.9 AS confidence_level))
Plot of historical daily number of bike trips with forecasts and prediction intervals, and the time series component breakdown using ML.EXPLAIN_FORECAST.

Compared to the previous figure which only shows the forecasting results, this figure shows much richer information to explain how the forecast is made.  

First, it shows how the input time series is adjusted by removing the spikes and dips anomalies, and by compensating the level changes. That is:

time_series_adjusted_data = time_series_data - spikes_and_dips - step_changes

Second, it shows how the adjusted input time series is decomposed into different components such as both weekly and yearly seasonal components, holiday effect component and trend component. That is

time_series_adjusted_data = trend + seasonal_period_yearly + seasonal_period_weekly + holiday_effect + residual

Finally, it shows how these components are forecasted separately to compose the final forecasting results. That is:

time_series_data = trend + seasonal_period_yearly + seasonal_period_weekly + holiday_effect

For more information on these time series components, please see the documentation here.

Conclusion

With the GA of BigQuery Explainable AI, we hope you will now be able to interpret your machine learning models with ease. 

Thanks to the BigQuery ML team, especially Lisa Yin, Jiashang Liu, Amir Hormati, Mingge Deng, Jerry Ye and Abhinav Khushraj. Also thanks to the Vertex Explainable AI team, especially David Pitman and Besim Avci.

3371

Of your peers have already watched this video.

16:30 Minutes

The most insightful time you'll spend today!

Case Study

Scaling Data-Driven Insights Across a Complex Global Organization with Looker and BigQuery

In this video, SpringML and Iron Mountain share how migrating data to Google Cloud not only saved hundreds of thousands of dollars in licensing consolidation but allows stakeholders across a complex global organization to:

  • Consolidate 1,200 data management applications into one data lake
  • Unify information governance practices across disparate corporate verticals and departments
  • Provide a single view of operational and customer data for enhanced decision making across 43 countries and 200 acquisitions
  • Unlock the power of data science and predictive analytics

Standardizing across varying systems and processes onto one platform took just a matter of weeks for a project that could have taken about a year.

SpringML and Iron Mountain share how this was done, lessons learned and next steps for their collaboration adopting Looker and BigQuery into Iron Mountain.

Trend Analysis

2022 Healthcare Trends: Healthcare Data, M&As, Better Patient Care, AI in Drug Development & Strategic Partnerships

6602

Of your peers have already read this article.

3:00 Minutes

The most insightful time you'll spend today!

Despite the challenges that put global healthcare systems to test, the COVID-19 pandemic was an opportune moment to embed technology at the core of healthcare and life sciences. This year, unlocking data will dictate its growth and value generation!

The COVID-19 pandemic continues to push the healthcare and life sciences industry in entirely new ways. In record time, we’ve witnessed public health officials, vaccine developers, equipment manufacturers, and essential workers take life saving actions—regularly putting their own lives at risk—to respond to the exceptional challenges of our time. 

Yet after all the turmoil and uncertainty of the pandemic, record breaking levels of investment continue to come into the market to fuel innovations. 

Vaccine development is now measured in weeks rather than years; providers are leveraging telehealth technologies to improve the physician and patient experience; and individuals have embraced a variety of devices to assume greater ownership and control of their personal health. 

With that in mind, here are five of the many innovations I see driving healthcare and life sciences for at least the next 12 months: 

1. Unleashing the power of healthcare and life sciences data

People have arguably never had as much access and understanding to their personal health data than they can in 2022. They have the ability to understand their genetic makeup, medical history, family history, and activity levels to ensure they are living a healthy lifestyle. Taken together, this longitudinal profile can become the basis for more personalized medicine. Add wearable devices to the mix and patients gain real- or near-real-time updates on their health status. 

Further, global regulatory agencies have continued to mandate the need for providers to maintain a longitudinal health record that provides a holistic view of the patient across all the encounters they have had with a health system. Physician surveys conducted by the The Harris Poll and Google Cloud show that a 360-degree view of the patient, across all provider encounters, leads to faster, more accurate diagnosis, and better outcomes. 

If secure data access and interoperability can begin to include insurers, researchers, public health officials, and others in the field, the powerful network effects benefiting patients will only grow. When combined with those longitudinal phenome profiles, the opportunities for personalized and preventative medicine enter a whole new era.

2. Healthcare and life sciences M&A boom continues

Although the pandemic initially slowed activity on mergers and acquisitions in the early part of 2020, the healthcare and life sciences industry has seen a rapid rebound and acceleration of deals ever since. The success of COVID-19 vaccines, the importance of telehealth, and the focus on molecular modeling and genomic-based drug development have all boosted investment as organizations look to enter new markets, develop new therapies, and leverage low interest rates while they last. 

In 2021, the total funding of US digital health startups surpassed $29 billion across 729 deals, according to advisory Rock Health. That’s almost double the levels of 2020, which itself set records. Analysts at PWC meanwhile estimate M&A investments in biopharmaceutical and life sciences could approach $400 billion this year across all sub-sectors

Clearly, the pandemic has been a primary driver of investment as the focus on healthcare has dramatically increased. Yet the increased activity also reflects changing business models and emerging technologies that are now required to compete in the rapidly evolving space. For organizations to capitalize on these investments, it will take not only great vision and intellectual property but also the right technologies—like cloud—and the right data interoperability models, to make partnerships and acquisitions more scalable, feasible, and seamless.

3. Transforming the patient experience at a new rate

The pandemic has shined a spotlight on the inefficiencies and complexities that exist in healthcare markets across the globe. As wave after wave has surged, global healthcare systems remain overwhelmed on most every aspect of patients’ treatment journeys. Even before COVID-19, healthcare was already one of the largest spend areas for governments around the world. The pandemic has only exacerbated the known issues. 

Clearly, administrators, regulators, physicians, nurses, and patients would agree that the processes and models need to change. There’s a need to maintain this momentum and even increase the tempo to achieve lasting change.

Take telehealth. Within months of the start of the pandemic, providers moved to provide more remote capabilities so physicians could still meet with patients virtually to ensure health and safety on all sides. Payers recognized the importance of telehealth and began to update reimbursement rules. And organizations are now reimagining policies in areas such as prior authorization, submission, and adjudication to reduce complexity and bureaucracy while improving responsiveness.

Looking beyond the system, organizations are also recognizing and deepening their understanding of the structural and social determinants of health that impact patient care and health outcomes, especially for historically underserved communities.  Private and public sectors are learning from, and increasingly partnering with, the social sciences, public health, biomedical informatics, computer science, public policy and community groups around how to build a mI’ore equitable and inclusive consumer products and Health IT strategies. 

The newfound levels of transparency, visibility, and accountability that patients, caregivers, and organizations are achieving will ultimately increase competition and provide a more effective, equitable, efficient and, above all, healthier marketplace for all patients. As we move past the worst of the pandemic, regulators and organizations should keep fighting for progress over business as usual.

4. AI is now a core competency for Drug Development

The ability of organizations like Pfizer, Moderna, Johnson & Johnson, and Astrazeneca to develop COVID-19 vaccines has been a remarkable accomplishment—particularly the historic speed with which they were created and deployed. This innovation acceleration was largely enabled by the use of new drug development platforms that allow researchers to use artificial intelligence and machine learning to model protein and cellular interactions to rapidly advance the science.

No longer must researchers rely on traditional laboratory testing (and retesting). With their improved understanding of the molecular and genetic structure of a patient and, for example, their tumor, researchers can use AI to enable simulations on computers rather than testing in live conditions. This technology can process thousands, even millions of simulations to help  identify high-potential candidates for treatment consideration and subsequent analysis. 

AI-enabled drug discovery models can eliminate months and years from the research process, which can reduce the time to develop a drug and accelerate the time to treatment for an individual patient. As just one example, consider the work on AlphaFold2 by Google’s DeepMind unit who leverages AI to predict effective protein shapes for new drugs. Healthcare and life sciences organizations already recognize the potential of AI. Now comes the investments to leverage this rapidly evolving technology to support their efforts now and in the future.   

5. Ecosystem partnerships tackle complexity and spur innovation

As the importance and growth of the healthcare and life sciences industry continues, we will see even more new players and partnerships emerging to address old problems in new ways. This trend will touch all aspects of the healthcare value chain and will, increasingly, see three- and four-player partnerships emerge to address the complex challenges of today’s healthcare marketplace.

Technology will continue to play a key role as capabilities and platforms will transform all aspects of the marketplace. Cell phones, wearable devices, and other technologies will provide real-time updates and notifications to patients on everything from glucose levels to payments for healthcare services. Voice recognition software will document physician and patient discussions to reduce the burden of record keeping. Real-world data will be used to simplify and confidentially recruit patients for participation in clinical trials. 

New players will continue to enter the market to improve health outcomes and reduce costs. Major retailers are among the companies extending their pharmacies to provide additional diagnostic and concierge services, saving patients from additional appointments while boosting prevention. Community organizations are emerging to help identify and care for underserved communities whose health outcomes are significantly lower than the average patient. 

For all the exhausting and heart-wrenching challenges of the past two years, the opportunities the pandemic has laid bare cannot be overlooked. We owe it to those who have worked and fought so hard for every life to forge even more new partnerships—and make it easier to do so—so that the next crisis, when it does arise, will never be as bad as the one we’re now conquering. Technology can be the enabler in this effort and help bring us together to continue to conquer the challenges that lie ahead.

Case Study

Skyscanner Supercharges Ability to Turn Raw Data into Deep Understanding of Consumer Behaviour, Conversion Jumps 40%

3803

Of your peers have already read this article.

2:30 Minutes

The most insightful time you'll spend today!

By integrating Google Analytics Premium with BigQuery, Skyscanner was able to run various pieces of analysis, from one-off investigations to powering daily dashboards. This sped up analysis and action, resulting in conversion rate improvements, among other benefits.

Skyscanner is a leading global travel search company covering flights, hotels, and car hire around the world. Founded in 2003, the company helps over 40 million people each month find the best travel options across its portfolio of websites and mobile apps.

Skyscanner wanted to understand the anonymized behaviour of consumers on a more granular level than was possible via standard reports in the Google Analytics Premium web interface or even using the reporting APIs.

“These methods work well for high-level analysis and standard marketing reporting, but we were keen to dig deeper into the data to get more insight and further optimise our products,” explains Mark Shilton, Principal Analyst in the Skyscanner Data Team.

For example, the company wanted to create detailed cohorts to understand how users interacted with Skyscanner over time. Also, different teams in the business were keen to understand the performance of individual pieces of functionality that fell within their remit.

To do this, they needed to understand not just the overall conversion rate, but how users who interact with a given piece of functionality convert compared to those who do not. Skyscanner also wanted drill down into specific markets, devices types, and marketing channels.

Mapping a plan for deeper insights

Skyscanner opted to address all of these needs by integrating Google Analytics Premium with BigQuery. The integration has become the starting point for many detailed investigations across the business.

For example, analysts and engineers now run cohort analyses to understand how frequently users return to Skyscanner and which channels are most effective at which part of the customer journey.

“This type of analysis is allowing a much deeper understanding of our marketing activity and is informing our future strategy and spend,” Mark says.

He reveals that using BigQuery in conjunction with other tools such as Tableau and Python also helps Skyscanner execute analysis much more quickly and efficiently than before.

“While in the past it was tricky to get a fully unsampled report based on specific segments of users flowing directly from Google Analytics Premium into a Tableau dashboard, now it is simply a matter of writing the query, creating a connection in Tableau to automatically refresh the data daily, and publishing this dashboard to the rest of the company.”

Another key benefit of using a flexible combination of tools is the ability to keep an eye on costs.

“Where the aggregations and segments of data are required on a regular basis, we have to consider the potential cost of querying the entire BigQuery dataset multiple times for the same data,” Mark explains. “To minimise this, we use Python scripts to automate these aggregations into new, smaller tables that are much more cost efficient to query.”

Excellent visibility and a clear path ahead

Combining Google Analytics Premium with BigQuery has supercharged Skyscanner’s ability to turn raw data into deep understanding of consumer behaviour.

“We have been using BigQuery for various pieces of analysis, from one-off investigations to powering daily dashboards,” Mark affirms. “In all cases it has speeded up our workflow and enabled us to gain greater insight more quickly. In a fast moving internet economy, this is key. Instead of setting up and scheduling one or more unsampled API reports, analysts can now write a query against BigQuery and have results almost instantly.”

“BigQuery has also allowed us to more easily isolate the effects of marketing from the effects of site changes,” Mark says. “By writing queries that focus on the conversion rate from specific pages in the funnel, we can better segment our traffic. We can separate traffic from various sources, including: specific marketing campaigns, users who make it to specific parts of the funnel, or users who interact with new functionality. This greater understanding has played a key role in improving overall conversion rates on our websites, particularly on mobile where we’ve achieved conversion rate improvements of 30 to 40% on smartphone and tablet devices in the last six months.”

Looking to the future, Skyscanner’s next steps include exploring how this data can be used to segment, cluster, and classify users for machine learning analyses, which would not have been possible using standard reporting functionality.

5200

Of your peers have already watched this video.

5:00 Minutes

The most insightful time you'll spend today!

How-to

How to Build and Maintain Spark and Hadoop Clusters Quickly, Easily and Inexpensively

How often have you struggled to create Spark and Hadoop clusters only to have them become unstable?

Building and maintaining clusters that are the building blocks of big data analytics should not be a time-consuming, repetitive, and expensive process. But with the current set of tools, it often is.

Google Cloud Dataproc solves this. It is a fast, easy-to-use, fully-managed cloud service for running Spark and Hadoop clusters in a simple, more cost-efficient way.

Operations that used to take hours or days take now seconds or minutes instead, and you pay only for the resources you use (with per-second billing).

Watch this five-minute demo to find out how—with just a few clicks—you can set up clusters that build and heal themselves. You’ll find out how to creating a large Dataproc cluster with preemptible VMs, which help run large clusters at a low total cost.

3033

Of your peers have already watched this video.

24:00 Minutes

The most insightful time you'll spend today!

Case Study

How to Modernize Your Data infrastructure and Analytics: Case Study

Data modernization is key to digital transformation. Legacy systems are slow, expensive and cannot keep up with changing requirements. Data teams often spend more time on data pipelines than they do on data analysis or data science.

in this video, Harish Ramachandraiah, Director Eng. & Analytics, Sunrun, speaks with Joel Mckelvey, Product Marketing Manager, Google Cloud. They discuss how Sunrun leveraged Looker and BigQuery to reduce the complexity of legacy extract-transform-load processes, improve database performance, and adapt quickly to changes in data.

Now Sunrun makes data accessible where it’s needed, in near-real time.

Mckelvey shows how a modern data stack helps companies simplify the data pipeline, consolidate vital data, and accelerate analytics performance while improving agility, efficiency, and data governance.

More Relevant Stories for Your Company

Podcast

Setting up Data Pipelines Easily for Streaming and Non-Streaming Data

We live in a world of incessant data. Our phones, factory equipment, smart watches, health devices, and other IoT devices are constantly streaming data. Some of the most popular uses of machine learning and AI are also based on streaming data. Think real-time credit card fraud detection, or software that

Blog

STAC-M3 Tick History Analytics in Google Cloud Benchmark Results Reveals it is 18X Faster than Previous Version

The Securities Technology Analysis Center (STAC®), an organization that improves technology discovery and assessment in the finance industry through dialog and research, recently audited the STAC-M3™ benchmark suite on Google Cloud (SUT ID KDB211210). These enterprise tick-analytics benchmarks assess the ability of a solution stack such as database software, servers,

Case Study

S4 Agtech Transforms Agriculture with Google Cloud

Like countless other industries, farming is going digital and undergoing big changes—driven by access to more actionable information. The agriculture business can now gather and analyze georeferenced data from satellites, combined with data from IoT sensors in fields, crop rotation and yield histories, weather patterns, seed genotypes and soil composition to help

How-to

Guide for Measuring Cloud Spanner Performance for Your Custom Workload

Database migration to a new database platform or technology can be daunting for various reasons. One of the common concerns is database performance. It is hard to evaluate database performance early in the evaluation cycle without performing actual data migration and application changes. This becomes even more important in the

SHOW MORE STORIES