Google Migration and BigQuery Brings PedidosYa Closer towards its Goal of Becoming Data-driven - Build What's Next
Case Study

Google Migration and BigQuery Brings PedidosYa Closer towards its Goal of Becoming Data-driven

7238

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

PedidosYa, a Latin American leader in the online food ordering space with over 20 million app downloads was looking to democratize data by gaining a secured access and building a comprehensive information ecosystem. PedidosYa was also challenged with its legacy data warehouse that couldn't keep with the brand's increasing analytics demands and was time-consuming and costly. To modernize their data warehouse and transform the analytics environment, PedidosYa chose Google Cloud for its serverless, managed, and integrated data platform, coupled with its seamless integration across open-source solutions. Also, Google Cloud's advanced cost and workload management coupled with its transparent log analytic gave the brand full visibility into any query performance issues to make improvements as they wanted. BigQuery helped PedidosYa achieve cost reduction per query by 5x and leverage its AL/ML-based stack for productivity benefits. Read on to learn more about PedidosYa's BigQuery and Google Cloud migration journey as a step closer towards becoming a data-driven organization.

Editor’s note: PedidosYa is the market leader for online food ordering in Latin America, serving 15 markets and over 400 cities. It’s also one of the largest brands within the German multinational company Delivery Hero SE. With over 20 million app downloads, PedidosYa provides the best online delivery experience through 71,000+ online partners, including restaurants, shops, drugstores, and specialized markets. 

Having constant access to fresh customer data is a key requirement for PedidosYa to improve and innovate our customer’s experience. Our internal stakeholders also require faster insights to drive agile business decisions. Back in early 2020, PedidosYa’s leadership tasked the data team to make the impossible possible. Our team’s mission was to democratize data by providing universal and secure access while creating a comprehensive information ecosystem across PedidosYa. We also had to achieve this goal while keeping costs under control— even during the migration stage and removing operational bottlenecks. 


Challenges with legacy cloud infrastructure

PedidosYa first built its data platform on top of AWS. Our data warehouse ran on Redshift, and our data lake was in S3. We used Presto and Hue as the user interfaces for our data analysts. However, maintaining this infrastructure was a daunting task. Our legacy platform couldn’t keep up with the increasing analytics demands. For example, the data stored on S3 complemented by Presto/Hue required high operational overhead. This was because Presto and our IAM (identity access management) didn’t integrate well in our legacy ecosystem. Managing individual users and mapping IAM roles with groups and Kerberos was operationally time-consuming and costly. Further, sharding access on the S3 files was far too complicated to enable seamless ACLs (access control lists).  

There were also challenges with workload management. Our data warehouse had batch data loaded overnight. If one analyst scheduled a query to run during the overnight ETL (extract, transform, load) workload, it would disrupt the current ETL task. This could stop the entire data pipeline. We’d have to wait until data engineers intervened with a manual fix.

It was also difficult to understand whether a query error was due to performance issues or platform resource exhaustion. This lack of clarity affected our data analysts’ ability to autonomously improve querying efficiency. Data team members needed to manually inspect personal queries looking for performance issues. Also,  the current architecture was prone to a ‘tragedy of the commons’ situation; it was seen as an unlimited and free resource. As a result, it was impossible to disentangle the infrastructure from different stakeholder teams, as all had very different needs. 

The decision to modernize our data warehouse

Given the growing challenges from our legacy platform, our tech team decided to transform our analytics environment with a modern data warehouse. They required the following key criteria from their next data platform: 

  • Scalability – The ability to grow with elastic infrastructure.
  • Cost control – Cost management and transparency. These factors promote efficiency and ownership—both key aspects of data democratization.
  • Metadata management – Intuitive data platform focusing on users’ previous SQL knowledge. Plus, being able to enrich the informational ecosystem with metadata,  to diminish data gatekeepers.
  • Ease of management – The team needed to reduce operational costs with a serverless solution. Data engineers wanted to focus on their key roles rather than acting as database administrators and infrastructure engineers. The team also wanted much higher availability, and to reduce the impact of maintenance windows and vacuum/analysis.
  • Data governance and access rights – With a growing employee base with varying data access requirements, the team needed a simple yet comprehensive solution to understand and track user access to data.

Migrating to Google Cloud

After exploring other alternatives, we concluded Google Cloud had an answer to each of our decision drivers. Google Cloud’s serverless, managed, and integrated data platform, coupled with its seamless integration across open-source solutions, was the perfect answer for our organization. In particular, the natural integration with Airflow as a job orchestrator and Kubernetes for flexible on-demand infrastructure was key.  

We  used Dataflow together with Pub/Sub and Cloud Functions for our data ingestion requirements, which has made our deployment process with Terraform seamless. Because we set up everything in our environment programmatically, operation time has diminished. Google Cloud reduced the deployment process from about 16 hours in our legacy platform to 4 hours.  This is partly due to the friendliness of automating the deployment (such as schema check, load test, table creation, build.) process with Terraform, Cloud Functions, Pub/Sub, Dataflow, and BigQuery on GCP. Input messages processed with Dataflow allow us to abstract and plan the schema changes according to the needs of the functional team. For example, schema changes raise an alarm, and then we can modify the raw layer table schema. By doing this, we ensure that backend modifications that we don’t control do not affect upper layers.

A key reason why we picked Google Cloud was because of its advanced cost and workload management coupled with its transparent log analytics. This information gives us a complete view into any query performance issues to make improvements on the fly. Further, we achieved a significant amount of cost savings by consolidating multiple tools to BigQuery.With BigQuery, we’ve been able to reduce our total cost per query by 5x.

This was due to a number of reasons:

  • Automating pipeline deployment made it much simpler to maintain the data processing processes. 
  • Analysts are conscious about what queries they’re running, resulting in running better, more optimized queries. 
  • Analysts use a Data Studio dashboard to see their queries and all the associated costs. As a result, there’s a lot more transparency for each persona.

 With these changes, we can easily manage and assign costs associated with each workload with their own cost centers using specific Google Cloud projects.

Change management is always challenging. However, BigQuery is intuitive and doesn’t have a steep learning curve from Hue/Hive on SQL basics. BigQuery also allowed the team to expand its capabilities and enabled them to properly work with nested structures, avoiding unnecessary joins and improving query efficiency. Additionally, we now use Data Catalog as our unique point of truth for metadata management. This allows our team to break the data access barriers and enable federation of data across the organization. By using Airflow to orchestrate everything, we keep track of every data stream. With this information, each end user can see their regularly used data entities’ status via the dashboard. This also adds transparency to our everyday data processes.

Finally, with Google Cloud’s IAM rules applied across the different products, data sharing and access is close to a noOps experience. We have programmatically implemented access according to roles and level access within the company. This allows certain pre-validated roles to view more sensitive information. These solutions help drive a more automated data governance experience. 

Up next: Google Cloud AI/ML

The new stack based on BigQuery has created significant productivity gains. Freed from the burden of operational management, PedidosYa’s data team can now focus on adding value through data tools and products.  

  • Our data engineers are better equipped to integrate constantly changing transactional and operational data.
  • The dataOps team can automate the infrastructure and provide autonomy to the end user.
  • Our data quality team can focus on bringing added value to data stakeholders. 
  • Data scientists and data analytics can spend more time analyzing data and less time asking data gatekeepers for data access.

PedidosYa can now democratize data access with a well-governed architecture. We are still at the beginning of our journey, but we are closer to achieving our vision of building a data-driven organization. Up next: expanding our artificial intelligence and machine learning capabilities.

Tune in to Google Cloud’s Applied ML Summit on June 10th, 2021, or listen on-demand later, to learn how to apply groundbreaking machine learning technology in your projects.

Blog

How Google Cloud’s PSO Supports Customers’ Migration Goals

5029

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

Google Cloud's Professional Services Organization (PSO) ensures customers are supported throughout their cloud journey. Read the blog to explore how PSO is equipped to ease cloud migration journey with necessary tools and guidance.

Google Cloud’s Professional Services Organization (PSO) engages with customers to ensure effective and efficient operations in the cloud, from the time they begin considering how cloud can help them overcome their operational, business or technical challenges, to the time they’re looking to optimize their cloud workloads. 

We know that all parts of the cloud journey are important and can be complex.  In this blog post, we want to focus specifically on the migration process and how PSO engages in a myriad of activities to ensure a successful migration.

As a team of trusted technical advisors, PSO will approach migrations in three phases:

  1. Pre-Migration Planning
  2. Cutover Activities
  3. Post-Migration Operations

While this post will not cover in detail all of the steps required for a migration, it will focus on how PSO engages in specific activities to meet customer objectives, manage risk, and deliver value.  We will discuss the assets, processes and tools that we leverage to ensure success.

Pre-Migration Planning

Assess Scope

Before the migration happens, you will need to understand and clarify the future state that you’re working towards.  From a logistical perspective, PSO will be helping you with capacity planning to ensure sufficient resources are available for your envisioned future state.

While migration into the cloud does allow you to eliminate many of the considerations for the physical, logistical, and financial concerns of traditional data centers and co-locations, it does not remove the need for active management of quotas, preparation for large migrations, and forecasting.  PSO will help you forecast your needs in advance and work with the capacity team to adjust quotas, manage resources, and ensure availability. 

Once the future state has been determined, PSO will also work with the product teams to determine any gaps in functionality.  PSO captures feature requests across Google Cloud services and makes sure they are understood, logged, tracked, and prioritized appropriately with the relevant product teams.  From there, they work closely with the customer to determine any interim workarounds that can be leveraged while waiting for the feature to land, as well as providing updates on the upcoming roadmap.  

Develop Migration Approach and Tooling

Within Google Cloud, we have a library of assets and tools we use to assist in the migration process.  We have seen these assets help us successfully complete migrations for other customers efficiently and effectively.

Based on the scoping requirements and tooling available to assist in the migration, PSO will help recommend a migration approach.  We understand that enterprises have specific needs; differing levels of complexity and scale; regulatory, operational, or organization challenges that will need to be factored into the migration.  PSO will help customers think through the different migration options and how all of the considerations will play out.

PSO will work with the customer team to determine the best migration approach for moving servers from on-prem to Google Cloud. PSO will walk customers through different migration approaches, such as refactoring, lift-shift, or new installs. From there, the customer can determine the best fit for their migration. PSO will provide guidance on best practices and use cases from other customers with similar use cases. 

Google offers a variety of cloud native tools that can assist with asset discovery, the migration itself, and post-migration optimization. PSO, as one example, will help work with project managers to determine the best tooling that accommodates the customer’s requirements for migrating servers. PSO will also engage Google product team to ensure the customer fully understands the capabilities of each tool and the best fit for the use case. Google understands from a tooling perspective, one size does not fit all, thus PSO will work with the customer on determining the best migration approach and tooling for different requirements. 

Cutover Activities

Once all of the planning activities have been completed, PSO will assist in making sure the cutover is successful.

During and leading up to critical customer events, PSO can provide proactive event management services which deliver increased support and readiness for key workloads.  Beyond having a solid architecture and infrastructure on the platform, support for this infrastructure is essential and TAMs will help ensure that there are additional resources to support and unblock the customer where challenges arise.

As part of event management activities, PSO liaises with the Google Cloud Support Organization to ensure quick remediation and high resilience for situations where challenges arise.  A war room is usually created to facilitate quick communication about the critical activities and roadblocks that arise.  These war rooms can give customers a direct line to the support and engineering teams that will triage and resolve their issues.

Post-Migration Activities

Once cutover is complete, PSO will continue to provide support in areas such incident management, capacity planning, continuous operational support, and optimization to ensure the customer is successful from start to finish.

PSO will serve as the liaison between the customer and Google engineers. If support cases need to be escalated, PSO will ensure the appropriate parties are involved and work to get the case resolved in a timely manner. Through operational rigor, PSO will work with the customer in determining if certain Google Cloud services will be beneficial to the customer objectives. If services will add value to the customer, PSO will help enable the services so it aligns with the customer’s goal and current cloud architecture. In cases where there are missing gaps in services, PSO will proactively work with the customer and Google engineering teams to close the gaps by enabling additional functionality in the services.  

PSO will continue to work with the engineering teams to consistently review and provide recommendations on the customer’s cloud architecture in ensuring the most optimal and cost efficient design along with adhering to Google’s best practices guidelines. 

Aside from migrations, PSO is also responsible for providing continuous training of Google Cloud to customers. To ensure consistent development of Google Cloud, PSO will work with the customer to jointly develop a learning roadmap to ensure the customer has the necessary skills to succeed in delivering successful projects in Google Cloud.

Conclusion

Google PSO will be actively engaged throughout the customer’s cloud journey to ensure the necessary guidance, methodology, and tools are presented to the customer. PSO will engage in a series of activities from pre-migration planning to post migration in key areas such as capacity planning to ensure sufficient resources are allocated for future workloads to providing support on technical cases for troubleshooting. PSO will serve as a long-term trusted advisor who will be the voice of the  customer and provide the reliability and stability of the customer’s Google Cloud environment.

Click here if you’d like to engage with our PSO team on your migration. Or, you can also get started with a free discovery and assessment of your current IT landscape.

Blog

Taking Maps Further: New Website Experience for Product Discovery, Budgeting and Access to Dev Documentation

6630

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

As new technologies emerge and customer preferences evolve, businesses and users of Google Maps Platform can leverage its brand new 'website experience' to explore products and solution that meet their objectives! Learn More.

For more than 15 years, developers have used Google Maps Platform to deliver location-based experiences to their end users and used location intelligence to optimize their businesses. Along this journey, we’ve made a variety of changes to better support our community as needs have changed and new industries and technologies have emerged. We started rolling out a new website experience, at https://mapsplatform.google.com, to help you better understand the products and solutions best suited to address your objectives. Plus, now you can directly connect to the developer documentation for each product to get started quickly, and you can visualize usage and associated costs to have a better idea of what to expect before getting started. 

Getting to your solution faster 

Maps, Routes, Places are building blocks that let you develop implementations for any use case. Building for specific use cases, however, typically requires using a combination of APIs and SDKs. To help you quickly understand what’s possible and what you need to build for your use case, you can now visit the solutions tab to select from a list of popular use cases or industries. Once you’ve selected a use case or industry, you’re taken to a page where you can explore relevant products, read helpful blog posts, see how other customers have deployed for similar use cases, and more.  

Find the ideal location

Direct access to developer documentation

Did you know there are more than a thousand pages of developer documentation created to help you get started, unblock you when you’re stuck, and share best practices? Now when you explore a product or solution from the Google Maps Platform website, you can easily navigate back and forth between our website and documentation. Just tap on JS, iOS, Android or API under the product name to get to the documentation you need. 

Link to documentation

Budgeting for your project

To help you calculate pricing for your project, we’ve introduced a new pricing calculator. Once you find the product and API or SDK you plan to use, pull the slider to reflect your estimated number of monthly requests. This will automatically update the “monthly cost” column for each product and API or SDK you plan to use. If your estimated monthly requests exceed the slider limit, contact our sales team to ​​learn about volume discounts that start at 20% off. 

Pricing calculator

We hope our new website makes it easier to discover our products and solutions, estimate your budget, and start building with our documentation so you can deliver helpful experiences to your users and optimize your business. 

For more information on Google Maps Platform, visit https://mapsplatform.google.com.

Case Study

Rubin Observatory Leverages Google Cloud to Power Astronomical Research

6067

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

The Vera C. Rubin Observatory launched its the first preview of new Rubin Science Platform (RSP) for an initial cohort of astronomers, powered by Cloud Storage, Google Kubernetes Engine (GKE), and Compute Engine to drive astronomical researches.

This week, the Vera C. Rubin Observatory is launching the first preview of its new Rubin Science Platform (RSP) for an initial cohort of astronomers. The observatory, which is located in Chile but managed by the U.S. National Science Foundation’s NOIRLab in Tucson, AZ and SLAC in California, is jointly funded by the NSF and the U.S. Department of Energy. The platform provides an easy-to-use interface to store and analyze the massive datasets of the Legacy Survey of Space and Time (LSST), which will survey a third of the sky each night for ten years, detecting billions of stars and galaxies, and millions of supernovae, variable stars, and small bodies in our Solar System.

The LSST datasets are unprecedented in size and complexity, and will be far too large for scientists to download to their personal computers for analysis. Instead, scientists will use the RSP to process, query, visualize, and analyze the LSST data archives through a mixture of web portal, notebook, and other virtual data analysis services. An initial launch with simulated data, called Data Preview 0, builds on the Rubin Observatory’s three-year partnership with Google to develop an Interim Data Facility (IDF) on Google Cloud to prototype hosting of the massive LSST dataset. This agreement marks the first time a cloud-based data facility has been used for an astronomy application of this magnitude.

Bringing the stars to the cloud

For Data Preview 0, the IDF leverages Cloud StorageGoogle Kubernetes Engine (GKE), and Compute Engine to provide the Rubin Observatory user community access to simulated LSST data in an early version of the RSP. The simulated data were developed over several years by the LSST Dark Energy Science Collaboration to imitate five years of an LSST-like survey over 300 square degrees of the sky (about 1,500 times the area of the moon). The resulting images are very realistic: they have the same instrumental characteristics, such as pixel size and sensitivity to photons, that are expected from the Rubin Observatory’s LSST Camera, and they were processed with an early version of the LSST Science Pipelines that will eventually be used to process LSST data. “This will be the first time that these workloads have ever been hosted in a cloud environment. Researchers will have an opportunity to explore an early version of this platform,” says Ranpal Gill, senior manager and head of communications at the Rubin Observatory.

Broadening access for more researchers

Over 200 scientists and students with Rubin Observatory data rights were selected to participate in Data Preview 0 from a pool of applicants that represents a wide range of demographic criteria, regions, and experience level. Participants will be supported with resources such as tutorials, seminars, communication channels, and networking opportunities—and they will be free to pursue their own science at their own pace using the data in the RSP. 

“The revolutionary nature of the future LSST dataset requires a commensurately innovative system for data access and analysis paired with robust support for scientists,” says Melissa Graham, lead community scientist for the Rubin Observatory and research scientist in the astronomy department at the University of Washington. “I’m personally excited to enhance my own skills by using the RSP’s tools for big data analysis, while also helping others to learn and to pursue their LSST-related science goals during Data Preview 0.” 

At the same time, the fact that the RSP is hosted in the cloud provides researchers at smaller institutions access to state-of-the-art astronomy infrastructure that is comparable to that of the largest national research centers.

The launch benefits the observatory too: the development team can learn what researchers are interested in while also testing and debugging the platform. Graham says that “the platform is still in active development so researchers using it will be able to follow along in the progress, and provide feedback on ways that we can optimize the development of the tools.”

Next steps

The LSST aims to begin the ten-year survey in 2023-24 and expects it to include 500 petabytes of data. Through the cloud, Google aims to help make this extraordinary project scalable and accessible to researchers everywhere. To learn more about Data Preview 0, watch this video.


Want to ramp up your own research in the cloud? We offer research credits to academics using Google Cloud for qualifying projects in eligible countries. You can find our application form on Google Cloud’s website or contact our sales team.

Blog

Supercharging Security with Generative AI

1497

Of your peers have already read this article.

3:30 Minutes

The most insightful time you'll spend today!

Google Cloud introduces its Security AI Workbench and Sec-PaLM, employing AI to advance threat response, transform cybersecurity, and empower all levels of security professionals. Read more...

At Google Cloud, we continue to invest in key technologies to progress towards our true north star on invisible security: making strong security pervasive and simple for everyone. Our investments are based on insights from our world-class threat intelligence teams and experience helping customers respond to the most sophisticated cyberattacks. Customers can tap into these capabilities to gain perspective and visibility on the most dangerous threat actors that no one else has. 

Recent advances in artificial intelligence (AI), particularly large language models (LLMs), accelerate our ability to help the people who are responsible for keeping their organizations safe. These new models not only give people a more natural and creative way to understand and manage security, they give people access to AI-powered expertise to go beyond what they could do alone. 

At the RSA Conference 2023, we are excited to announce Google Cloud Security AI Workbench, an industry-first extensible platform powered by a specialized, security LLM, Sec-PaLM. This new security model is fine-tuned for security use cases, incorporating our unsurpassed security intelligence such as Google’s visibility into the threat landscape and Mandiant’s frontline intelligence on vulnerabilities, malware, threat indicators, and behavioral threat actor profiles. 

Google Cloud Security AI Workbench powers new offerings that can now uniquely address three top security challenges: threat overload, toilsome tools, and the talent gap. It will also feature partner plug-in integrations to bring threat intelligence, workflow, and other critical security functionality to customers, with Accenture being the first partner to utilize Security AI Workbench. 

The platform will also let customers make their private data available to the platform at inference time; ensuring we honor all our data privacy commitments to customers. Because Security AI Workbench is built on Google Cloud’s Vertex AI infrastructure, customers control their data with enterprise-grade capabilities such as data isolation, data protection, sovereignty, and compliance support. 

Preventing threats from spreading beyond the first infection

We already provide best-in-class capabilities to help organizations immediately respond to threats. But what if we could not just identify and contain initial infections, but also help prevent them from happening anywhere else? With our AI advances, we can now combine world class threat intelligence with point-in-time incident analysis and novel AI-based detections and analytics to help prevent new infections. These advances are critical to help counter a potential surge in adversarial attacks that use machine learning and generative AI systems. That’s why we’re excited to introduce:  

  • VirusTotal Code Insight uses Sec-PaLM to help analyze and explain the behavior of potentially malicious scripts, and will be able to better detect which scripts are actually threats. 
  • Mandiant Breach Analytics for Chronicle leverages Google Cloud and Mandiant Threat Intelligence to automatically alert you to active breaches in your environment. It will use Sec-PaLM to help contextualize and respond instantly to these critical findings.

These new updates build on the existing AI in Google’s industry-leading solutions. For example, Chronicle Security Operations already uses frontline intelligence, integrated reasoning, and machine learning to identify initial infections, prioritize impact, and contain threats. Another example is reCAPTCHA Enterprise, which uses image noising capabilities to help protect your site from adversaries that leverage novel AI advances, greatly enhancing our defenses against bots. 

Adding intelligence to reduce toil

At Google Cloud, we help organizations modernize security wherever they are, in part by simplifying their security tools and controls whenever possible. Advances in generative AI can help reduce the number of tools organizations need to secure their vast attack surface areas and ultimately, empower systems to secure themselves. This will minimize the toil it takes to manage multiple environments, to generate security design and capabilities, and to generate security controls. Today, we’re announcing: 

  • Assured OSS will use LLMs to help us add even more open-source software (OSS) packages to our OSS vulnerability management solution, which offers the same curated and vulnerability-tested packages that we use at Google.
  • Mandiant Threat Intelligence AI, built on top of Mandiant’s massive threat graph, will leverage Sec-PaLM to quickly find, summarize, and act on threats relevant to your organization. 

These announcements build on existing capabilities that help customers centralize visibility and control, detect targets, and improve security across their platform. For example, Security Command Center (SCC) uses always-on machine learning to detect malicious scripts executing in the customer container environment and immediately alert the customer. In addition, Cloud Data Loss Prevention leverages machine learning to find and classify data, and with Confidential Computing you can collaborate on, train, and deploy sensitive and regulated AI models in the cloud, all while preserving confidentiality. 

Evolving how practitioners do security to close the talent gap 

At Google, we believe that to truly democratize security, we need to first acknowledge that AI will soon usher in a new era for security expertise that will profoundly impact how practitioners “do” security. Most people who are responsible for security — developers, system administrators, SRE, even junior analysts — are not security specialists by training. 

Imagine a world where novices and security experts are paired with AI expertise to free themselves from repetition and burnout, and accomplish tasks that seem impossible to us today. To help power this evolution, we’re embedding Sec-PaLM-based features that can make security more understandable while helping to improve effectiveness with exciting new capabilities in two of our solutions: 

  • Chronicle AI: Chronicle customers will be able to search billions of security events and interact conversationally with the results, ask follow-up questions, and quickly generate detections, all without learning a new syntax or schema.
  • Security Command Center AI: Security Command Center will translate complex attack graphs to human-readable explanations of attack exposure, including impacted assets and recommended mitigations. It will also provide AI-powered risk summaries for security, compliance, and privacy findings for Google Cloud.

These new releases bolster our existing efforts to tackle these issues through capabilities like IAM Recommender, which suggests permissions better suited to actual usage patterns. We will soon be augmenting this capability to cover organizational policies, further enabling the administrator to help improve the security posture of their organization. In addition, Mandiant Automated Defense applies machine learning to help reduce the repetitive Tier 1 alert triage problem and address alert fatigue. 

Offering availability 

VirusTotal Code Insight, available now in Preview, is our first example of putting Security AI Workbench to work for our customers. We will be rolling out other offerings to trusted testers in coming months, and they will be available in Preview more broadly this summer. Click here for the demo

Security AI Workbench, including Sec-PaLM and partner integrations, in addition to the product innovations described in our demo, are all building blocks for a larger effort to elevate security across the ecosystem. So far, that effort: 

  • Provides assistive functions to rapidly develop IT generalist talent to Tier 1 security operator status in a way that wasn’t previously feasible. Security Command Center now can summarize threat intelligence insights and findings for Google Cloud, and Chronicle can quickly generate YARA-L rules or other detections. 
  • Provides advanced functions such as iterative query and multivariate detection generation, conversational filtering and interaction with results, and smart case awareness to empower advanced Tier 2 and 3 security operators to focus on threat analysis instead of struggling with process and toil. Mandiant Threat Intelligence users now can elevate their core competencies to hunt, investigate, and remediate threats — using the same tools our own Mandiant experts use. 
  • Fuses threat intelligence and AI-based analytic capabilities, which are unsurpassed in the market. VirusTotal Code Insight enables security teams to help gain insights and identify threats in suspicious code. This can significantly enhance their ability to detect and mitigate potential attacks.

However, this is just an initial step. We’ll continue to iterate and innovate, and we encourage customers and partners to leverage Security AI Workbench in new and exciting ways. Moving forward, we anticipate many new use cases to emerge over time. 

Building a safer future

While generative AI has recently captured the imagination, Sec-PaLM is based on years of foundational AI research by Google and DeepMind, and the deep expertise of our security teams. This work includes new efforts to expand our partner ecosystem to provide businesses with security capabilities at every layer of the cybersecurity stack. We have only just begun to realize the power of applying generative AI to security, and we look forward to continuing to leverage this expertise for our customers and drive advancements across the security community.

Case Study

This Diagnostic Company is Revolutionising Healthcare Delivery with AI

5398

Of your peers have already read this article.

5:30 Minutes

The most insightful time you'll spend today!

How do you shrink the time it takes to deliver MRI results from a minimum of 2 days to a mere 15 minutes?

Dr. Elliot Smith cannot be accused of lacking ambition. A high achiever with a Ph.D. in Electrical Engineering and a specialist in magnetic resonance imaging (MRI) systems, Smith aims to deliver top quality healthcare to anyone in the world — regardless of their location or wealth.

Dr. Smith has already made strides on this journey with his Brisbane, Queensland-headquartered business, Maxwell MRI. “I saw there was a big gap in the market around automating the diagnosis of health conditions,” he says. “Existing processes were typically manual and involved a lot of people.”

Artificial intelligence (AI) and machine learning can remove a key obstacle to scaling out medicine and improve the efficiency and accuracy of diagnosing conditions, the healthcare entrepreneur believes.

“Realistically a cloud product like GCP is the only way we can grow from an Australian-based company to a global company. If we had the burden of looking to set up our own infrastructure, it simply wouldn’t be feasible.”
-Dr. Elliot Smith, Founder and CTO, Maxwell MRI

“Our grand vision is to build an AI doctor that anyone can receive affordable support from and connect to in order to obtain results,” explains Dr. Smith.

Maxwell MRI presently enables clinicians to submit anonymised MRI scans to a machine learning enabled AI platform to help diagnose prostate cancer. The service is sold to clinicians who can then charge a per-session fee to clients. As well as obtaining results for individual cases, the MRI scans and associated information is used to ‘train’ the platform to deliver accurate diagnoses faster and in a more affordable way than existing systems do.

Dr. Smith and his team started by running a number of functions and processes on a single server with graphics processing units (GPUs) and sizable hard disk capacity. However, this infrastructure could not scale to support the planned growth of the business. Each case Maxwell MRI processes involves about 200MB of data in MRI scans alone. Once supplementary data, blood test result, pathology results and genetic information is included, this load can reach more than 1GB of data per patient.

The business aimed to process 150,000 cases by the end of 2018. This required a service that could deliver massive scale in data storage and compute, and could easily be accessed from any location. “We wanted to move from three GPUs to 30 GPUs without having to buy more servers or other associated equipment, so the cloud was the natural next step,” says Dr. Smith.

“We’re saying that with our platform running on GCP, we’ll deliver you results in 10 to 15 minutes, regardless of the number of patients coming in.”
-Dr Elliot Smith, Founder and CTO, Maxwell MRI

Maxwell MRI evaluated Google Cloud Platform (GCP) and determined that the managed services component of GCP would remove the burden of infrastructure deployment and administration. In addition, Google Cloud Machine Learning Engine would enable the business to scale to as many GPUs as needed to meet demand.

Maxwell MRI started with some small experiments to determine that GCP met all its requirements and completed its migration to the platform in February 2017. “We really started to scale up the data we had and consequently our computing requirements at that time,” Dr. Smith says.

The Maxwell MRI platform features an upload service that enables clinicians to upload imaging and associated data. This service triggers several different upload pipelines that clean and standardise data. They then write imaging data to Google Cloud Storage, and more structured data to a combination of Google Cloud Datastore and Google Cloud Spanner.

“We wanted to move from three GPUs to 30 GPUs without having to buy more servers or other associated equipment, so the cloud was the natural next step.”
-Dr. Elliot Smith, Founder and CTO, Maxwell MRI

The platform then converts the information into records that can be used to ‘train’ new machine learning configurations or run evaluations through existing machine learning pipelines.

“The tasks we perform including segmenting various anatomical regions for analysis and sending those results back into Google Cloud Storage,” says Dr. Smith. “This then commences that repeated process of running Google Cloud Dataflow pipelines and machine learning algorithms, and presenting those outcomes back to the clinicians.”

Existing Literature Validated

The data processed and analysed to date has, Dr Smith says, enabled Maxwell MRI to help validate existing literature that indicates clinicians lack confidence in existing early-stage testing procedures for prostate cancer. This prompts them to move quickly to the biopsy stage to assure themselves their diagnosis is valid. “New technologies have a lot of potential to rectify this situation and guide treatment to be more accurate, specific and cost-effective,” he says.

Results Delivered in 10-15 Minutes

More specifically, using GCP has enabled Maxwell MRI to guarantee to clinicians that results will be delivered within minutes. “Clinicians are used to getting results back in two days to a week,” says Dr. Smith. “We’re saying that with our platform running on GCP we’ll deliver you results in 10 to 15 minutes, regardless of the number of patients coming in.”

Running on GCP has enabled the business to accelerate its development cycles, test new ideas easily on a subset of data, test in parallel and deliver new services considerably faster than in another environment. In addition, the flexible GCP charging model aligned with the ability to scale compute capabilities quickly and easily has enabled the fledgling business to control its costs.

Google technologies are poised to play an integral role in the business’s future. “With Google available, it doesn’t make sense for us to use our own infrastructure,” Dr. Smith says. “Our expertise in AI, machine learning and clinical engagement complements cloud platform specialties of infrastructure, managed services and ease of use. We see a bright future ahead in helping to transform healthcare globally.”

More Relevant Stories for Your Company

How-to

Becoming Future-Proof: AI-Powered Integrated Business Planning on Google Cloud

If there’s one thing that the pandemic will leave us with it’s this: A deep desire to be better prepared. Not prepared for the next possible catastrophe because that’s hard to do, but better prepared for the repercussions of such disaster. In this video, Stephan de Barse, Executive Vice President,

Explainer

Google’s Lesson on Leveraging AI and More to Optimize Cloud Value for Innovation

Google has managed to stay ahead of the curve and demonstrate its 'growth while staying innovative' approach using advanced open technologies, AI/ML based analytics solutions, as well as tools for team collaboration and skill development, automation and storage security. Businesses too can achieve the same by deriving maximum value from

Explainer

Driving Business Transformation in Healthcare Using Google Cloud and AI/ML

Before the COVID-19 pandemic, when you thought of healthcare and AI, a number of ideas sprang to mind. But the world, especially as it relates to healthcare has seen a completely different type of transformation as a result of COVID-19. This video is about how Google Cloud is helping organizations

Blog

Custom AI Solutions, Global Delivery Centers and More Resources Dedicated for Customer Success

At Google Cloud, our customers are at the forefront of digital transformation—launching entirely new businesses and products built in the cloud, redefining entire industries with data and artificial intelligence, delivering innovative new consumer experiences, or committing to sustainable new ways of doing business. We’re committed to our customers’ success, and

SHOW MORE STORIES