June is the month that holds the summer solstice, and some of us in the northern hemisphere get to enjoy the longest days of sunshine out of the entire year. We used all the hours we could in June to deliver a flurry of new features across BigQuery, Dataflow, Data Fusion, and more.  Let’s take a look!

Simple, Sophisticated, and Secure

Usability is a key tenant of our data analytics development efforts. Our new user-friendly BigQuery improvements this month include:

• Flexible data type casting
• Formatting to change column descriptions 
• GRANT/REVOKE access control commands using SQL

We hope this will delight data analysts, data scientists, DBAs, and SQL-enthusiasts who can find out more details in our blog here.

Beyond simplifying commands, we also recognize that it’s equally important to have more sophistication when dealing with transactions. That’s why we introduced multi-statement transactions in BigQuery.

As you probably know, BigQuery has long supported single-statement transactions through DML statements, such as INSERT, UPDATE, DELETE, MERGE and TRUNCATE, applied to one table per transaction. With multi-statement transactions, you can now use multiple SQL statements, including DML, spanning multiple tables in a single transaction. 

This means that any data changes across multiple tables associated with all statements in a given transaction are committed atomically (all at once) if successful—or all rolled back atomically in the event of a failure. 

We also know that organizations need to control access to data, down to the granular level and that, with the complexity of data platforms increasing day by day, it’s become even more critical to identify and monitor who has access to sensitive data. 

To help address these needs,  we announced the general availability of BigQuery row-level security. This capability gives customers a way to control access to subsets of data in the same table for different groups of users. Row-level security in BigQuery enables different user personas access to subsets of data in the same table and can easily be created, updated, and dropped using DDL statements. To learn more, check out the documentation and best practices.

Simple, Safe, and Smart

Beyond building a simpler, more sophisticated and more secure data platform for customers, our team has been focused on providing solutions powered by built-in intelligence. One of our core beliefs is that for machine learning to be adopted and useful at scale, it must be easy to use and deploy.  

BigQuery ML, our embedded machine learning capabilities, have been adopted by 80% of our top customers around the globe and it has become a cornerstone of their data to value journey.  

As part of our efforts, we announced the general availability of AutoML tables in BigQuery ML.  This no-code solution lets customers automatically build and deploy state-of-the-art machine learning models on structured data. With easy integration with Vertex AI, AutoML in BQML makes it simple to achieve machine learning magic in the background. From preprocessing data to feature engineering and model tuning all the way to cross validation, AutoML will “automagically” select and ensemble models so everyone—even non-data scientists—can use it.   

Want to take this feature for a test drive? Try it today on BigQuery’s NYC Taxi public dataset following the instructions in this blog! 

Speaking of public datasets, we also introduced the availability of Google Trends data in BigQuery to enable customers to measure interest in a topic or search term across Google Search.  This new dataset will soon be available in Analytics Hub and will be anonymized, indexed, normalized, and aggregated prior to publication. 

Want to ensure your end-cap displays are relevant to your local audience?  You can take signals from what people are looking for in your market area to inform what items to place. Want to understand what new features could be incorporated into an existing product based on what people are searching for?  Terms that appear in these datasets could be an indicator of what you should be paying attention to.

All this data and technology can be put to use to deploy critical solutions to grow and protect your business. For example,  it can be difficult to know how to define anomalies during detection. If you have labeled data with known anomalies, then you can choose from a variety of supervised machine learning model types that are already supported in BigQuery ML. 

But what if you don’t know what kind of anomaly to expect, and you don’t have labeled data? Unlike typical predictive techniques that leverage supervised learning, organizations may need to be able to detect anomalies in the absence of labeled data. 

That’s why, we were particularly excited to announce the public preview of new anomaly detection capabilities in BigQuery ML that leverage unsupervised machine learning to help you detect anomalies without needing labeled data.  

Our team has been working with a large number of enterprises who leverage machine learning for better anomaly detection. In financial services for example, customers have used our technology to detect machine-learned anomalies in real-time foreign exchange data.  

To make it easier for you to take advantage of their best practices, we teamed up with Kasna to develop sample code, architecture guidance, and a data synthesizer that generates data so you can test these innovations right away. 

Simple, Scalable, and Speedy

Capturing, processing and analyzing data in motion has become an important component of our customer architecture choices. Along with batch processing, many of you need the flexibility to stream records into BigQuery so they can become available for query as they are written.  

Our new BigQuery Storage Write API combines the functionality of streaming ingestion and batch loading into a single API. You can use it to stream records into BigQuery or even batch process an arbitrarily large number of records and commit them in a single atomic operation.

Flexible systems that can do batch and real-time in the same environment is in our DNA: Dataflow, our serverless, data processing service for streaming and batch data was built with flexibility in mind.  

This principle applies not just to what Dataflow does but also how you can leverage it—whether you prefer using Dataflow SQL right from the BigQuery web UI, Vertex AI notebooks from the Dataflow interface, or the vast collection of pre-built templates to develop streaming pipelines.

Dataflow has been in the news quite a bit recently. You might have noted the recent introduction of Dataflow Prime, a new no-ops, auto-tuning functionality that optimizes resource utilization and further simplifies big data processing. You might have also read that Google Dataflow is a Leader in The 2021 Forrester Wave™: Streaming Analytics, giving Dataflow a score of 5 out of 5 across 12 different criteria.  

We couldn’t be more excited about the support the community has provided to this platform. The scalability of Dataflow is unparalleled and as you set your company up for more scale, more speed, and “streaming that screams”, we suggest you take a look at what leaders at Sky, RVU or Palo Alto Networks have already accomplished.

If you’re new to Dataflow, you’re in for a treat: this past month, Priyanka Vergadia (AKA CloudGirl) released a great set of resources to get you started. Read her blog here and watch her introduction video below!

https://youtube.com/watch?v=WRspZRG9e90%3Fenablejsapi%3D1%26

Simple structure that sticks together

We thrive to be the partner of choice for your transformation journey, regardless where your data comes from and how you choose to unify your data stack.  

Our partners at Tata Consultancy Services (TCS) recently released research that highlights the importance of a unifying digital fabric and how data integration services like Google Cloud Data Fusion can enable their clients to achieve this vision.

We also  announced SAP Integration with Cloud Data Fusion, Google Cloud’s native data integration platform, to seamlessly move data out of SAP Business Suite, SAP ERP and S4/HANA. To date, we provide more than 50 pipelines in Cloud Data Fusion to rapidly onboard SAP data.  

This past month, we introduced our SAP Accelerator for Order to Cash.  This accelerator is a sample implementation of the SAP Table Batch Source feature in Cloud Data Fusion and will help you get started with your end-to-end order to cash process and analytics. 

It includes sample Cloud Data Fusion pipelines that you can configure to connect to your SAP data source, perform transformations, store data in BigQuery, and set up analytics in Looker. It also comes with LookML dashboards which you can access on Github.

Countless great organizations have chosen to work with Google for their SAP data. In June, we wrote about ATB Financial’s journey and how the company uses data to better serve over 800,000 customers, save over CA$2.24 million in productivity, and realize more than CA$4 million in operating revenue through “D.E.E.P”, a data exposure enablement platform built around BigQuery.

Finally, if you are an application developer looking for a unified platform that brings together data from Firebase Crashlytics, Google Analytics, Cloud Firestore, and third party datasets, we have good news!  

This past month, we released a unified analytics platform that combines Firebase, BigQuery, Google Looker and FiveTran to easily integrate disparate data sources,  and infuse data into operational workflows for greater product development insights and increased customer experience. This resource comes with sample code, a reference guide and a great blog!  We hope you enjoy it. See you all next month!

https://youtube.com/watch?v=L25Vfzr2Ciw%3Fenablejsapi%3D1%26

Mindful Containers is a fictitious company that is creating containerized microservice applications. They need a fully managed compute environment for deploying and scaling serverless containerized microservices. So, they are considering Cloud Run. 

They are excited about Cloud Run because it abstracts away the cluster configuration, monitoring, and management so they can focus on building the features for their apps. Cloud Run is a fully-managed compute environment for deploying and scaling serverless containerized microservices.

What is Cloud Run?

Cloud Run is a fully-managed compute environment for deploying and scaling serverless HTTP containers without worrying about provisioning machines, configuring clusters, or autoscaling.

• No vendor lock-in – Because Cloud Run takes standard OCI containers and implements the standard Knative Serving API, you can easily port over your applications to on-premises or any other cloud environment. 
• Fast autoscaling – Microservices deployed in Cloud Run scale automatically based on the number of incoming requests, without you having to configure or manage a full-fledged Kubernetes cluster. Cloud Run scales to zero— that is, uses no resources—if there are no requests.
• Split traffic – Cloud Run enables you to split traffic between multiple revisions, so you can perform gradual rollouts such as canary deployments or blue/green deployments.
• Custom domains – You can set up custom domain mapping in Cloud Run and it will provision a TLS certificate for your domain. 
• Automatic redundancy – Cloud Run offers automatic redundancy so you don’t have to worry about creating multiple instances for high availability

How to use Cloud Run

With Cloud Run, you write your code in your favorite language and/or use a binary library of your choice. Then push it to Cloud Build to create a container build. With a single command—“gcloud run deploy”—you go from a container image to a fully managed web application that runs on a domain with a TLS certificate and auto-scales with requests.

How does Cloud Run work?

Cloud Run service can be invoked in the following ways:

HTTPS: You can send HTTPS requests to trigger a Cloud Run-hosted service. Note that all Cloud Run services have a stable HTTPS URL. Some use cases include: 

• Custom RESTful web API
• Private microservice
• HTTP middleware or reverse proxy for your web applications
• Prepackaged web application

gRPC: You can use gRPC to connect Cloud Run services with other services—for example, to provide simple, high-performance communication between internal microservices. gRPC is a good option when you: 

• Want to communicate between internal microservices
• Support high data loads (gRPC uses protocol buffers, which are up to seven times faster than REST calls)
• Need only a simple service definition you don’t want to write a full client library
• Use streaming gRPCs in your gRPC server to build more responsive applications and APIs

WebSockets: WebSockets applications are supported on Cloud Run with no additional configuration required. Potential use cases include any application that requires a streaming service, such as a chat application.

Trigger from Pub/Sub: You can use Pub/Sub to push messages to the endpoint of your Cloud Run service, where the messages are subsequently delivered to containers as HTTP requests. Possible use cases include:

• Transforming data after receiving an event upon a file upload to a Cloud Storage bucket
• Processing your Google Cloud operations suite logs with Cloud Run by exporting them to Pub/Sub
• Publishing and processing your own custom events from your Cloud Run services

Running services on a schedule: You can use Cloud Scheduler to securely trigger a Cloud Run service on a schedule. This is similar to using cron jobs. Possible use cases include:

• Performing backups on a regular basis
• Performing recurrent administration tasks, such as regenerating a sitemap or deleting old data, content, configurations, synchronizations, or revisions
• Generating bills or other documents

Executing asynchronous tasks: You can use Cloud Tasks to securely enqueue a task to be asynchronously processed by a Cloud Run service. Typical use cases include:

• Handling requests through unexpected production incidents
• Smoothing traffic spikes by delaying work that is not user-facing
• Reducing user response time by delegating slow background operations, such as database updates or batch processing, to be handled by another service, 
• Limiting the call rate to backend services like databases and third-party APIs

Events from Eventrac: You can trigger Cloud Run with events from more than 60 Google Cloud sources. For example:

• Use a Cloud Storage event (via Cloud Audit Logs) to trigger a data processing pipeline 
• Use a BigQuery event (via Cloud Audit Logs) to initiate downstream processing in Cloud Run each time a job is completed

How is Cloud Run different from Cloud Functions?

Cloud Run and Cloud Functions are both fully managed services that run on Google Cloud’s serverless infrastructure, auto-scale, and handle HTTP requests or events. They do, however, have some important differences:

• Cloud Functions lets you deploy snippets of code (functions) written in a limited set of programming languages, while Cloud Run lets you deploy container images using the programming language of your choice. 
• Cloud Run also supports the use of any tool or system library from your application; Cloud Functions does not let you use custom executables. 
• Cloud Run offers a longer request timeout duration of up to 60 minutes, while with Cloud Functions the requests timeout can be set as high as 9 mins. 
• Cloud Functions only sends one request at a time to each function instance, while by default Cloud Run is configured to send multiple concurrent requests on each container instance. This is helpful to improve latency and reduce costs if you’re expecting large volumes. 

Pricing

Cloud Run comes with a generous free tier and is pay per use, which means you only pay while a request is being handled on your container instance. If it is idle with no traffic, then you don’t pay anything.

Conclusion

After learning about the ease of set up, scalability, and management capabilities of Cloud Run the Mindful Containers team is using it to deploy stateless microservices. If you are interested in learning more, check out the documentation.https://www.youtube.com/embed/oR4btKLRdn4?enablejsapi=1&

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev

As one of the most trusted cloud platform providers, Google is committed to providing our clients secure and reliable environments for their workloads. Google believes the future of computing will increasingly shift to private, encrypted services where users can be confident that their data is not being exposed to cloud providers or their own insiders. Confidential Computing helps make this future possible by keeping data encrypted in memory, and elsewhere outside the CPU, while it is being processed.

Since our first Confidential Computing offering in 2018, Google has been a pioneer in making the technology widely available through our cloud, while also constantly adding new features that allow a wide range of organizations to benefit from it. 

Expanding the Confidential Computing portfolio

Today at the RSA Conference 2023, we are announcing several exciting updates to our Confidential Computing portfolio.

We are expanding our portfolio with the next generation of our Confidential Computing VM instances that utilize AMD Infinity Guard technology, specifically AMD SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging). SEV-SNP adds additional new hardware-based security protections such as strong memory integrity protection and hardware rooted attestation. Confidential VMs on general purpose N2D VMs with SEV-SNP enabled are now in Private Preview. Learn more in our deep dive here.

“Advanced, hardware-based security features – such as AMD Infinity Guard built into AMD EPYC processors – are a crucial part of cloud offerings and the further expansion of cloud computing,” said Ram Peddibhotla, corporate vice president, Cloud Business, AMD. “Our work with Google Cloud delivers on the leadership security features our joint customers demand with nearly effortless implementation and minimal performance impact as more mission critical and business critical workloads move to the cloud.”

Google is committed to ensuring Confidential Computing technology is as secure as possible before releasing products to customers. We constantly evaluate various attack vectors to make certain that Google Cloud Confidential Computing environments are protected against a wide range of attacks. 

Throughout last year, Google and Intel collaborated on a research project to identify potential security vulnerabilities in Intel’s new Confidential Computing technology, Trust Domain Extensions (TDX). We’ve recently released the full report and made it available in our blog here.

“We want to make it such that people don’t worry about the security and trustworthiness of their data,” said Anil Rao, vice president and general manager of systems architecture and engineering in the Office of the CTO at Intel. “Organizations use confidential computing to control their data and provide access to trusted parties in a manner that is verifiable, revocable and time sensitive – we have an obligation to make sure the technology is secure. Our early effort with Google solidifies our commitment to perform thorough analysis to address all potential vulnerabilities.”

Confidential Space is now in general availability. Confidential Space builds on Confidential Computing and provides a secure enclave, also known as a Trusted Execution Environment (TEE). Google Cloud customers can leverage the TEE for various privacy-focused use cases such as joint data analysis and machine learning (ML) model training, with trust guarantees that data can stay protected — including hardened protection against cloud service provider access. 

Confidential Space allows for new privacy-preserving technologies to take shape, and one such technology is The Privacy Sandbox. This is a crucial step forward as Privacy Sandbox for the Web will phase out third-party cookies and limit covert tracking. By supporting a choice of trusted execution environments, including Confidential Space, the Privacy Sandbox will provide ad-techs with safer alternatives to existing technology, so they can continue building digital businesses while keeping user data private.

“Trusted Execution Environments (TEEs) are already in use across industries such as digital advertising to enable privacy preserving analytics, and we are excited to leverage Google Cloud’s built-in TEE called Confidential Space,” said Gaurav Bhaya, vice president and general manager, Google Ads Measurement.

Building on our history of innovation with Confidential Computing

Google has worked diligently with internal and external partners to validate, audit and publish the security assessment of the hardware, firmware, and software that underpins Confidential Computing. 

Last year, a joint security effort between Google Project Zero, Google Cloud, and AMD led to more secure technology for the entire industry, regardless of what service provider you choose for your Confidential Computing deployment. You can learn more in our full report here.

Since then, we have continued to expand the portfolio of Confidential Computing products and services, including Confidential Virtual Machines (CVMs), Confidential GKE, Confidential Dataproc, and Confidential Space. These offerings give our customers more choice in their deployments. 

Currently, our Confidential Computing services are widely available across 80% of Google Cloud regions and growing, and we recently added support for Compute-Optimized C2D VMs for Confidential GKE.

Organizations are now leveraging Confidential Computing for joint data analysis and machine learning (ML) model training with trust guarantees that the data they own stays protected and across various other industries by organizations, including AstraZeneca, Bullish, HashiCorp, Matrixx Software, MonetaGo and Yellowdog.

Confidential Computing has also proven to be a helpful additional control for organizations implementing digital sovereignty strategies, providing an encryption capability, and protection for data-in-use where encryption keys are not accessible by the cloud provider.

We can’t wait to see the possibilities this technology will open up for your organization. Check out our Confidential Computing page to learn more.

Cloud transformation and the adoption of modern DevOps technology presents both opportunities and challenges for IT compliance functions. With DevOps style application development, the feedback loop for developers and engineers is much tighter than with traditional application development pipelines, enabling speed and agility of application release cycles. While speedy CI/CD is a critical advantage of DevOps, it also shifts compliance left in the development timeline, and therefore puts pressure on the IT risk & compliance organization to modernize their approach to regulatory compliance as well. With the ongoing shift towards cloud technologies and DevOps, modernization of regulatory compliance is no longer optional for an IT compliance function

Compliance modernization is a broad mandate that spans the way the function is governed; the tools, technology, and analytics it uses; the number and nature of its connections to other parts of the business; verifiability and auditability of the controls’ evidence, the expectations assigned to it; and more.

Public cloud technology is becoming a core part of many industries today, and with this comes some potential risks such as cloud misconfigurations exposing intellectual property, loss of physical control of assets, skillset scarcity around cloud based security and compliance. 

Given the constantly changing risk landscape, it is critical that regulations more closely align to address these risks. As regulations and risks evolve, the aim of a modern compliance function is to help an organization stay compliant as it goes through a digital transformation. As organizations go through digital transformation, IT compliance also needs to transform — via upgrading the technology stack, modifying the business processes and most importantly re-skilling people to become cloud aware.

Today we are releasing the new paper by Google Cloud’s Office of the CISO. In the paper we reveal a new approach for modernizing your compliance approach using modern approaches and Google Cloud toolsets. Your team can leverage the paper to add value to enterprises, both by charting a course to the safe use of cloud technology and by reducing risk through the use of the public cloud.

Read the paper “Assuring Compliance in the Cloud.”

Also, review these related resources:

• “Risk Governance of Digital Transformation in the Cloud“ paper
• “Making Compliance Cloud-native” (episode 14) with Zeal Somani
• Google Cloud Compliance resource center
• Our compliance blueprints: PCI DSS on GKE and GCP FedRAMP Blueprint
• Google Cloud security best practices center

With unemployment application submissions reaching record numbers over the past year, state and local agencies in the United States have faced the challenge of processing unprecedented numbers of claims per week. The digital infrastructure most agencies have in place is unable to handle this volume, resulting in constituents waiting longer, and bad actors taking advantage of vulnerable systems. The Department of Labor Inspector General estimates that $63 billion in claims distributed is either an improper payment or fraud. 

Validating claims also requires secure data sharing with other agencies for document and identity verification. Government leaders need a way to allow case adjudicators to quickly and confidently release backlogged claims, integrate with existing systems, and segment legitimate claims from potentially fraudulent ones — all within limited government budgets — securely and at scale.

Implementing a fraud detection solution on Google Cloud

States were under pressure to release payments, while also filtering out potentially fraudulent claims. SpringML and Google Cloud developed a framework to give adjudicators a reliable verification process that quickly filters potentially fraudulent claims, while processing the remaining claims so benefits reach citizens in a timely manner. SpringML and Google Cloud, applied AI-powered machine learning models to detect anomalous patterns in large datasets. Using Google Cloud tools, SpringML implemented a solution to streamline workflows, improve efficiencies, automate processes and identify potentially fraudulent claims.

SpringML used a variety of Google Cloud products to deliver a fraud detection solution, including:

• Google Cloud Storage to store and manage data
• BigQuery to store tabular data and BigQuery Machine Learning (BQML) to conduct machine learning on that data
• AutoML solutions to build predictive models and risk scoring
• Visualization tools such as Looker and Data Studio to present data and help government leaders make informed decisions.

Implementing machine learning to detect improper payments allows agencies to classify claims as “fraud” or “not fraud” based on the number of flags, as well as prioritize the most urgent claims. Deploying intelligent virtual agents to handle frequently asked questions meant that live agents could focus their time on more challenging cases. 

Even once the pandemic is behind us, there will be bad actors trying to take advantage of overwhelmed or legacy systems. We’ve identified a few best practices for agencies managing enormous case loads and looking to improve improper payment analytics: 

• Move your systems to the cloud. Many on-premises legacy systems can’t update their applications and scale to meet the volume of claims. Moving to a cloud environment enables rapid solution deployment and ingestion of large amounts of data without fear of overloading the system. The cloud scales with you–cost-effectively and securely. 
• Understand patterns in the data. The answer is always in the data — we used deep analysis to help uncover suspicious patterns in large data sets. We implemented unsupervised machine learning to learn behaviors and create configurable rules that adjust to new information that comes into the system. We can uncover patterns that are likely associated with fraud – ones that a human might have missed. 
• Use AI/ML tools to automate your existing systems and teams. These tools enable humans to work smarter and more efficiently. We automate anomaly detection and create dashboards for adjudicators to rapidly process claims. We are enabling the Wisconsin Department of Workforce Development by implementing automatic calculations and processing of recharge amounts, resulting in faster processing times and fewer human errors. Proactive fraud detection and timely calculation of recharge payment allowed DWD to ensure the benefits reached the right individuals.
• Build flexibility into your systems. We discovered that fraud patterns change over time. For instance,flags for fraud during March-May 2020 were vastly different from those we found in June-July 2020. Google Cloud tools make it easy to continually update algorithms to detect patterns and integrate external data sources.

Using Google Cloud tools, we can update digital infrastructure and incorporate machine learning best practices to help organizations efficiently process large volumes of claims and identify high probability fraudulent ones. SpringML provides consulting and implementation services and industry-specific analytics solutions that deliver high-impact business value to accelerate data-driven digital transformation. Learn more about fraud detection and how to improve improper payments analytics by watching our webinar.