In every industry, in every part of the world, cybersecurity concerns continue to grow in the wake of attacks on critical infrastructure and the software supply chain. Governments and businesses of all sizes recognize that they must do more to protect their employees, customers and citizens. 

But doing more of the same, like putting security band-aids on legacy infrastructure, is no longer helpful or productive. We need an enduring commitment, in products, people and monetary terms, to drive meaningful improvements in our collective security posture. Google recently announced a $10 billion investment to advance the security of governments around the world, and by extension, help enterprises and organizations to do the same. 

Over the past year, Google Cloud has been delivering on our vision of Invisible Security for our customers, where capabilities are continuously engineered into both our trusted cloud platform and market-leading products to bring the best of Google’s security to wherever your IT assets are. But being a provider of best-in-class technology is not enough. We want to share our expertise to help organizations with their security transformation and offer even more ways to accelerate these essential improvements. This week, at Google Cloud Next ‘21, we’re sharing the first of what will be many more steps in advancing these efforts.

Introducing our Cybersecurity Action Team

While access to the latest, most advanced security technology is important, the expertise of what it will take to become resilient in the face of today’s risk and threat environment is foundational.  

Today, we’re announcing the formation of the Google Cybersecurity Action Team. The Google Cybersecurity Action Team marshals experts from across Google to form what we believe will be the world’s premier security advisory team. It has a singular mission to support the security and digital transformation of governments, critical infrastructure, enterprises and small businesses.

Building on existing security solutions engineering efforts, today the Google Cybersecurity Action Team announced a security and resilience framework that delivers a roadmap for a comprehensive security management program aligned with the National Institute of Standards and Technology’s Cybersecurity Framework using cloud technologies from Google Cloud and our partners. 

“Google Cloud has been a critical partner in the BBVA security journey, helping us protect our customers’ sensitive and proprietary data with modern frameworks like zero trust and secure-by-default products like Google Workspace,” said Alvaro Garrido, Chief Security Officer at BBVA. “We look forward to the strategic services and guidance the Google Cybersecurity Action Team will deliver as we continue on our security transformation.”

Learn more about the Cybersecurity Action Team here. 

Announcing a safer way to work

Bringing you the expertise and hands-on guidance to help with your security transformation is just one valuable step. We recognize that too many organizations can’t wait any longer to begin their modernization efforts – they need a new baseline, and they need it now. 

That’s why today, we’re announcing the launch of our Google Work Safer offering, designed to help organizations, their employees, and partners collaborate and communicate securely and privately in today’s hybrid work environment. Work Safer provides companies with access to best-in-class security for email, meetings, messages, documents, and more. It uniquely brings together the cloud-native, zero-trust solutions of Google Workspace with BeyondCorp Enterprise for secure access with integrated threat and data protection. For customers who want secure devices, Work Safer includes Pixel phones managed with Android Enterprise, Chrome Enterprise Upgrade, and HP Chromebooks. Customers can also leverage Google’s Titan Security Keys for account protection, reCAPTCHA Enterprise for website fraud prevention, Chronicle for security analytics, and a variety of migration services for a seamless transition.

The program is designed to meet the needs of all organizations, including small businesses, enterprises and public sector institutions, many of which are reliant on legacy technology and often lack expertise to fully address rising security challenges associated with hybrid work. To learn more, visit our Work Safer homepage. 

Advancing our trusted cloud with new security capabilities

Above all, security has been and continues to be the cornerstone of our product strategy. There are three critical areas where Google Cloud’s capabilities can make a meaningful difference for any business’ or government’s digital security transformation:

At Next’ 21, we’re introducing new security products and partnerships that will enable you to:

• Protect your employees with new zero trust access capabilities: We’re delivering new features that expand the surface area for our zero trust access solution, BeyondCorp Enterprise, to cover all your apps – both modern and legacy. The new client connector, now in preview, enables identity and context-aware access to non-web applications running in Google Cloud and non-Google Cloud environments. We are also making it easier for admins to diagnose access failure, triage events, and unblock users with the new Policy Troubleshooter feature. You can learn more about both of these new enhancements in the live BeyondCorp Enterprise demo on October 13.
• Improve your detection and response capabilities: We announced a new collaboration with Cybereason for Extended Detection and Response (XDR) across endpoints, networks, cloud and workspaces. The combination of these capabilities delivers a cloud-native XDR solution, Cybereason XDR powered by Chronicle, that automates prevention for common attacks, guides analysts through security operations and incident response, and enables threat hunting with precision at a pace never before achieved. We are also deepening the integration between Chronicle and Security Command Center (SCC) on GCP. New integrations in preview centralize alerts and investigative workflows across the two platforms, and enable threat-specific pivots by enriching SCC alerts with intelligence on associated threat actors and entities. 
• Automate and bolster protection of your sensitive data: Automatic DLP, now in preview, is a prime example of how we are making Invisible Security a reality. It’s a game-changing capability that discovers and classifies sensitive data for all the BigQuery projects across your entire organization without you needing to do a single thing. We’re also introducing Ubiquitous Data Encryption, a solution which combines our generally available Confidential Computing, External Key Management, and Cloud Storage products to seamlessly encrypt data as it’s sent to the cloud. Using your External Key Management solution, data can now only be decrypted and run in a confidential VM environment, greatly limiting potential exposure.
• Protect your IP and implement a zero trust software supply chain: Today, we’re building on our zero trust software supply chain with new launches. Cloud Build is SLSA Level -1 compliant by default, with scripted builds and available provenance. With the new Build Integrity feature, Cloud Build automatically generates a verifiable build manifest that includes a signed certificate describing the sources that went into the build, the hashes of artifacts used, and other parameters. Additionally, Binary Authorization’s integration with Cloud Build makes it easy to set up deploy-time constraints. You can also now easily pair Binary Authorization with Cloud Run to ensure only trusted images make it to production. These integrations are now generally available.
• Protect your users and brand: We recently announced the preview of Cloud Armor Bot Management, which integrates Cloud Armor and reCAPTCHA Enterprise. You can enable protection without any server-side changes to your applications, and because detection and enforcement happens in-line, at the edge of Google’s network, you can mitigate threats before they have a chance to impact your applications, whether they run on GCP, on premise, or in a hybrid or multicloud deployment.
• Ensure secure collaboration: Today, Google Workspace is also introducing new security features. Client-side encryption for Google Meet, in beta, gives customers direct control of encryption keys and the identity service used to access keys. Data Loss Prevention (DLP) for Google Chat, in beta, helps prevent sensitive information from leaking outside of your organization. Check out the blog post from Google Workspace to learn more. 

As we head into three days of great content and engagements from our security and technology teams across Google Cloud at Next ‘21, we want to leave you with this: You are not alone on your security journey. Our goal is to ensure that every day, you are making your organization safer because you have partnered with us. Please make sure to watch our security experts, customers, and partners in our track sessions to go deep on the topics and products that matter most to you.

Cloud-based solutions were a technology life raft for organizations during the COVID-19 pandemic as employees took to the virtual office and companies scrambled to adjust to a distributed, remote reality. However, these rapid and substantial changes in the role of cloud technologies on the business came with an increased focus on security. 

The accelerated move to the cloud also meant companies needed to rapidly evolve existing security practices to protect everything that matters at the core of business—from their people and their operational and transactional data to customers and their most sensitive personal information. Suddenly, enterprises were keenly aware of where business practices, employee training, and security policies were falling short. 

A recent Google-commissioned study by IDG explored the details behind the heightened focus on security solutions since the start of the pandemic while highlighting the role cloud-based security solutions are playing in helping keep customers safe. The survey of 2,000 global IT leaders serves to illustrate that in this new and unfamiliar world, enterprises are more ready than ever to embrace cloud security.LEARN MOREIT Leaders Research ReportWhat are the biggest IT priorities in 2021? Explore insights from a survey of 2,000 Senior IT leaders conducted by IDG, in collaboration with Google Cloud.

Security is an even higher priority post-pandemic

In the wake of the pandemic, many organizations are facing a broader attack surface than ever before as employees moved to temporarily working from remote home offices (and in some cases, encouraged to stay there for the foreseeable future). With fewer inherent security protections on personal internet connections and more work meetings happening via video conferencing, attackers have launched a cyber pandemic of their own designed to take advantage of and exploit new weaknesses. 

However, even as businesses amp up security initiatives and preventative measures, the growing wave of threats continue to keep security top of mind for IT leaders. Security risks and concerns remain one of the top pain points impeding innovation according to the IDG study respondents—only surpassed by insufficient IT and developer skills. 

Enterprises looking to cloud providers for help with security

As a result, addressing security risks is a leading area where IT leaders turn to cloud providers for support. For these organizations, the ability to control access to data while using cloud services was the most required infrastructure security and compliance features from a cloud provider.

Cloud security is more trusted than ever 

A deeper look into the results also revealed a shift in perspective about whether cloud security is really up to the task of protecting enterprises against modern attacks. Despite skepticism in the past, the majority of IT leaders are now comfortable with using cloud-based security solutions. 

Confidence in the security of cloud infrastructure is extremely high with 85% of respondents stating they feel secure (or more secure) than on-premises infrastructure—compared to just 15% who believe on-premises is still safer.

This is a clear indication that there are fewer reservations around the efficacy of cloud-based security solutions, signaling an increase in trust as organizations invest in cloud-based infrastructure and solutions. 

We are committed to safe, secure solutions 

Google Cloud protects your data, applications, and infrastructure, as well as your customers, from fraudulent activity, spam, and other types of online abuse. We protect you against a growing list of cybersecurity threats using the same infrastructure foundation and security services that we use for our own operations, so you never have to compromise between ease of use and advanced security. 

To learn more about the IDG findings and how IT leaders are addressing security concerns post-COVID, download the full report.

^{Interested in how Google Cloud’s commitment to providing safe, secure solutions helps you address your security needs?}

^{Our networking, data storage, and compute services encrypt data at rest and in transit to ensure the integrity, authenticity, and privacy of your data and your customers’ data. We also offer the ability to encrypt data in use, while it’s being processed in VM and container workloads, and our advanced security tools support compliance and data confidentiality with minimal operational overhead.}

The financial services industry is changing—an estimated $68 trillion in wealth transferring from baby boomers to millennials.¹ This means financial service providers will have to deliver the speed, ease-of-use, technological sophistication, and tailored services that millennials have come to expect. In fact, half of all millennials are willing to switch to a competing institution if it offers a better digital experience.² This and many other trends are driving unprecedented growth for mobile Fintech experiences in banking, digital payments, financial management and insurance.³

Google Maps Platform financial services solutions

To help you respond to customer’s changing demands, we’re launching financial services solutions that can help you improve your customer experience, security and operations. We’ve outlined the technical guidance and APIs you need to build out three financial services solutions: Enriched Transactions, Quick and Verified Sign-up, and Branch and ATM Locator Plus. We’ve also highlighted two use cases that customers are using our APIs to solve: Contextual Experiences and Fraud Detection. 

Clarify financial statements with Enriched Transactions solution

Transaction statements are often hard for customers to understand, using abbreviations like “ACMEHCORP” instead of customer-facing names like “Acme Houseware”. Our Enriched Transactions solution clarifies these transactions and makes them instantly recognizable by adding the merchant name and business category, a photo of the storefront, its location on a map, and full contact info. Making transactions easier to recognize not only boosts consumer confidence, with reported increases in NPS of 15% or higher, but decreases costly support calls by approximately 67%.⁴

In addition, you can help customers easily visualize a series of transactions by adding the merchant name to the transaction amount and date, and displaying their transactions on a Google map. This enables you to give customers insights about where and how they spend money. See the guide to implement Enriched Transactions today.

Enable faster sign-up with Quick and Verified Sign-up solution

Manually entered addresses can lead to lowered conversions, erroneous customer data, and costly delivery mistakes. Our Quick and Verified Sign-up solution makes sign-up faster, suggesting nearby addresses with just a few thumb taps—cutting sign-up time by up to 64% and increasing conversion rates by up to 15%.⁵  

The solution also provides one additional level of address verification that helps reduce the risk of fraudulent account sign-ups—and companies have decreased fraudulent account setups by approximately 30% through using geospatial data to verify customer identities.⁴  See the Quick and Verified Sign-up solution guide to get started today.

• Faster sign-ups 1An application form requires an address
• Faster sign-ups 2Autocomplete quickly suggests addresses
• Faster sign-ups 3Select the address with visual confirmation
• Faster sign-ups 4Address verification options are presented
• Faster sign-ups 5Location permission is granted by the user
• Faster sign-ups 6The address is verified

Help customers visit you with Branch and ATM Locator Plus solution

74% of customers now search for specific details prior to their visit, which makes detailed, accurate profiles for each location a must.⁵ Our Branch and ATM Locator Plus solution enhances your own websites and apps with the same information shown about your branches and ATMs on Google Maps. Include hours of operation, available services, user reviews, photos of the location, driving directions and more.

Financial services companies using geospatial data to provide additional information (e.g. opening hours, available services, etc.) on branch and ATM services have seen a 14% increase in Net Promoter Score (NPS), and a 7% decrease in customer support calls.⁴  Implement Branch and ATM Locator Plus today using the guide or build it in minutes with Quick Builder.

• ATM locator 1Customers can enable location permissions, or enter their address
• ATM locator 2Quickly enter the address with Autocomplete
• ATM locator 3Nearby location listings, ranked by distance and ETA
• ATM locator 4Map view and directions

Enable offers and rewards with Contextual Experiences                    

Real-time, geo-targeted offers can power deals, rewards, and cash-back programs—all visualized with rich Google Maps. By combining the insights of purchase histories with customer opt-in to location-based features, companies can implement the Contextual Experiences use case to enable personalized offers and rewards programs that drive engagement with brands while putting money in customers’ pockets at the same time.This is a win for banks and their customers, validated by encouraging metrics like NPS rating boosts of 8% or higher, and an increase of 8% or more time spent in-app.⁴  Learn how Current uses Google Maps Platform to create innovative customer rewards programs with location intelligence.

Detect suspicious transactions with Fraud Detection

With the Fraud Detection use case, companies can use customer opted-in mobile device location to flag suspicious activity based on geographic distance, such as an ATM withdrawal that is far from the customer’s phone. Our APIs can also help companies recognize suspicious transaction patterns such as a purchase made at a location that is physically distant from a recent transaction. 

Financial services companies that use geospatial data to verify customers’ identities have reduced fraudulent transactions by approximately 70%, and false positives in fraud detection by 45%, on average.⁴  Learn how Starling Bank uses Google Maps Platform to enable real-time notification of transactions and their locations, and enhance data-driven decision-making.

Start elevating customer experiences, reducing risk and increasing efficiency today with our financial services offerings. Visit our financial services solutions page to learn more about how to start implementing these solutions.

For more information on Google Maps Platform, visit our website.

In fast-moving organizations, it’s not uncommon for cloud resources, including entire projects, to occasionally be forgotten about. Not only such unattended resources can be difficult to identify, but they also tend to create a lot of headaches for product teams down the road, including unnecessary waste and security risks. 

To help you prune your idle cloud resources, we’re excited to introduce Unattended Project Recommender. It’s a new feature of Active Assist that provides you with a one-stop shop for discovering, reclaiming, and shutting down unattended projects. With actionable and automatic recommendations, you no longer have to worry about wasting money or mitigating security risks presented by your idle resources. Unattended Project Recommender uses machine learning to identify, with a high degree of confidence, projects that are likely abandoned based on API and networking activity, billing, usage of cloud services, and other signals. This feature is available via the Recommender API today, making it easy for you to integrate with your company’s existing workflow management and communication tools, or export results to a BigQuery table for custom analysis.

Thousands of projects can be unattended in large organizations, presenting major security risks

Your cloud projects can go abandoned or unattended for a number of reasons — ranging from a test environment that’s no longer needed, to project cancellation, to project owner switching jobs, and more. Not only can such projects contribute to your cloud bill (waste) but they may contain security issues such as open firewalls or privileged service account keys that attackers can exploit to get a hold of your cloud resources for cryptocurrency mining or, worse, compromise your company’s sensitive data. These security risks tend to grow over time because the latest best practices and patches are usually not applied to unattended projects. 

We experience this issue here at Google, too… In fact, it has been on Google’s internal security team’s radar for some time now, so we joined forces and looked into this problem together, starting with our very own “google.com” organization cloud projects. We quickly found some projects that were unattended, but remediating this issue was easier said than done due to challenges in several areas:

• Detection: With lots of signals available to you via sources like Cloud Monitoring, what are the right ones you should look at (e.g. API, networking, user activity)? How can you tell the difference between an unattended project and a project that has a low level of activity by design (e.g. a “shell” project that holds an auth token)?
• Remediation: Once you have identified a project that seems abandoned, how do you go about ensuring that it’s indeed an unattended project? How do you reduce the risk of deleting something that might be essential to a production workload, causing irreversible data loss? How do you solve this at the scale of your entire organization, beyond a one-time cleanup? 

Over the course of 2021 we built and tested a Google-internal prototype first, cleaning up many of our internal unattended projects, and then worked with a number of Google Cloud customers to build and tune this feature based on real-life data (thank you to all of our early adopters for working with us and your generous feedback that helped us shape this feature!) It was not uncommon for us to come across organizations with thousands of unattended projects, and we’re very excited to bring Unattended Project Recommender to all customers, in public preview.

Discovering and acting on unattended project recommendations

Unattended Project Recommender analyzes usage activity across all projects under your organization, including the following data:

• API activity (e.g. service accounts with authentication activity, API calls consumed)
• Networking activity (ingress and egress)
• Billing activity (e.g. services with billable usage)
• User activity (e.g. active project owners)
• Cloud services usage (e.g. number of active VMs, BigQuery jobs, storage requests)

Based on these signals, it can generate recommendations to clean up projects that have low usage activity (where “low usage” is defined using a machine learning model that ranks projects in your organization by level of usage), or recommendations to reclaim projects that have high usage activity but no active project owners. Here’s what an example post-processed summary list of recommendations can look like for the “foobar” organization that has 3 projects:

  Project ID: demo-project-307815
Recommendation: CLEANUP_PROJECT

Project ID: new-project
Recommendation: N/A

Project ID: bobs-playground-project
Recommendation: RECLAIM_PROJECT

In addition to the recommendations, you can also examine the underlying project activity insights that the recommendations are based upon. The insights provide additional information that can be useful for integration with your organization’s existing workflows and automation (e.g. send an auto-generated email or chat message to project owners based on the list provided by the owners field). Here’s an example insight payload:

  content:
  activeAppengineInstanceDailyCount: 0
  activeCloudsqlInstanceDailyCount: 0
  activeGceInstanceDailyCount: 3
  activeServiceAccountDailyCount: 1
  apiClientDailyCount: 18922           // Daily average API calls produced
  bigqueryInflightJobDailyCount: 0
  bigqueryInflightQueryDailyCount: 0
  bigqueryStorageDailyBytes: 0
  bigqueryTableDailyCount: 0
  consumedApiDailyCount: 0             // Daily average API calls consumed
  datastoreApiDailyCount: 0
  gcsObjectDailyCount: 11
  gcsRequestDailyCount: 0
  gcsStorageDailyBytes: 2663548
  hasActiveOauthTokens: false          // OAuth tokens used in the last 180 days
  hasBillingAccount: true
  numActiveUserOwners: 1
  owners:                              // List of project owners
  - activeOnProject: false
    member: user:user1@example.com
  - activeOnProject: true
    member: user:user2@example.com
  serviceWithBillableUsage:
  – Cloud Storage
  - Compute Engine
  vpcEgressDailyBytes: 264456938       // Daily average VPC egress bytes
  vpcIngressDailyBytes: 392435047      // Daily average VPC ingress bytes
  usagePercentile: 20                  // Level of usage relative to other projects

GCP projects are used in many different ways and for many different purposes. In case you get a recommendation to delete a project that’s being used in a way that’s out of the scope for this feature, you can dismiss the recommendation and it will stop showing up for the given project. 

Restoring deleted projects

When you choose to shut down a project using the projects.delete() method, it gets marked for deletion. After a project is marked for deletion, it becomes unusable, all resources within that project are shut down, and a 30-day wait period for the project and all of its data to get fully deleted begins.

In case a useful project is accidentally shut down, you have the option to restore the project within that 30-day wait period. Since restoring allows you to recover most but not necessarily all of your project data and resources, we recommend carefully examining the utilization insights associated with a project and considering any additional utilization signals that may not be captured by the Unattended Project Recommender before taking the cleanup action.

Early customer success stories

A number of enterprise customers are already using Unattended Project Recommender to keep their organizations clean of unattended projects and resources.

Decathlon, a French sporting goods retailer, is excited for the insight Unattended Project Recommender will bring to their environment, and are already deploying it as a part of their latest cloud security initiatives.

“After a thorough test of this feature and the validation of our CISO, we ended up deleting our first 775 projects, and no one complained! A great help to improve our security. The next step for us will be to operationalize it at scale, and implement a company wide policy for unattended resource management.” —Adeline Villette, Cloud Security Officer

For Veolia, one of the world’s largest water, waste and energy management companies, not only does this feature reduce security risks and waste, but also helps drive cultural shift and alignment with its ecological transformation strategy.

“This feature allows us to reduce our costs and security debt on assets that are no longer in use, and is also fully in line with Veolia’s philosophy of limiting its carbon footprint. After having tested Unattended Project Recommender on more than 3,000 projects throughout our organization, we are looking to bring it as proactive alerts to our project owners at scale.”—Thomas Meriadec, Product Manager

Box, a secure cloud content management provider, views it as a foundation for building a repeatable process to remediate unused resources.

“Unattended Project Recommender is a great fit for us. It gives us a unified view of project usage across our entire organization and enables us to address security risks of legacy projects in a systematic and organized manner, ensuring an even safer environment.” —Matt Bowes, Staff Security Engineer

Getting started with the Unattended Project Recommender

To help you get started, we’ve prepared a Cloud Shell tutorial (source code) that you can use to find unattended project recommendations within your own Projects/Folders/Organization. Click this button to clone the tutorial from GitHub and run in your Cloud Shell environment:

As you can see, listing recommendations for your projects only takes a few clicks with the tutorial (special thanks to Lanre Ogunmola, Security & Compliance Specialist, for making this look so easy)! For additional detail on using the gcloud CLI or API to discover unattended project recommendations, please refer to the documentation page.

You can also automatically export all recommendations from your Organization to BigQuery and then investigate the recommendations with DataStudio or Looker, or use Connected Sheets that let you use Google Workspace Sheets to interact with the data stored in BigQuery without having to write SQL queries.

As with any other Recommender, you can choose to opt out of data processing at any time by disabling the appropriate data groups in the Transparency & control tab under Privacy & Security settings.

We hope that you can leverage Unattended Project Recommender to improve your cloud security posture and reduce cost, and can’t wait to hear your feedback and thoughts about this feature! Please feel free to reach us at active-assist-feedback@google.com and we also invite you to sign up for our Active Assist Trusted Tester Group if you would like to get early access to the newest features as they are developed.

June is the month that holds the summer solstice, and some of us in the northern hemisphere get to enjoy the longest days of sunshine out of the entire year. We used all the hours we could in June to deliver a flurry of new features across BigQuery, Dataflow, Data Fusion, and more.  Let’s take a look!

Simple, Sophisticated, and Secure

Usability is a key tenant of our data analytics development efforts. Our new user-friendly BigQuery improvements this month include:

• Flexible data type casting
• Formatting to change column descriptions 
• GRANT/REVOKE access control commands using SQL

We hope this will delight data analysts, data scientists, DBAs, and SQL-enthusiasts who can find out more details in our blog here.

Beyond simplifying commands, we also recognize that it’s equally important to have more sophistication when dealing with transactions. That’s why we introduced multi-statement transactions in BigQuery.

As you probably know, BigQuery has long supported single-statement transactions through DML statements, such as INSERT, UPDATE, DELETE, MERGE and TRUNCATE, applied to one table per transaction. With multi-statement transactions, you can now use multiple SQL statements, including DML, spanning multiple tables in a single transaction. 

This means that any data changes across multiple tables associated with all statements in a given transaction are committed atomically (all at once) if successful—or all rolled back atomically in the event of a failure. 

We also know that organizations need to control access to data, down to the granular level and that, with the complexity of data platforms increasing day by day, it’s become even more critical to identify and monitor who has access to sensitive data. 

To help address these needs,  we announced the general availability of BigQuery row-level security. This capability gives customers a way to control access to subsets of data in the same table for different groups of users. Row-level security in BigQuery enables different user personas access to subsets of data in the same table and can easily be created, updated, and dropped using DDL statements. To learn more, check out the documentation and best practices.

Simple, Safe, and Smart

Beyond building a simpler, more sophisticated and more secure data platform for customers, our team has been focused on providing solutions powered by built-in intelligence. One of our core beliefs is that for machine learning to be adopted and useful at scale, it must be easy to use and deploy.  

BigQuery ML, our embedded machine learning capabilities, have been adopted by 80% of our top customers around the globe and it has become a cornerstone of their data to value journey.  

As part of our efforts, we announced the general availability of AutoML tables in BigQuery ML.  This no-code solution lets customers automatically build and deploy state-of-the-art machine learning models on structured data. With easy integration with Vertex AI, AutoML in BQML makes it simple to achieve machine learning magic in the background. From preprocessing data to feature engineering and model tuning all the way to cross validation, AutoML will “automagically” select and ensemble models so everyone—even non-data scientists—can use it.   

Want to take this feature for a test drive? Try it today on BigQuery’s NYC Taxi public dataset following the instructions in this blog! 

Speaking of public datasets, we also introduced the availability of Google Trends data in BigQuery to enable customers to measure interest in a topic or search term across Google Search.  This new dataset will soon be available in Analytics Hub and will be anonymized, indexed, normalized, and aggregated prior to publication. 

Want to ensure your end-cap displays are relevant to your local audience?  You can take signals from what people are looking for in your market area to inform what items to place. Want to understand what new features could be incorporated into an existing product based on what people are searching for?  Terms that appear in these datasets could be an indicator of what you should be paying attention to.

All this data and technology can be put to use to deploy critical solutions to grow and protect your business. For example,  it can be difficult to know how to define anomalies during detection. If you have labeled data with known anomalies, then you can choose from a variety of supervised machine learning model types that are already supported in BigQuery ML. 

But what if you don’t know what kind of anomaly to expect, and you don’t have labeled data? Unlike typical predictive techniques that leverage supervised learning, organizations may need to be able to detect anomalies in the absence of labeled data. 

That’s why, we were particularly excited to announce the public preview of new anomaly detection capabilities in BigQuery ML that leverage unsupervised machine learning to help you detect anomalies without needing labeled data.  

Our team has been working with a large number of enterprises who leverage machine learning for better anomaly detection. In financial services for example, customers have used our technology to detect machine-learned anomalies in real-time foreign exchange data.  

To make it easier for you to take advantage of their best practices, we teamed up with Kasna to develop sample code, architecture guidance, and a data synthesizer that generates data so you can test these innovations right away. 

Simple, Scalable, and Speedy

Capturing, processing and analyzing data in motion has become an important component of our customer architecture choices. Along with batch processing, many of you need the flexibility to stream records into BigQuery so they can become available for query as they are written.  

Our new BigQuery Storage Write API combines the functionality of streaming ingestion and batch loading into a single API. You can use it to stream records into BigQuery or even batch process an arbitrarily large number of records and commit them in a single atomic operation.

Flexible systems that can do batch and real-time in the same environment is in our DNA: Dataflow, our serverless, data processing service for streaming and batch data was built with flexibility in mind.  

This principle applies not just to what Dataflow does but also how you can leverage it—whether you prefer using Dataflow SQL right from the BigQuery web UI, Vertex AI notebooks from the Dataflow interface, or the vast collection of pre-built templates to develop streaming pipelines.

Dataflow has been in the news quite a bit recently. You might have noted the recent introduction of Dataflow Prime, a new no-ops, auto-tuning functionality that optimizes resource utilization and further simplifies big data processing. You might have also read that Google Dataflow is a Leader in The 2021 Forrester Wave™: Streaming Analytics, giving Dataflow a score of 5 out of 5 across 12 different criteria.  

We couldn’t be more excited about the support the community has provided to this platform. The scalability of Dataflow is unparalleled and as you set your company up for more scale, more speed, and “streaming that screams”, we suggest you take a look at what leaders at Sky, RVU or Palo Alto Networks have already accomplished.

If you’re new to Dataflow, you’re in for a treat: this past month, Priyanka Vergadia (AKA CloudGirl) released a great set of resources to get you started. Read her blog here and watch her introduction video below!

https://youtube.com/watch?v=WRspZRG9e90%3Fenablejsapi%3D1%26

Simple structure that sticks together

We thrive to be the partner of choice for your transformation journey, regardless where your data comes from and how you choose to unify your data stack.  

Our partners at Tata Consultancy Services (TCS) recently released research that highlights the importance of a unifying digital fabric and how data integration services like Google Cloud Data Fusion can enable their clients to achieve this vision.

We also  announced SAP Integration with Cloud Data Fusion, Google Cloud’s native data integration platform, to seamlessly move data out of SAP Business Suite, SAP ERP and S4/HANA. To date, we provide more than 50 pipelines in Cloud Data Fusion to rapidly onboard SAP data.  

This past month, we introduced our SAP Accelerator for Order to Cash.  This accelerator is a sample implementation of the SAP Table Batch Source feature in Cloud Data Fusion and will help you get started with your end-to-end order to cash process and analytics. 

It includes sample Cloud Data Fusion pipelines that you can configure to connect to your SAP data source, perform transformations, store data in BigQuery, and set up analytics in Looker. It also comes with LookML dashboards which you can access on Github.

Countless great organizations have chosen to work with Google for their SAP data. In June, we wrote about ATB Financial’s journey and how the company uses data to better serve over 800,000 customers, save over CA$2.24 million in productivity, and realize more than CA$4 million in operating revenue through “D.E.E.P”, a data exposure enablement platform built around BigQuery.

Finally, if you are an application developer looking for a unified platform that brings together data from Firebase Crashlytics, Google Analytics, Cloud Firestore, and third party datasets, we have good news!  

This past month, we released a unified analytics platform that combines Firebase, BigQuery, Google Looker and FiveTran to easily integrate disparate data sources,  and infuse data into operational workflows for greater product development insights and increased customer experience. This resource comes with sample code, a reference guide and a great blog!  We hope you enjoy it. See you all next month!

https://youtube.com/watch?v=L25Vfzr2Ciw%3Fenablejsapi%3D1%26