June is the month that holds the summer solstice, and some of us in the northern hemisphere get to enjoy the longest days of sunshine out of the entire year. We used all the hours we could in June to deliver a flurry of new features across BigQuery, Dataflow, Data Fusion, and more.  Let’s take a look!

Simple, Sophisticated, and Secure

Usability is a key tenant of our data analytics development efforts. Our new user-friendly BigQuery improvements this month include:

• Flexible data type casting
• Formatting to change column descriptions 
• GRANT/REVOKE access control commands using SQL

We hope this will delight data analysts, data scientists, DBAs, and SQL-enthusiasts who can find out more details in our blog here.

Beyond simplifying commands, we also recognize that it’s equally important to have more sophistication when dealing with transactions. That’s why we introduced multi-statement transactions in BigQuery.

As you probably know, BigQuery has long supported single-statement transactions through DML statements, such as INSERT, UPDATE, DELETE, MERGE and TRUNCATE, applied to one table per transaction. With multi-statement transactions, you can now use multiple SQL statements, including DML, spanning multiple tables in a single transaction. 

This means that any data changes across multiple tables associated with all statements in a given transaction are committed atomically (all at once) if successful—or all rolled back atomically in the event of a failure. 

We also know that organizations need to control access to data, down to the granular level and that, with the complexity of data platforms increasing day by day, it’s become even more critical to identify and monitor who has access to sensitive data. 

To help address these needs,  we announced the general availability of BigQuery row-level security. This capability gives customers a way to control access to subsets of data in the same table for different groups of users. Row-level security in BigQuery enables different user personas access to subsets of data in the same table and can easily be created, updated, and dropped using DDL statements. To learn more, check out the documentation and best practices.

Simple, Safe, and Smart

Beyond building a simpler, more sophisticated and more secure data platform for customers, our team has been focused on providing solutions powered by built-in intelligence. One of our core beliefs is that for machine learning to be adopted and useful at scale, it must be easy to use and deploy.  

BigQuery ML, our embedded machine learning capabilities, have been adopted by 80% of our top customers around the globe and it has become a cornerstone of their data to value journey.  

As part of our efforts, we announced the general availability of AutoML tables in BigQuery ML.  This no-code solution lets customers automatically build and deploy state-of-the-art machine learning models on structured data. With easy integration with Vertex AI, AutoML in BQML makes it simple to achieve machine learning magic in the background. From preprocessing data to feature engineering and model tuning all the way to cross validation, AutoML will “automagically” select and ensemble models so everyone—even non-data scientists—can use it.   

Want to take this feature for a test drive? Try it today on BigQuery’s NYC Taxi public dataset following the instructions in this blog! 

Speaking of public datasets, we also introduced the availability of Google Trends data in BigQuery to enable customers to measure interest in a topic or search term across Google Search.  This new dataset will soon be available in Analytics Hub and will be anonymized, indexed, normalized, and aggregated prior to publication. 

Want to ensure your end-cap displays are relevant to your local audience?  You can take signals from what people are looking for in your market area to inform what items to place. Want to understand what new features could be incorporated into an existing product based on what people are searching for?  Terms that appear in these datasets could be an indicator of what you should be paying attention to.

All this data and technology can be put to use to deploy critical solutions to grow and protect your business. For example,  it can be difficult to know how to define anomalies during detection. If you have labeled data with known anomalies, then you can choose from a variety of supervised machine learning model types that are already supported in BigQuery ML. 

But what if you don’t know what kind of anomaly to expect, and you don’t have labeled data? Unlike typical predictive techniques that leverage supervised learning, organizations may need to be able to detect anomalies in the absence of labeled data. 

That’s why, we were particularly excited to announce the public preview of new anomaly detection capabilities in BigQuery ML that leverage unsupervised machine learning to help you detect anomalies without needing labeled data.  

Our team has been working with a large number of enterprises who leverage machine learning for better anomaly detection. In financial services for example, customers have used our technology to detect machine-learned anomalies in real-time foreign exchange data.  

To make it easier for you to take advantage of their best practices, we teamed up with Kasna to develop sample code, architecture guidance, and a data synthesizer that generates data so you can test these innovations right away. 

Simple, Scalable, and Speedy

Capturing, processing and analyzing data in motion has become an important component of our customer architecture choices. Along with batch processing, many of you need the flexibility to stream records into BigQuery so they can become available for query as they are written.  

Our new BigQuery Storage Write API combines the functionality of streaming ingestion and batch loading into a single API. You can use it to stream records into BigQuery or even batch process an arbitrarily large number of records and commit them in a single atomic operation.

Flexible systems that can do batch and real-time in the same environment is in our DNA: Dataflow, our serverless, data processing service for streaming and batch data was built with flexibility in mind.  

This principle applies not just to what Dataflow does but also how you can leverage it—whether you prefer using Dataflow SQL right from the BigQuery web UI, Vertex AI notebooks from the Dataflow interface, or the vast collection of pre-built templates to develop streaming pipelines.

Dataflow has been in the news quite a bit recently. You might have noted the recent introduction of Dataflow Prime, a new no-ops, auto-tuning functionality that optimizes resource utilization and further simplifies big data processing. You might have also read that Google Dataflow is a Leader in The 2021 Forrester Wave™: Streaming Analytics, giving Dataflow a score of 5 out of 5 across 12 different criteria.  

We couldn’t be more excited about the support the community has provided to this platform. The scalability of Dataflow is unparalleled and as you set your company up for more scale, more speed, and “streaming that screams”, we suggest you take a look at what leaders at Sky, RVU or Palo Alto Networks have already accomplished.

If you’re new to Dataflow, you’re in for a treat: this past month, Priyanka Vergadia (AKA CloudGirl) released a great set of resources to get you started. Read her blog here and watch her introduction video below!

https://youtube.com/watch?v=WRspZRG9e90%3Fenablejsapi%3D1%26

Simple structure that sticks together

We thrive to be the partner of choice for your transformation journey, regardless where your data comes from and how you choose to unify your data stack.  

Our partners at Tata Consultancy Services (TCS) recently released research that highlights the importance of a unifying digital fabric and how data integration services like Google Cloud Data Fusion can enable their clients to achieve this vision.

We also  announced SAP Integration with Cloud Data Fusion, Google Cloud’s native data integration platform, to seamlessly move data out of SAP Business Suite, SAP ERP and S4/HANA. To date, we provide more than 50 pipelines in Cloud Data Fusion to rapidly onboard SAP data.  

This past month, we introduced our SAP Accelerator for Order to Cash.  This accelerator is a sample implementation of the SAP Table Batch Source feature in Cloud Data Fusion and will help you get started with your end-to-end order to cash process and analytics. 

It includes sample Cloud Data Fusion pipelines that you can configure to connect to your SAP data source, perform transformations, store data in BigQuery, and set up analytics in Looker. It also comes with LookML dashboards which you can access on Github.

Countless great organizations have chosen to work with Google for their SAP data. In June, we wrote about ATB Financial’s journey and how the company uses data to better serve over 800,000 customers, save over CA$2.24 million in productivity, and realize more than CA$4 million in operating revenue through “D.E.E.P”, a data exposure enablement platform built around BigQuery.

Finally, if you are an application developer looking for a unified platform that brings together data from Firebase Crashlytics, Google Analytics, Cloud Firestore, and third party datasets, we have good news!  

This past month, we released a unified analytics platform that combines Firebase, BigQuery, Google Looker and FiveTran to easily integrate disparate data sources,  and infuse data into operational workflows for greater product development insights and increased customer experience. This resource comes with sample code, a reference guide and a great blog!  We hope you enjoy it. See you all next month!

https://youtube.com/watch?v=L25Vfzr2Ciw%3Fenablejsapi%3D1%26

We created Traffic Director to bring to you a fully managed service mesh product that includes load balancing, traffic management and service discovery. And now, we’re happy to announce the availability of a fully-managed zero-trust security solution using Traffic Director with Google Kubernetes Engine (GKE) and Certificate Authority (CA) Service.

When platform administrators and security professionals think about modernizing their applications with a forward-looking security posture, they look for “zero-trust” security. This security posture is based on few fundamental blocks:

1. A means of allocating and asserting service identity (for example, using X.509 certificates)
2. Mutual authentication (mTLS) or server authentication (TLS)
3. Encryption for all traffic flows (TLS encryption)
4. Authorization checks and minimal privileges
5. Infrastructure to make all of the above manageable and reliable

Traffic Director does this by integrating with CA Service, a highly available private CA which issues private certificates expressing service identities, and provides a managed mTLS certificate infrastructure with full certificate lifecycle management. Together, these solve both certificate issuance and CA rotation complexities. 

With Traffic Director managing your service-to-service security, you can now enjoy end-to-end encryption, service-level authentication and granular authorization policies for your service mesh.

With this new capability, you can now:

• Implement mutual TLS (mTLS) and TLS between your services, including certificate lifecycle management. Communications within your mesh are authenticated and encrypted.
• Enable identity-based authorization, as well as authorization based on other parameters (such as the request method). These concepts underpin role-based access controls (RBAC) and enable you to take a “least privileges” stance where only authorized services can communicate with each other based on ALLOW/DENY rules.

mTLS is supported whether you’re using Envoy or proxyless gRPC for your service mesh. Authorization support for proxyless gRPC is coming later this year. Check out our documentation to learn more and get started with Envoy or proxyless gRPC.

State and local government (SLG) agencies are reeling from a combination of unbudgeted COVID-related expenses and reduced tax revenue caused by unemployment and business closures. Any way you look at it, the situation is challenging. To understand how SLG agencies are coping, Google Cloud collaborated with MeriTalk to survey 200 SLG IT and program managers, uncovering some revealing trends in SLG technology innovation. Unsurprisingly, approximately 84% of SLG organizations report making budgetary tradeoffs to bridge funding the gaps the ongoing pandemic has created. 

However, researchers discovered a silver lining: The pandemic has also been a catalyst to modernize the legacy infrastructure in states and cities. The majority of survey respondents (88%) reported that their agency made greater modernization progress this past year than in the prior 10 years.

Walking a tightrope between innovation and budget pressure

According to 89% of state and local leaders, now is the time to invest in technology modernization. But 80% are experiencing a funding gap due to unbudgeted expenses related to the pandemic and declining tax revenue, which makes finding that balance between innovation and budget a serious challenge.

Some agencies are achieving the impossible, though. For example, the City of Pittsburgh Department of Innovation and Performance is working with Google Cloud to migrate  and modernize its legacy IT infrastructure. By decommissioning their data center and moving to Google Cloud, the city can build new data analytics tools to drive smart city initiatives and create entirely new applications to improve digital service delivery for its residents. As a result, the city will save costs, abandon its brittle legacy IT structure, and create a cloud-based technology platform for the future—becoming the region’s leader in cloud-native software development.

Google Cloud is enabling the city’s IT team by curating and delivering our certification training at no cost. The program includes live training sessions as well as on-demand training. 

Bridging funding gaps

In their drive to modernization, many SLG leaders are turning to grants as an important source of funding. Approximately 84% of those surveyed report making tradeoffs to bridge funding gaps, such as moving resources away from operations and maintenance (37%), increasing reliance on pandemic-related funding (31%), and delaying internal modernization efforts to enable remote work for employees (29%). One way that states are dealing with this tension between budget gaps and the need for innovation is to turn to Google Cloud for cost savings and improved capabilities.

For example, Google Cloud is helping the State of West Virginia innovate and enhance IT security despite decreased state funding. The state entered a multi-year agreement to ensure full access to enterprise-level Google Workspace capabilities for 25,000 state employees, keeping the state at the forefront of technology advancements at a projected cost savings of $11.5 million.

Similarly, Google Cloud helped build the Rhode Island Virtual Career Center to help the state’s constituents get back to work. Using familiar productivity tools within Google Workspace, employees can access new career opportunities quickly, while employers can reach more candidates. Skipper, the CareerCompass RI bot, uses data and machine learning to connect Rhode Islanders with potential new career paths and reskilling opportunities.

Enhancing services

Google Cloud is also helping agencies enhance services, including working with the State of Illinois to get unemployment funding to constituents in need.The state is using Contact Center AI to rapidly deploy virtual agents that help more than 1 million out of work citizens file unemployment claims faster. Capable of engaging in human-like conversations, these intelligent agents provide constituents with 24/7 access and enable government employees to focus on more complex, mission-critical tasks—such as combating fraud. In summer 2020, the virtual agents handled more than 140,000 phone and web inquiries per day, including 40,000 after-hours calls every night. The state anticipates an estimated annual savings of $100 million from the solution, which was deployed in just two weeks.

Working with Google, Ohio also uncovered $2 billion in fraudulent unemployment claims. We will continue to partner with the state to find fraudulent claims, and prioritize the processing of legitimate claims.

 Focusing on cybersecurity

Despite expanding security threats topping NASCIO’s list of 2021 State CIO priorities, more than one in three IT managers (35%) say their organization reduces security measures to expedite timelines. Partnering with Google Cloud has enabled many agencies to enhance their security measures while modernizing and staying within budget, investing in support for remote work devices, digital services for residents, and cybersecurity.

NYC Cyber Command works with city agencies to ensure systems are designed, built, and operated in a highly secure manner. NYC3 followed a cloud-first strategy using the Google Cloud Platform. The virtual operations demanded by the pandemic have increased the importance of security and compliance in SLG. Google Cloud is committed to act as a security transformation partner and be the trusted cloud for public sector agencies. 

Finally, to strengthen public and private partnerships, SLG organizations told MeriTalk that they need vendor partners to support modernization efforts for flexibility and collaboration (46%), need innovation-focused leadership groups to help balance technology needs with budget constraints (41%), and they expect significant returns on investments in cloud computing (38%), and data management/analytics (33%).

Google Cloud is helping SLG customers across the country invest in innovation to walk the tech tightrope—balancing innovation and budgets—and helping to build a more resilient future. Visit the State and Local Government solutions page to learn more.

API security incidents are increasingly common and disruptive. With the growth of API traffic, enterprises across the world are also experiencing an uptick in malicious API attacks, making API security a heightened priority. According to our latest API Security Research Report, 50% of organizations surveyed have  experienced an API security incident in the past 12 months and of those, 77% delayed the rollout of a new service or application. 

At the RSA Conference 2023 today, we’re making it faster and easier to help detect API abuse incidents with the introduction of Advanced API Security Machine Learning powered abuse-detection dashboards. Our newly introduced Machine Learning models are trained to detect business logic attacks. 

These types of attacks are notoriously hard to identify, and target APIs tied to intellectual property, business processes, or sensitive information, such as user data, listing of goods, or crediting accounts. These APIs must be accessible to provide business value, but have also become targets for attackers.

API security incidents can have a considerable impact on an organization’s operations and its bottom line. In June 2022, Imperva released a report titled Quantifying the Cost of API Insecurity, which estimates that lack of secure APIs could result in an average annual API-related total global cyber loss of between $41 billion to $75 billion annually. Furthermore, according to IBM’s  2022 Cost of a Data Breach Report, the average cost of a data breach is $4.35 million. It’s vital that organizations detect and mitigate API abuse incidents early to prevent prolonged fiscal and reputational damage to the business.

However, business logic attacks are harder to detect using static security policies, which allows attackers to manipulate legitimate functionality to achieve a malicious goal without triggering any static security alerts. For example, if a malicious actor gains control of a server and makes subtle changes, the shift in activity patterns of the server is generally undetectable to most monitoring tools. However, in this scenario, the Advanced API Security’s ML-powered API abuse detection model can help differentiate between legitimate and deviant traffic and immediately notify key stakeholders to act quickly and minimize blast radius of the problem.

The ML models that power API abuse detection have been trained and used by Google’s internal teams to help protect our public-facing APIs. The models rely on years of learning and best practices and are now available to all Apigee Advanced API Security customers.

Another challenge in detecting API abuse incidents is the volume of alerts. To reduce the risk of missing key security incidents, many static rules that detect less sophisticated attacks are incredibly sensitive: They generate a high volume of alerts. This makes finding the critical incidents within API traffic and acting to resolve them like “finding a needle in a haystack” for many IT teams. Apigee Advanced Security’s ML-powered dashboards more accurately identify critical API abuses and find similar patterns within the large number of bot alerts to help reduce the time to find and act on most important incidents.

With the help of Apigee Advanced API Security’s ML-powered abuse detection dashboards, customers can uncover critical API abuse incidents, including business logic attacks, scraping, and anomalies, faster. Critical threats are surfaced with clear and concise descriptions to capture the essence of the attack along with the most important characteristics such as the source of the attack, the number of API calls, and the duration of the attack, to help resolve the incident more rapidly. 

Machine Learning powered abuse-detection dashboards are available in Advanced API Security, a feature of Apigee API management that enables you to more easily detect API security misconfigurations, bad bots, and malicious activities. 

To get started with Advanced API Security’s ML-powered dashboards, start your free Apigee trial now.

Over the past two decades, Google has innovated to build some of the largest and most secure computing systems in the world. This scale requires us to deliver pioneering approaches to cloud security, which we pass on to our Google Cloud customers. We are committed to solving hard security problems like only Google can, as the tip of the spear of innovation and threat intelligence.

Today we’re excited to share the next step in this journey with the completion of our acquisition of Mandiant, a leader in dynamic cyber defense, threat intelligence and incident response services. Mandiant shares our cybersecurity vision and will join Google Cloud to help organizations improve their threat, incident and exposure management.

Combining Google Cloud’s existing security portfolio with Mandiant’s leading cyber threat intelligence will allow us to deliver a security operations suite to help enterprises globally stay protected at every stage of the security lifecycle. With the scale of Google’s data processing, novel analytics approaches with AI and machine learning, and a focus on eliminating entire classes of threats, Google Cloud and Mandiant will help organizations reinvent security to meet the requirements of our rapidly changing world.

We will retain the Mandiant brand and continue Mandiant’s mission to make every organization secure from cyber threats and confident in their readiness.

Context and threat intelligence from the frontlines

Our goal is to democratize security operations with access to the best threat intelligence and built-in threat detections and responses. Ultimately, we hope to shift the industry to a more proactive approach focused on modernizing Security Operations workflows, personnel, and underlying technologies to achieve an autonomic state of existence – where threat management functions can scale as customers’ needs change and as threats evolve.

Today Google Cloud security customers use our cloud infrastructure to ingest, analyze and retain all their security telemetry across multicloud and on-premise environments. By leveraging our sub-second search across petabytes of information combined with security orchestration, automation and response capabilities, our customers can spend more time defending their organizations.

The addition of Mandiant Threat Intelligence—which is compiled by their team of security and intelligence individuals spread across 22 countries, who serve customers located in 80 countries—will give security practitioners greater visibility and expertise from the frontlines. Mandiant’s experience detecting and responding to sophisticated cyber threat actors will offer Google Cloud customers actionable insights into the threats that matter to their businesses right now. We will continue to share groundbreaking Mandiant threat research to help support organizations, even for those who don’t run on Google Cloud.

Advancing shared fate for security operations

Google Cloud operates in a shared fate model, taking an active stake in the security posture of our customers. For security operations that means helping organizations find and validate potential security issues before they become an incident.

Detecting, investigating and responding to threats is only part of better cyber risk management. It’s also crucial to understand what an organization looks like from an attacker’s perspective and if an organization’s cybersecurity controls are as effective as expected.

By adding Mandiant’s attack surface management capabilities to Google Cloud’s portfolio, organizations will be able to continually monitor assets for exposures, enabling intelligence and red teams to move security programs from reactive to proactive to understand what’s vulnerable, misconfigured and exposed.

Once an organization’s attack surface is understood, validating existing security controls is critical. With Mandiant Security Validation, organizations will be able to continuously validate and measure the effectiveness of their cybersecurity controls across cloud and on-premise environments.

Transforming security operations and incident response

Security leaders and their teams often lack the resources and expertise required to keep pace with today’s ever changing threats. Organizations already harness Google’s security tools, expert advice and rich partner ecosystem to evolve their security program. Google’s Autonomic Security Operations also serves as a prescriptive solution to guide our customers through this modernization journey.

With the addition of Mandiant to the Google Cloud family, we can now offer proven global expertise in comprehensive incident response, strategic readiness and technical assurance to help organizations mitigate threats and reduce business risk before, during and after an incident.

In addition, Google Cloud’s security operations suite will continue to provide a central point of intelligence, analysis and operations across on-premise environments, Google Cloud and other cloud providers. Google Cloud is also deeply committed to supporting our technology and solution partners, and this acquisition will enable system integrators, resellers and managed security service providers to offer broader solutions to customers.

Comments on the news

“The power of stronger partnerships across the cybersecurity ecosystem is critical to driving value for clients and protecting industries around the globe. The combination of Google Cloud and Mandiant and their commitment to multicloud will further support increased collaboration, driving innovation across the cybersecurity industry and augmenting threat research capabilities. We look forward to working with them on this mission.” – Paolo Dal Cin, Global Lead, Accenture Security

“Google’s acquisition of Mandiant, a leader in security advisory, consulting and incident response services will allow Google Cloud to deliver an end-to-end security operations suite with even greater capabilities and services to support customers in their security transformation across cloud and on-premise environments.” – Craig Robinson, Research VP, Security Services, IDC

“Bringing together Mandiant and Google Cloud, two long-time cybersecurity leaders, will advance how companies identify and defend against threats. We look forward to the impact of this acquisition, both for the security industry and the protection of our customers.” – Andy Schworer, Director, Cyber Defense Engineering, Uber

We welcome Mandiant to the Google Cloud team, and together we look forward to helping security teams achieve so much more in defense of their organizations. You can read our release and Kevin Mandia’s blog for more on this exciting news.