When you think about climate change, you might not consider a daily commute to work or a drive around town as big contributing factors. And yet, transport is the fastest growing source of CO2 emissions from fossil fuel, which in turn is the largest contributor to climate change. This is the key insight behind the mission of Spanish carbon-neutral multinational Acciona. The group, which provides sustainable solutions for infrastructure and renewable energy projects across 65 countries, launched an electric scooter sharing service called Acciona Mobility in 2018.

Aiming to contribute to the decarbonization of the transport sector while helping relieve traffic congestion within cities, Acciona Mobility enables users to rent electric scooters powered 100% by energy from renewable sources. Users can find, reserve, and rent the scooters at the tap of their mobile screens via the Acciona Mobility application and pay on the basis of minutes spent riding. The app, which is available free to download online, is also the gateway through which new users can register for the service, which welcomes anyone with a valid driver’s license for motorbikes. After having their right to drive, identity, and card details validated, citizens can start enjoying the service, available 24 hours a day, every day of the year.

“We needed to deploy a reliable infrastructure for a new service before understanding exactly what its scale and demand would be. Our strategy was to adopt a serverless architecture that can grow with us to sustain our long-term vision. This kind of thinking led us directly to Google Cloud.”

—Jose Luis Rosell, CIO Services Division, Acciona

“Acciona Mobility is exciting because it stems from our vision to make people’s lives easier in a sustainable way,” says Jose Luis Rosell, CIO of Acciona’s Services Division. “40% of the pollution near cities comes from private transportation. We want to help solve that problem with zero-carbon, electric, multimodal transportation that’s also convenient for people to use.”

However, Jose Luis says that building the appropriate IT infrastructure to bring the new service to life required external support. “We needed to deploy a reliable infrastructure for a new service before understanding exactly what its scale and demand would be. Our strategy was to adopt a serverless architecture that can grow with us to sustain our long-term vision,” he explains. “This kind of thinking led us directly to Google Cloud.”

Developing the transportation platform of the future

Because Acciona was already a Google Cloud customer, Jose Luis and his team reached the decision to develop the new service on Google Cloud with ease, as he explains: “We had a very positive experience working closely with the Google Cloud team during Acciona’s first project on Google Cloud, and that collaborative mentality is very important for us. We feel Google Cloud is a trustworthy partner that is willing to take risks with us and is flexible enough to lead us through business uncertainty with the right technology,” he says. “So we were confident that its technology and people could help us bring the Acciona Mobility service from theory to reality.”

This time, Acciona partnered with cloud consultancy Altostratus as well, to consult on a selection of Google Cloud products that would help develop the application quickly by making use of managed services. Acciona’s strategy was to launch the new services first in Madrid, where the company is headquartered, and then spread the vision across more cities later, if the idea worked out well in practice. Due to the scale of the project and its data needs, BigQuery was chosen as the primary data warehouse to store all the information related to the service, such as the location of scooters and their availability status. With all its data readily available on BigQuery, Acciona is able to run specific queries that help it to gain insights such as which urban areas have the highest volume of scooters being rented. Knowing this, Acciona can reorganize availability to make sure there are always enough scooters in that specific location.

Integrating it with Pub/Sub and Dataflow, Acciona ensures that the data generated by scooters is ingested and processed in real time so that users searching for scooters nearby, using the Acciona Mobility application on their phones, can always have up-to-date information at hand. Using Google Maps Platform APIs such as the Directions API, Acciona’s mobile app translates the geographical coordinates of scooters into an easy-to-read address displayed beside a visual map. Using Cloud SQL, Acciona automates the storage capacity management of the database as the number of scooters and active users grow.

Meanwhile, Google Cloud Armor protects the service against cyber breaches and distributed denial of service attacks to keep it running uninterrupted. In addition, Google Cloud itself helps to reinforce the cyber security measures with secure-by-default managed services, such as data encryption. “The managed services mean more resilience and more uptime, because we don’t need to worry about maintenance or external threats,“ says Jose Luis of the solutions protecting all information generated by the new service, which as CIO is a topic he holds dearly. “I face cybersecurity issues more confidently as a Google Cloud partner,” he adds.

“Today, we have data coming in every 15 seconds from 10,000 electric scooters around Europe. BigQuery and Cloud SQL-managed services help us handle that data very carefully and with precision so that our users don’t experience any lags when searching for scooters or returning them.”

—Jose Luis Rosell, CIO Services Division, Acciona

Expanding an 100% sustainable transport solution throughout Spain

Within two months of the decision to deploy Acciona Mobility on Google Cloud, the service was ready for launch, filling the streets of Madrid with 500 electric scooters that run 100% on renewable energy. By 2020, the number of scooters has grown to 10,000 and the service has scaled to multiple cities in and outside of Spain, including Lisbon, Milan, and Rome. To date, more than 3.5 million intracity rides have used the Acciona Mobility app, reducing CO2 emissions by more than 1,000 tons.

“Today, we have data coming in every 15 seconds from 10,000 electric scooters around Europe. BigQuery and Cloud SQL-managed services help us handle that data very carefully and with precision so that our users don’t experience any lags when searching for scooters or returning them,” says Jose Luis of the fast-growing scale of the project.

“Google Cloud enables us to keep our services always available and to scale quickly to respond to its growing demand. Our ratings have been very positive as a result. More importantly, our vision of transportation as a cleaner, more sustainable commodity is really spreading and resonating with the public.”

—Jose Luis Rosell, CIO Services Division, Acciona

Set to continue expanding, Acciona is using Google Kubernetes Engine to make sure the Acciona Mobility application has the computing capacity it needs to continue scaling to reach new markets and serve more users. On that note, Google operations tools (formerly Stackdriver) are also being deployed to help monitor, troubleshoot, and improve the performance of the application as it scales.

“Google Cloud enables us to keep our services always available and to scale quickly to respond to its growing demand. Our ratings have been very positive as a result. More importantly, our vision of transportation as a cleaner, more sustainable commodity is really spreading and resonating with the public,” he concludes.

As cloud adoption continues to grow, so too does the number of cloud-born security threats. However, cloud environments can present significant opportunities to improve security with the right tools and processes in place.

When it comes to effective threat detection, investigation and response (TDIR) in the cloud, modern solutions must ensure that the entire security operations workflow — from data analysis through detection to response — are working in tandem to deliver the insights, context, and processes needed for cyber defenders to respond to threats with speed and precision. 

At Google Cloud, we believe that modern security operations should rely less on customer engineering and more on packaged outcomes delivered by solution providers. With this in mind, we are excited to announce today at our annual Security Summit that Chronicle Security Operations now provides turnkey TDIR for Google Cloud. By integrating with our cloud-focused Security Command Center Premium (SCC) and Google Cloud telemetry, Chronicle can collect and analyze data from Google Cloud, detect and investigate threats, and automate responses to mitigate risks.

In our recent State of Cloud Threat Detection and Response Survey, 71% of respondents said that “entire classes of threats are eliminated by migrating to the cloud,” and 82% stated that “the cloud affords the ability to process more data, including on-prem data, which can improve detection across the board.” 

To take advantage of all that the cloud can do for security, organizations should to do more than “lift and shift” their existing security tools and processes to the cloud. The cloud presents a different attack surface, often across several cloud services and data repositories, and each can have different attack tactics, potential misconfigurations, and context.  

This update to Chronicle helps enable teams to:

• Detect with confidence. Out-of-the-box detection rule sets developed by Google threat researchers surface cloud attack vectors and provide high fidelity, contextualized alerts that quickly give insight into potential threats in your Google Cloud environment. 
• Investigate with full context. Visualize threat storylines, complete with cloud-specific context that is correlated with additional data and context from across your environment for fast and efficient investigations.
• Respond with speed and precision. Streamline workflows and automate response actions with prebuilt playbooks and best practices designed specifically for Google Cloud. Chronicle SOAR’s case management and team collaboration help ensure fast and timely response.
• Simplify data ingestion. Chronicle automatically ingests, normalizes and contextualizes cloud telemetry from a variety of Google Cloud services (such as Cloud Asset Inventory, Google Kubernetes Engine, Google Compute Engine, cloud audit logs, and Cloud DLP), reducing the need for complex and time-consuming engineering.

Putting Google Cloud TDIR to work 

Let’s take a closer look at how our end-to-end cloud TDIR workflow manages a potential Google Cloud attack. 

Setting up cloud TDIR in Chronicle only takes a few clicks. Security Command Center’s built-in threat detection identifies attacks against Google Cloud resources. These findings, as well as audit, NAT, DNS, and firewall logs, are ingested into Chronicle to provide additional insight and context into Google Cloud threats.

Chronicle now provides detection rules for Google Cloud threats. These rules correlate SCC findings with Chronicle’s advanced detection engine to reveal the broad scope of malicious activity, giving you visibility and more contextual information into what’s going on in your environment. In our example, Chronicle alerts on suspicious activity that can indicate an attempt to exfiltrate data from BigQuery.

The new Chronicle Alert Graph surfaces key details of primary alerts you’re investigating in seconds. Combining cloud alerts and telemetry and correlating that with vital context from other sources (such as user data, endpoint data, and threat intelligence,) you can explore the visual representation of an alert’s relationship to other alerts and entities, dig into the potential attack paths, get quick summaries of implicated security artifacts, and pivot into your Google Cloud Console to do a deeper dive into potentially-impacted resources.  

In our example, we can see a BigQuery exfiltration event associated with a customer and that it’s associated with a particular service account tied to a Google Cloud org. Alert context below tells us more about what’s been impacted. It shows us that cloud credentials in the form of encryption keys and email addresses were associated with the event.

Chronicle case management automatically groups any related alerts into threat-centric cases, uniting the information that matters and making it simple for you to see and understand the scope of the event. 

In our example, Chronicle groups these alerts together based on common source address and username, and the case wall provides the summary of the alerts and actions that are taking place across the entire case.

Chronicle playbooks, designed specifically for Google Cloud, automate your desired response processes. In our example, when an alert was generated a playbook automatically ran through predefined steps that gathered data, enrichment, and took automated remediation steps to prevent this service account and instance from continuing.

Ready to detect with confidence, investigate with broad context, respond with speed and precision, and simplify data ingestion? Chronicle Security Operations is your go-to for turnkey, end-to-end threat detection, investigation, and response on Google Cloud.    

Tune in to our Security Summit session to learn more and see a demo of the new Google Cloud TDIR capabilities.

Employees use tools at their disposal to get work done, but if these tools (often legacy) hamper collaboration or are inflexible, they’ll turn to less secure options for the sake of convenience. According to Gartner, a third of successful attacks experienced by enterprises will come from Shadow IT usage by 2020. 

And this problem is not unknown. Eighty-three percent of IT professionals reported that employees stored company data in unsanctioned cloud services, a challenge especially apparent with file sync and share tools. When people work around their legacy systems to use tools like Google Drive, it’s often because they find their current systems to be clunky or that they can’t collaborate with others as easily. They’re unable to do three key things in legacy file sync and share systems (like Microsoft SharePoint):

1. Unable to work on their phones. By now, people expect to be able to work on the go—and this means not just opening an attachment, but actually making edits to and comments on work. It gives them freedom to work when it’s convenient for them and to help teammates anytime. 

2. Unable to create workspaces independently and easily. This might sound counterintuitive, but if an employee needs to contact IT to have a new project folder made on a drive, the bar is too high. Employees need to be able to quickly, and independently, create documents that can be shared simply because of the changing nature of collaboration. Work happens ad-hoc, on the go (like we mentioned above), and with people inside and outside of your organization. If someone has to contact IT to create a new folder, they’re more likely to neglect the request or use a different tool altogether to get started. 

3. Unable to make the data work for them. Traditional file storage is just that, storage. Like an attic, we store things in these systems, but at some point stuff gets stale and it’s hard to tell what we should keep or pitch. People need their storage systems to not only house their data, but to help them categorize and find information quicker so that they can make this data work better for them.

You have two choices when it comes to making a decision on file sync and share systems:

Option 1: Continue to let your employees work on unsanctioned products, some of which may open your business up to unintended security issues (and, in some instances, scary terms of service).

Option 2: Buy the tools that your users want to use because these tools are making them more productive.

If you want to create a more productive workforce, take cues from your employees. Your tools should not only meet the highest security standards for IT, but let people work the way they want to (and be intelligent enough to guide them along the way).

Imagine if your technology could flag that a file contains confidential information before an employee accidentally shares it. Or surface files as they’re needed to help people work faster. Google Drive does this.

Remember, if the technology doesn’t suit your employees, they’re just going to work around it anyway. Instead of investing time and resources on routine maintenance, shift this energy toward helping your employees stay productive in ways that work for both you and them.

Diagnosing and treating chronic pain can be complex, difficult, and full of uncertainties for a patient and their treating physician. Depending on the condition of the patient and the knowledge of the physician, making the correct diagnosis takes time, and experimenting with different treatments might be required.

This trial-and-error process can leave the patient in a world of pain and confusion until the best remedies can be prescribed. It’s a situation similar to the daily struggle that many of today’s security operations teams face.

Screaming from the mountain tops “just patch it!” isn’t very helpful when security teams aren’t sure if applying a patch might create even worse issues like crashes, incompatibility, or downtime. Like a patient with chronic pain, they may not know the source of the pain in their system. Determining which vulnerabilities to prioritize patching, and ensuring those fixes actually leave you with a more secure system, is one of the hardest tasks a security team can face. This is where a Vulnerability Exploitability eXchange (VEX) comes in.

The point of VEX

In previous blogs, we’ve discussed how establishing visibility and awareness into patient safety and technology is vital to creating a resilient healthcare system. We’ve also looked at how combining software bills of materials (SBOM) with Google’s Supply chain Levels for Software Artifacts (SLSA) framework can help build more secure technology that enables resilience.

The SBOM provides visibility into the software you’re using and where it comes from, while SLSA provides guidelines that help increase the integrity and security of software you then build. Rapid diagnostic assessments can be added to that equation with VEX, which the National Telecommunications and Information Administration describes as a “companion” document that lives side-by-side with SBOM.

To go back to our medical metaphor, VEX is a mechanism for software providers to tell security teams where to look for the source of the pain. VEX data can help with software audits when inventory and vulnerability data need to be captured at a specific point in time. That data also can be embedded into automated security tools to make it easier to prioritize vulnerability patching.

You can then think of SBOM as the prescription label on a bottle of medication, SLSA as the child-proof lid and tamper-proof seal guaranteeing the safety of the medication, and VEX as the bottle’s safety warnings. As a diagnostic aide, a VEX can help security teams make accurate diagnoses of “what could hurt” and system weaknesses before the bad guys do.

Yet making an accurate assessment of that threat model can be challenging, especially when looking at the software we use to run systems. The ability to quickly and accurately evaluate an organizations’ weaknesses and pain points can be vital to hastening response to a vulnerability and stopping cyberattacks before they become destructive. We believe that VEX is an important part of the equation to help secure the software supply chain.

As an example, look no further than the Apache Log4j vulnerabilities revealed in December 2021. Global industries including healthcare were dealt another blow when Apache’s Log4j 2 logging system was found to be so vulnerable that relatively unsophisticated threat actors could quickly infiltrate and take over systems. Through research conducted by Google and information contributed by CISA, we learned of examples of where vulnerabilities in Log4j 2, a single software component, could potentially impact thousands of companies using software that depend on it because of its near-ubiquitous use.

While a VEX would not capture zero-day vulnerabilities, it would be able to inform security teams of other known vulnerabilities in Log4j 2. Once vulnerabilities have been published, security teams could use SBOM to find them, and use VEX to understand if remediation is a priority or not.

How does VEX contribute to visibility?

A key reason we focus on visibility mechanisms like SBOM and SLSA is because they give us the ability to understand our risks. Without the ability to see into what we must protect, it can be difficult to determine how to quickly reduce risk.

Visibility is a crucial first step to stopping malicious hackers. Yet without context, visibility leaves security teams overwhelmed with data. Why? Well, where would you start when trying to mitigate the 30,000 known vulnerabilities affecting just open source software, according to the Open Source Vulnerabilities database(OSV)? NIST’s National Vulnerability Database (NVD) is tracking close to 181,000 vulnerabilities. We’ll be patching into the next millennium if we adopt a “patch everything” approach.

It’s impossible to address every vulnerability individually. To make progress, security teams need to be able to prioritize findings and go after the ones that will have the greatest impact first. The goal of a VEX artifact is to make prioritization a little easier.

While SBOMs are created or changed when the material included in a build is updated, VEXs are intended to be changed and distributed when a new vulnerability or threat has changed. This means that VEX and SBOM should be maintained separately. Since security researchers and organizations are constantly discovering new cybersecurity vulnerabilities and threats, a more dynamic mechanism like VEX can help ensure builders and operators have the ability to quickly ascertain the risks of the software they are using.

Let’s dig into this VEX example from CycloneDX. You can see the list of vulnerabilities found, third parties who track and report those vulnerabilities, vulnerability ratings per CVSS, and most importantly, a statement from the developer that guides the operator reading the VEX to those vulnerabilities that are exploitable and need to be protected. At the bottom, you’ll see the VEX “affects” an SBOM.

This information allows the user of the VEX document to refer to its companion SBOM. By necessity, the VEX is intentionally decoupled from the SBOM because they need to be updated at different times. A VEX document will need to be updated when new vulnerabilities emerge. An SBOM will need to be updated when changes to the software are made by a manufacturer. Although they can and need to be updated separately, the contents of each document can stay aligned because they are linked.

Increasing resilience powered by visibility—SBOM+VEX+SLSA

VEX could dramatically improve how security vulnerabilities are handled. It’s not uncommon to find operators buried in vulnerabilities, best-guessing the ones that need fixing, and trying to make sense of tens (and sometimes hundreds) of pages of documentation to determine the best, lowest impact fix.

With SBOM+SLSA+VEX, operators are using software-driven mechanisms to conduct analyses and evaluate risk instead of relying on intuition and best guesses. The tripartite SBOM+SLSA+VEX approach provides an up-to-date list of issues and perspective on what needs attention. This is a transformative development in security—enabling teams to get a better handle on doing vulnerability mitigation, starting where it could hurt the most.

Driven by repeated cyberattacks on critical infrastructure such as healthcare, government regulators have taken a more interested stance in software security and supply chains. Strengthening the effectiveness of SBOMs in the United States is a big part of the newly proposed Protecting and Transforming Cyber Health Care (PATCH) Act. The law would require medical device manufacturers adhere to minimum cybersecurity standards in their products, including the creation of SBOMs for their devices, and plans to monitor and patch any cybersecurity vulnerabilities that are discovered during the device’s lifetime.

Meanwhile, new draft medical device cybersecurity guidance from the FDA continues that agency’s involvement in aggressively encouraging medical device manufacturers to improve the cybersecurity resilience of their products. The White House spoke for SBOMs, as well. An Executive Order from May 2021 lays out requirements for secure software development, including the production and distribution of SBOM for software used by the federal government.

Regardless of how these initiatives pan out, Google believes controls like those provided by SBOM+SLSA+VEX are critical to protect software and build a resilient healthcare ecosystem. This approach provides detailed, critical risk exposure data to security teams so they can take necessary steps to reduce immediate and long-term risks.

What do we suggest you do?

At Google, we are working with the Open Source Security Foundation on supporting SBOM development. Our Know, Prevent, Fix report on secure software development creates a broader outline of how Google thinks about securing open source software from preventable vulnerabilities. You can read more about these efforts for securing workloads on Google Cloud from our Cloud Architecture Center. Take a look at Cloud Build, a Google Cloud service that can be used to generate up to SLSA Level 2 build artifacts.

Customers often have difficulty getting full visibility and control over vulnerabilities because of their dependence on open source software (OSS). Assured Open Source Software (Assured OSS) is the Google Cloud service that helps teams both secure the external OSS packages they use and overcome avoidable vulnerabilities by simply eliminating them from the code base. Finally, ask us about Google’s Cybersecurity Action Team, the world’s premier security advisory team and its singular mission supporting the security and digital transformation of governments, critical infrastructure, enterprises, and small businesses.

If you’re a software supplier, please consider our suggestions above. Whether you are or not, you should begin:

• Contractually mandating SBOM+VEX+SLSA (or their equivalent) artifacts to be provided for all new software.
• Train procurement teams to ask for and use SBOM+VEX+SLSA to make purchasing decisions. There should be no reason an organization procures software or hardware with known, preventable issues. Even if they do, the information these mechanisms provide should help security teams decide if they can live with the risks before equipment enters their networks.
• Establishing a governance program that ensures those who control procurement decisions are aware of and owning the risks associated with software they are buying.
• Enabling security teams to build pipelines to ingest SBOM+VEX+SLSA artifacts into their security operations and use it to strategically advise and drive mitigation activities.

At Google, we believe the path to resilience begins with building visibility and structural awareness into the software, hardware, and equipment it rides on as a critical first step. Time will tell if VEX becomes widely adopted, but the point behind it won’t change—we can’t know how we are vulnerable without visibility. VEX is an important concept in this regard.

Next month, we’ll be shifting gears slightly to focus on building resilience by establishing a security culture that obsesses over its patients and products.

June is the month that holds the summer solstice, and some of us in the northern hemisphere get to enjoy the longest days of sunshine out of the entire year. We used all the hours we could in June to deliver a flurry of new features across BigQuery, Dataflow, Data Fusion, and more.  Let’s take a look!

Simple, Sophisticated, and Secure

Usability is a key tenant of our data analytics development efforts. Our new user-friendly BigQuery improvements this month include:

• Flexible data type casting
• Formatting to change column descriptions 
• GRANT/REVOKE access control commands using SQL

We hope this will delight data analysts, data scientists, DBAs, and SQL-enthusiasts who can find out more details in our blog here.

Beyond simplifying commands, we also recognize that it’s equally important to have more sophistication when dealing with transactions. That’s why we introduced multi-statement transactions in BigQuery.

As you probably know, BigQuery has long supported single-statement transactions through DML statements, such as INSERT, UPDATE, DELETE, MERGE and TRUNCATE, applied to one table per transaction. With multi-statement transactions, you can now use multiple SQL statements, including DML, spanning multiple tables in a single transaction. 

This means that any data changes across multiple tables associated with all statements in a given transaction are committed atomically (all at once) if successful—or all rolled back atomically in the event of a failure. 

We also know that organizations need to control access to data, down to the granular level and that, with the complexity of data platforms increasing day by day, it’s become even more critical to identify and monitor who has access to sensitive data. 

To help address these needs,  we announced the general availability of BigQuery row-level security. This capability gives customers a way to control access to subsets of data in the same table for different groups of users. Row-level security in BigQuery enables different user personas access to subsets of data in the same table and can easily be created, updated, and dropped using DDL statements. To learn more, check out the documentation and best practices.

Simple, Safe, and Smart

Beyond building a simpler, more sophisticated and more secure data platform for customers, our team has been focused on providing solutions powered by built-in intelligence. One of our core beliefs is that for machine learning to be adopted and useful at scale, it must be easy to use and deploy.  

BigQuery ML, our embedded machine learning capabilities, have been adopted by 80% of our top customers around the globe and it has become a cornerstone of their data to value journey.  

As part of our efforts, we announced the general availability of AutoML tables in BigQuery ML.  This no-code solution lets customers automatically build and deploy state-of-the-art machine learning models on structured data. With easy integration with Vertex AI, AutoML in BQML makes it simple to achieve machine learning magic in the background. From preprocessing data to feature engineering and model tuning all the way to cross validation, AutoML will “automagically” select and ensemble models so everyone—even non-data scientists—can use it.   

Want to take this feature for a test drive? Try it today on BigQuery’s NYC Taxi public dataset following the instructions in this blog! 

Speaking of public datasets, we also introduced the availability of Google Trends data in BigQuery to enable customers to measure interest in a topic or search term across Google Search.  This new dataset will soon be available in Analytics Hub and will be anonymized, indexed, normalized, and aggregated prior to publication. 

Want to ensure your end-cap displays are relevant to your local audience?  You can take signals from what people are looking for in your market area to inform what items to place. Want to understand what new features could be incorporated into an existing product based on what people are searching for?  Terms that appear in these datasets could be an indicator of what you should be paying attention to.

All this data and technology can be put to use to deploy critical solutions to grow and protect your business. For example,  it can be difficult to know how to define anomalies during detection. If you have labeled data with known anomalies, then you can choose from a variety of supervised machine learning model types that are already supported in BigQuery ML. 

But what if you don’t know what kind of anomaly to expect, and you don’t have labeled data? Unlike typical predictive techniques that leverage supervised learning, organizations may need to be able to detect anomalies in the absence of labeled data. 

That’s why, we were particularly excited to announce the public preview of new anomaly detection capabilities in BigQuery ML that leverage unsupervised machine learning to help you detect anomalies without needing labeled data.  

Our team has been working with a large number of enterprises who leverage machine learning for better anomaly detection. In financial services for example, customers have used our technology to detect machine-learned anomalies in real-time foreign exchange data.  

To make it easier for you to take advantage of their best practices, we teamed up with Kasna to develop sample code, architecture guidance, and a data synthesizer that generates data so you can test these innovations right away. 

Simple, Scalable, and Speedy

Capturing, processing and analyzing data in motion has become an important component of our customer architecture choices. Along with batch processing, many of you need the flexibility to stream records into BigQuery so they can become available for query as they are written.  

Our new BigQuery Storage Write API combines the functionality of streaming ingestion and batch loading into a single API. You can use it to stream records into BigQuery or even batch process an arbitrarily large number of records and commit them in a single atomic operation.

Flexible systems that can do batch and real-time in the same environment is in our DNA: Dataflow, our serverless, data processing service for streaming and batch data was built with flexibility in mind.  

This principle applies not just to what Dataflow does but also how you can leverage it—whether you prefer using Dataflow SQL right from the BigQuery web UI, Vertex AI notebooks from the Dataflow interface, or the vast collection of pre-built templates to develop streaming pipelines.

Dataflow has been in the news quite a bit recently. You might have noted the recent introduction of Dataflow Prime, a new no-ops, auto-tuning functionality that optimizes resource utilization and further simplifies big data processing. You might have also read that Google Dataflow is a Leader in The 2021 Forrester Wave™: Streaming Analytics, giving Dataflow a score of 5 out of 5 across 12 different criteria.  

We couldn’t be more excited about the support the community has provided to this platform. The scalability of Dataflow is unparalleled and as you set your company up for more scale, more speed, and “streaming that screams”, we suggest you take a look at what leaders at Sky, RVU or Palo Alto Networks have already accomplished.

If you’re new to Dataflow, you’re in for a treat: this past month, Priyanka Vergadia (AKA CloudGirl) released a great set of resources to get you started. Read her blog here and watch her introduction video below!

https://youtube.com/watch?v=WRspZRG9e90%3Fenablejsapi%3D1%26

Simple structure that sticks together

We thrive to be the partner of choice for your transformation journey, regardless where your data comes from and how you choose to unify your data stack.  

Our partners at Tata Consultancy Services (TCS) recently released research that highlights the importance of a unifying digital fabric and how data integration services like Google Cloud Data Fusion can enable their clients to achieve this vision.

We also  announced SAP Integration with Cloud Data Fusion, Google Cloud’s native data integration platform, to seamlessly move data out of SAP Business Suite, SAP ERP and S4/HANA. To date, we provide more than 50 pipelines in Cloud Data Fusion to rapidly onboard SAP data.  

This past month, we introduced our SAP Accelerator for Order to Cash.  This accelerator is a sample implementation of the SAP Table Batch Source feature in Cloud Data Fusion and will help you get started with your end-to-end order to cash process and analytics. 

It includes sample Cloud Data Fusion pipelines that you can configure to connect to your SAP data source, perform transformations, store data in BigQuery, and set up analytics in Looker. It also comes with LookML dashboards which you can access on Github.

Countless great organizations have chosen to work with Google for their SAP data. In June, we wrote about ATB Financial’s journey and how the company uses data to better serve over 800,000 customers, save over CA$2.24 million in productivity, and realize more than CA$4 million in operating revenue through “D.E.E.P”, a data exposure enablement platform built around BigQuery.

Finally, if you are an application developer looking for a unified platform that brings together data from Firebase Crashlytics, Google Analytics, Cloud Firestore, and third party datasets, we have good news!  

This past month, we released a unified analytics platform that combines Firebase, BigQuery, Google Looker and FiveTran to easily integrate disparate data sources,  and infuse data into operational workflows for greater product development insights and increased customer experience. This resource comes with sample code, a reference guide and a great blog!  We hope you enjoy it. See you all next month!

https://youtube.com/watch?v=L25Vfzr2Ciw%3Fenablejsapi%3D1%26