Google Spanner Wins SIGOPS Hall of Fame Award 2022
2921
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Earlier this month, SIGOPS announced that it had selected the paper, “Spanner: Google’s Globally-Distributed Database” for the 2022 SIGOPS Hall of Fame Award, an honor bestowed on the most influential Operating Systems papers published by the organization.
In giving this award, the award committee stated:
“Spanner showed how to balance the requirements of consistency, availability, and partition tolerance for a globally replicated transactional database system. In doing so, Spanner simplified the implementation of a range of higher level services which could then scale across the planet through a familiar and powerful transactional API. The key insights of leveraging clock synchronization using TrueTime and a dedicated, redundant, high-capacity datacenter WAN distinct from the public Internet as the basis for scaling remain fundamental design patterns today.”
As one of the original authors of the paper, to say that I am humbled by this award is an understatement. It’s also taken me on a trip down memory lane, and gotten me thinking about all the ways — expected and unexpected — that Spanner has influenced not only how Google builds its applications, but also how it’s simplified application design and operations for organizations that aren’t running at hyperscale.
From very early on — think early 2000s — one of Google’s biggest challenges was scaling its software to keep up with the rapid growth of the internet. We needed a database that scaled, that could be built with a small team. Those needs led to Bigtable and the NoSQL movement: a feature-lean database that performed and scaled.
But as Google grew, and entered other lines of business, our priorities evolved. We needed to simplify the jobs of our ever-larger engineering staff and support business-critical services like the AdWords and Payments platforms, as well as then-new consumer products like Gmail. To do that, we knew we had to give our product teams database features like ACID transactions, multi-region consistency, schema-declared invariants, SQL, and more — without compromising on cost or scale. At the time it was an outrageous goal, but with our experience building Bigtable and Megastore, as well as new innovations like TrueTime, we knew it could be done – and it was clear talking to our internal customers that it needed to be done.
Like most good software, our plans for Spanner really came together when we found our first serious customer: AdWords. At the time, Google’s entire advertiser-facing interface was sharded across MySQL instances on specially acquired hardware. Ads had just completed a very challenging resharding project to accommodate Google’s growing business, and was resolute in its desire to never do it again. So the Spanner team and Ads partnered on a project, called F1, to transparently move the entire AdWords stack from MySQL to Spanner.
“Before Spanner, Google’s entire advertiser-facing interface was sharded across MySQL instances on specially acquired hardware.”
By 2012, when we published the Spanner paper, we had just completed the migration of AdWords frontend traffic to the new Spanner backend, and we knew we had the seed of something really interesting. Since then, we have been rapidly improving Spanner in every dimension. Google, like most businesses, needs databases that are fully managed, easy to deploy, highly available, and robust in the face of any workload. Google services set a high bar: Spanner stores a copy of the Internet (actually several, at different stages of the indexing pipeline) and is the bedrock of availability and durability for billions of users that depend on Google in their daily lives.
Then, in 2017, Google Cloud launched Cloud Spanner, a fully managed database service that brought the unique capabilities of Spanner to every organization, allowing them to deliver always-on experiences at any scale from thousands to millions of active users across the world without sacrificing consistency.
That took us down another road: innovating to make it easy for all enterprises to build data-driven applications using Cloud Spanner. This meant adding enterprise features such as CMEK and access approval, fine-grained access control, backup and restore, and point-in-time recovery (PITR). We also added VPC Service Controls support and compliance certifications and necessary approvals so that Spanner can be used for workloads requiring ISO 27001, 27017, 27018, PCI DSS, SOC1|2|3, HIPAA and FedRamp. More recently, we added granular instance sizing, a PostgreSQL interface, and free trial instances to lower the barrier to entry for Spanner, and make it accessible for any developer and for any application, big or small.
Speaking of small, one of the interesting things we discovered along the way is that Spanner isn’t only a great fit for huge mission-critical applications – we also use it internally for smaller applications and internal tools, secure in the knowledge that the data is safe, available, and nowhere near any limits. That’s the thing about infrastructure innovations: at their best, they aren’t only about scale, they’re about extracting complexity from applications and presenting the solution in an easy-to-use package.
As we have incorporated solutions to those problems into Spanner, customers have dramatically simplified their applications. For example, application teams often created a dedicated “storage” tier to provide higher-level features like indexes, transactions, or data synchronization needed to support their business logic. Spanner enables application teams to hollow out or even completely eliminate these tiers for significant cost and maintenance savings. We have also seen big reliability dividends: rather than each application discovering the corner cases of transactions, consistency or high availability architectures one at a time, Spanner’s customers are all able to benefit from a rigorously designed and battle-tested implementation out of the gate.
“At their best, infrastructure innovations aren’t only about scale, they’re about extracting complexity from applications and presenting the solution in an easy-to-use package.”
The tradeoff between the scale limitations of relational databases and the functionality limitations of NoSQL systems is really familiar, and the software world has learned to build great applications within those confines. But we never tire of seeing great applications transform as Spanner melts the distinction away.
Get started with Spanner
It has never been easier to try out Spanner. With a free trial instance, you can try out Spanner at no cost for 90 days. You can even prototype an entire application for free on Google Cloud by using the Spanner free trial along with the free tier offered by other Google Cloud products such as Compute Engine and BigQuery.
Create a 90-day Spanner free trial instance. Try Spanner free.
Take a deep dive into the new trial experience and learn more about Spanner.
3696
Of your peers have already watched this video.
61:30 Minutes
The most insightful time you'll spend today!
How to Move From Redshift to BigQuery Easily
Enterprise data warehouses are getting more expensive to maintain. Traditional data warehouses are hard to scale and often involve lots of data silos. Business teams need data insights quickly, but technology teams have to grapple with managing and providing that data using old tools that aren’t keeping up with demand. Increasingly, enterprises are migrating their data warehouses to the cloud to take advantage of the speed, scalability, and access to advanced analytics it offers.
With this in mind, we introduced the BigQuery Data Transfer Service to automate data movement to BigQuery, so you can lay the foundation for a cloud data warehouse without writing a single line of code. Earlier this year, we added the capability to move data and schema from Teradata and S3 to BigQuery via the BigQuery Data Transfer Service. To help you take advantage of the scalability of BigQuery, we’ve now added a service to transfer data from Amazon Redshift, in beta, to that list.
Data and schema migration from Redshift to BigQuery is provided by a combination of the BigQuery Data Transfer Service and a special migration agent running on Google Kubernetes Engine (GKE), and can be performed via UI, CLI or API. In the UI, Redshift to BigQuery migration can be initiated from BigQuery Data Transfer Service by choosing Redshift as a source.
The migration process has three steps:
- UNLOAD from Redshift to S3—The GKE agent initiates an UNLOAD operation from Redshift to S3. The agent extracts Redshift data as a compressed file, which helps customers minimize the egress costs.
- Transfer from S3 to Cloud Storage—The agent then moves data from Amazon S3 to a Cloud Storage bucket using Cloud Storage Transfer Service.
- Load from Cloud Storage to BigQuery—Cloud Storage data is loaded into BigQuery (up to 10 million files).
Watch the video to learn more!
Revolutionizing Healthcare Operations with Data Engine Accelerators
1432
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Healthcare leaders are increasingly challenged to drive operational improvements throughout their facilities, and inefficiencies can cost organizations both time and money and impact patient outcomes. Additionally, staffing shortages and employee burnout remain a major concern in healthcare. Addressing these challenges can help healthcare providers improve organizational operations, patient experiences and care. However, the information that they need to solve these problems is siloed and challenging to access, buried deep in the patient record or spread across IT systems that don’t speak to one another.
To generate a longitudinal patient view and improve patient care, healthcare providers need to reduce fragmentation, unify, and standardize data across siloed systems of records, care facilities and ultimately make it interoperable. This is the heart of Google Cloud’s Healthcare Data Engine.
In partnership with customers, we are prioritizing a ‘use case’ approach to accelerate the adoption of data interoperability. We initially introduced these accelerators a few months ago, and the response has been overwhelmingly positive. Since then, we have been incorporating feedback from early adopters and making improvements to ensure that these accelerators deliver the best possible value to our customers.
The accelerators are deployable use-case configured ‘starter-kits’ within Healthcare Data Engine designed to quickly surface tailored insights with the goal of enabling higher quality of care. Focused on patient flow, transition of care, and social determinants of health, the accelerators provide aggregated data and visualizations to help customers understand a broad range of facility efficiency metrics, patient insight and health equity measures across their patient populations:
- Patient Flow Explorer: Enables health systems to surface a range of facility efficiency metrics to identify trends and potential drivers of bottlenecks, manage patient admissions, transfers and length of stay, while optimizing departmental capacity management.
- Transition of Care Explorer: Provides medical professionals a consolidated view into a patient’s detail and medical history, illustrating their journeys in different settings within the health system and helps caregivers prioritize patient treatment.
- Social Determinants of Health Explorer: Leverages Healthcare Data Engine and social determinants of health data to identify at risk populations of patients in near-real-time and improve care in under-served communities.
The first three accelerator use cases were designed in collaboration with industry healthcare leaders including Highmark Health, Lifepoint Health, and others, helping organizations address common industry challenges.
“Interoperability is at the heart of Highmark Health’s Living Health strategy and we intend to leverage our blended structure of payor and provider to deliver a differentiated health experience for our patients and members,” said Richard Clarke, chief data and analytics officer, Highmark Health. “Healthcare Data Engine is a central component to enabling that strategy and we are excited about the new accelerators being announced today as they will speed up time-to-value for our members.”
“Google Cloud’s solution-oriented approach brings the best of technology and healthcare together to help improve quality, increase access and ensure equitable care for patients no matter where they live,” said Jessica Beegle, senior vice president and chief innovation officer of Lifepoint Health. “Instead of giving us building blocks that need to be assembled, they are delivering custom-built solutions to help us efficiently tackle key problems in our markets and provide more useful data for our clinical teams to take better care of their patients. Lifepoint is proud to partner with Google Cloud and bring the best of Silicon Valley to communities of all sizes across the United States.”
At Google Cloud, we will continue our partnership with healthcare leaders to identify the most impactful Healthcare Data Engine use cases to support, to help organizations accelerate their data transformations to bring better, more timely care to patients and communities. To learn more about Healthcare Data Engine and other solutions tailored to the healthcare and life sciences space, visit cloud.google.com/healthcare.
ActivStat: Enhancing Live Sports Broadcast with Real-time Stats and Metrics
5100
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Millions of people annually view esports, as top players and teams compete in thrilling, fast-paced tests of reflexes, strategy, and teamwork. Fans are a diverse group, sharing a passion for action. A new, revolutionary project we’re working on with Call of Duty League just made that action a lot better.
ActivStat brings fans, players, and commentators the power of competitive statistics in real-time—stats that matter not only to the game at hand, but also for a full roster of competitors globally. Using ActivStat, live broadcasts will soon be enhanced with more depth and color-of-play while they’re happening, building excitement and adding to the overall experience.
Technically, ActivStat is an entirely new capability for esports. It’s a constantly updating catalog of statistics that is sourced, analyzed, updated, and delivered in an easy-to-consume way across global-scale computing systems, with a latency of milliseconds or seconds, rather than minutes or hours. By comparison, many of these stats today are available to fans after a day or more of processing.
Call of Duty League plans to begin rolling out ActivStat during the 2021 season. The initial rollout will include critical information like player and team standings and winning ratios across multiple aspects of virtual combat—including ultimately what these numbers mean for rankings. ActivStats are delivered both in raw statistics and via visualizations and graphics, providing commentators with fast access to the types of insights fans crave.
For engineers at both companies, building the service has been an epic success all its own. Due to a sponsorship with Call of Duty signed last February, our dedicated game engineering team quickly innovated a new solution incorporating high-speed networking, data pipelines from multiple cloud sources, and data warehousing to create a user-friendly dashboard. Real data was flowing into commentator dashboards by April.
Esports are more complex to cover in many ways than regular sports. Instead of a well-defined physical playing field (often a simple rectangular space), multiplayer games involve complex and sprawling virtual environments that can be the size of a large campus or airport with multiple levels and hidden locations.
Gameplay between competitors happens across many of these locations simultaneously. In addition, competitors also each choose their own configurations of equipment, known as “loadouts,” which can dramatically affect gameplay and strategy. All of this additional complexity in online gaming involves data that needs to be captured, analyzed, and communicated to the fans in insightful ways.
Two Google Cloud technology capabilities play central roles in the operations of ActivStat. BigQuery—a planet-scale data warehouse that can store and query petabytes of information in real time—is the foundation of the ActivStat platform for gathering and summarizing millisecond-level statistics. Looker, an intuitive analytic dashboard, surfaces those insights to commentators in an easy-to-use, real-time dashboard that enables the commentators to speak to compelling insights and statistics in sync with the gameplay as it is happening in the live broadcast.
While the initial release of ActivStat for this season of Call of Duty League provides compelling and powerful capabilities, it’s only the beginning of this Call of Duty League-Google Cloud co-innovation partnership. Our future vision is mapping gameplay hotspots on the field, and predicting where to place cameras with machine learning as the match evolves–enabling broadcast producers to build excitement for fans by always being in the middle of the best action. Call of Duty League and Google will also look to drive statistics and metrics directly into the broadcast feed.
The implications of the real-time statistical capabilities of ActivStat go beyond gaming and esports. Historically, gaming has been at the leading edge of what computer processing, computer graphics, wide-area networking, data analysis, and insight can do. The real-time data ingestion and output used in ActivStat could one day be useful powering live broadcasts in other types of sporting events, as well as blending live video feeds and data to support use cases in media & entertainment, healthcare, finance, manufacturing, and other verticals. We’re excited about the potential for this bold new solution and partnership between Call of Duty League and Google in esports and beyond.
To learn more about Call of Duty League-Google Cloud partnership, visit our press release.
Enhancing Data Governance through Automation with Dataplex and BigLake
2996
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Unlocking the full potential of data requires breaking down the silo between open-source data formats and data warehouses. At the same time, it is critical to enable data governance team to apply policies regardless of where the data happens, whether – on file or columnar storage.
Today, data governance teams have to become subject matter experts on each storage system the corporate data happens to reside on. Since February 2022, Dataplex has offered a unified place to apply policies, which are propagated across both lake storage and data warehouses in GCP. Rather than specifying policies in multiple places, bearing the cognitive load of translating policies from “what you want the storage system to do” to “how your data should behave” Dataplex offers a single point for unambiguous policy management. Now, we are making it easier for you to use BigLake.
Earlier this year, we launched BigLake into general availability, BigLake unifies data fabric between Data Lakes and Data Warehouses by extending BigQuery storage to open file formats. Today, we announce BigLake Integration with Dataplex (available in preview). This integration eliminates the configuration steps for the admin taking advantage of BigLake and managing policies across GCS and BigQuery from a unified console.
Previously, you could point Dataplex at a Google Cloud Storage (GCS) bucket, and Dataplex will discover and extract all metadata from the data lake and register this metadata in BigQuery (and Dataproc Metastore, Data Catalog) for analysis and search. With the BigLake integration capability, we are building on this capability by allowing an “upgrade” of a bucket asset, and instead of just creating external tables in BigQuery for analysis – Dataplex will create policy-capable BigLake tables!
The immediate implication is that admins can now assign column, row, and table policies to the BigLake tables auto-created by Dataplex, as with BigLake – the infrastructure (GCS) layer is separate from the analysis layer (BigQuery). Dataplex will handle the creation of a BigQuery connection and a BigQuery publishing dataset and ensure the BigQuery service account has the correct permissions on the bucket.
But wait – there’s more.
With this release of Dataplex, we are also introducing advanced logging called governance logs. Governance logs allow tracking the exact state of policy propagation to tables and columns – adding an additional level of detail going beyond the high-level “status” for the bucket and into fine-grained status and logs for tables, columns.
What’s next?
- We have updated our documentation for managing buckets and have additional detail regarding policy propagation and the upgrade process.
- Stay tuned for an exciting roadmap ahead, with more automation around policy management.
For more information, please visit:
Google Introduces BigQuery Connector for SAP to Power Customers’ Data Analytics Strategy
5477
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Google Cloud has a genuine passion for solving technology problems that make a difference for our customers. With the release of our BigQuery Connector for SAP, we’re taking a another big step towards solving a major challenge for SAP customers with a quick, easy, and inexpensive way to integrate SAP data with BigQuery, our serverless, highly scalable, and cost-effective multi cloud data warehouse designed for business agility.
Solving for simplified data integration
Like most businesses today, SAP customers are eager to unlock the immediate insights and opportunities within their ever-growing stores of business data. However, many are discovering just how hard it can be to take the first step in any modern, cloud-enabled data analytics strategy: combining SAP data with other cloud-native, and enterprise data sets in real-time and at scale. According to a 2020 SAPInsider study, more than half of SAP customers surveyed said data integration was their top analytics pain point. These companies urgently need a rapid, sustainable, cost-effective and scalable way to integrate SAP data with modern cloud data analytics solutions.
The BigQuery Connector for SAP gives our customers a solution: a fast, simple, cost-effective and massively scalable way to make SAP data fully accessible within BigQuery by leveraging customers’ existing SAP Landscape Transformation Replication Server (SLT) tooling and skill sets. It’s the first SAP SLT direct near real-time connector for BigQuery without the need to set up additional infrastructure or third-party middleware, and can be deployed using a variety of embedded or stand-alone deployment options. In fact, most customers can install the BigQuery Connector for SAP in less than an hour—a remarkably easy way to start working with our industry-leading analytics solution that delivers proven and quantifiable business advantages for customers. Additionally, the BigQuery Connector for SAP is not restricted to customers who have deployed their SAP applications on Google Cloud. Customer’s who are running their SAP applications on-premise, or on any cloud, can also deploy and realize the analytical benefits of the solution.
Designing a solution with customer requirements and investments in mind
When the Google Cloud team started work on an analytics data integration tool for our SAP customers, we began with a set of requirements designed to root out the usual sources of cost and complexity. These included:
- The need for real-time performance with deltas replicated in milliseconds
- The ability to integrate data from almost any SAP Netweaver based application running today, regardless of its deployment location (on premises, any cloud, Google Cloud)
- Automatic BigQuery data type mapping with minimal transformation required
- Generation of target tables in BigQuery directly from source, if required
- Application layer integration that avoids the issues of direct database access
- Leveraging customers’ existing SAP skillsets, change data capture, and infrastructure
An important step towards meeting these requirements came when Alphabet, Google’s parent company, decided to leverage SAP SLT as a foundation for developing direct data replication between SAP and BigQuery for its internal corporate landscape. SLT as part of SAP’s strategic Business Technology Platform, supports real-time replication of data from SAP or third-party systems to SAP HANA, however, one of its limitations was direct integration with targets like BigQuery.
SAP SLT was a logical foundation for developing the connector for several reasons:
- It’s widely adopted among SAP customers who likely already leverage SLT for SAP analytics data integration
- It works with almost every non-SaaS SAP application environment running today
- It supports real-time replication performance at massive scale
It was an obvious choice for the Alphabet engineering team who saw immediate value from integrating SAP with BigQuery.
“The BigQuery Connector for SAP has enabled fast, low latency data replication for billions of records from 500+ tables of our most critical financial and supply chain data. Now in one cost-effective BigQuery data lake, this ERP data can be combined with other data sources for previously impossible real-time analytics and ML use cases. This allows us to drive much deeper strategic insights that support business and operational excellence, management and P&L reporting and more.”—Anil Nagalla, Sr. Engineering Director, Financial Systems, Google
SAP data integration with BigQuery enables new value
By leveraging SAP SLT, the BigQuery Connector for SAP can integrate real-time data streams from any SAP system—while also taking advantage of customers’ existing SAP investments and skillsets.
At the same time, the BigQuery Connector for SAP does a lot of heavy lifting on its own. For example, it automatically handles the complex, multi-step process of transforming SAP data types for use in BigQuery—mapping data-type transitions between the SAP and BigQuery environments, creating a target table schema on BigQuery for the transformed data types, building the target BigQuery table, and even adapting as new data types appear in your SAP environment.
For teams that want to fine-tune the BigQuery Connector for SAP’s automated recommendations, the connector supports additional levels of customization and choice. But if you simply want to get the job done and give your data analytics team greater support for their high-value work, then you’ll love just how quickly and easily the BigQuery Connector for SAP turns the complicated work of data integration and performance to process large volumes of data into a done deal. By integrating enterprise data sets in real time, customers can drive differentiated value and unlock new insights and actions that drive a competitive advantage.
The BigQuery Connector for SAP really shines as an enabling tool that transports and transforms your SAP data to power analytics solutions enabled by accelerators like the Google Cloud Cortex Framework: a comprehensive set of reference architectures, deployment accelerators, and integration services designed to give SAP customers a fast and seamless path to value with their data analytics investments. Simply put, the more SAP data you make available within Google Cloud, the easier it is to get meaningful—and often game-changing—insights from these solutions.
Learn more about the BigQuery Connector for SAP
Ready to get started with your own SAP data analytics strategy on Google Cloud? Install the Google Cloud BigQuery Connector for SAP, and discover a faster, simpler, more sustainable way to power your company’s data analytics strategy.
More Relevant Stories for Your Company
Demystifying Transactional Locking in Cloud Spanner
Cloud Spanner is a fully managed relational database with unlimited scale, strong consistency, and up to 99.999% availability. It is designed for highly concurrent applications that read and update data, for example, to process payments or for online game play. To ensure the consistency across multiple concurrent transactions, Cloud Spanner
Skincare Firm Scales 4X in Minutes with SAP on Google Cloud
As the number one skincare brand in the United States, Rodan + Fields must support its team of over 300,000 of independent contractors as well as work to ensure a really personalized experience for customers. To keep pace with the company’s growth, Rodan + Fields realized it needed a more
How Pantheon Improved Performance and Reliability by Moving to Google Cloud
Google Cloud Results Improves performance and reliability, enabling Pantheon to serve larger customersSupports 99.95% uptime across 200,000+ websitesReduces cloud infrastructure costs by 40%Enables future analytics offerings based on machine learning and big data analytics Nearly every business needs a web presence—but the vast majority of companies don’t want to be
How Connected-Stories Uses BigQuery and AI/ML to Craft Personalized Ad Experiences
Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery In the field of producing engaging video content such as ads, many marketers ignore the power of data to improve their creative efforts to meet the consumers' need for
