Editor’s note: The video platform Vimeo leverages managed database services from Google Cloud to serve up billions of views around the world each day. Here’s how they deliver a consistent user experience across all their content.

Vimeo is the world’s leading video software solution, powering hundreds of thousands of new uploads every day and billions of views. At our scale, it’s paramount that we retain a seamless user experience across all of the videos hosted, shared, and viewed on our platform. With managed database services from Google Cloud, we’ve been able to deliver a consistent and reliable user experience no matter where our users and their audiences are.

https://youtube.com/watch?v=ggpAUgEL2iE%3Fenablejsapi%3D1%26

Outgrowing the data center and gaining more scale

Our first exposure to Google Cloud was the excellent performance between Google Compute Engine (GCE) and Google Cloud Storage (GCS), which we used to design a really high quality playback and upload environment for our customers’ videos. At first we were simply creating cloud-based packaging and delivery of their content, but the bulk of our application ran on an on-premises MySQL database. Seeing the gains from our initial investment in GCE and GCS, we decided to fully migrate to Google Cloud. 

We started by lifting and shifting our workloads from our on-premises deployments to GCE. But we quickly realized we had to redesign our application to deliver higher availability and responsiveness for an exceptional user experience, which led us to Cloud Spanner, a distributed SQL database management and storage service. With Spanner, we found all the benefits of a relational semantics database with global scale, allowing us to add more nodes with the push of a button. 

Today we have a multi-region instance with 99.999% uptime, so we’re confident it can handle our video workloads. We use about 16 Spanner nodes to cover approximately 50.8 billion rows and about 4.5 terabytes of storage/disk space. And with the launch of Video Library, we have the scale we need to fundamentally change the way organizations handle content. 

Using Spanner as a time machine during migration

During the initial migration, we wanted to ensure an accurate data transfer with as few errors or bugs as possible. When we moved data from MySQL tables into Spanner, we ran spot analyses to check for potential problems. If the data didn’t match, Spanner allowed us to revert to the original in a single click. Through Spanner we could check stale timestamps or past users. It almost functioned like a time machine by offering point-in-time recovery functionality.

Serving up any video at any time

We work with a long tail of content. Depending on how popular a video is, we take a different approach to storing it. We cache the most popular material so it’s immediately accessible and use Cloud Spanner to ensure it’s highly available across multiple regions. On the flip side, we use Cloud SQL for some lower volume satellite services that support other components of the application that don’t require the same guarantees or performance that Cloud Spanner provides.  

As our millions of users continue to upload their own videos, Spanner also acts as a video metadata indexing and cataloging service. When we need to play back a particular video, we can quickly and easily find it in a sea of content. It’s crucial that our database is highly available and reliable so the playback experience is seamless for our users. 

To learn more about Vimeo, visit our site.

Learn more about how your organization can use Cloud Spanner.

Intro to Cloud Spanner

Cloud Spanner is an enterprise-grade, globally distributed, externally consistent database that offers unlimited scalability and industry-leading 99.999% availability. It requires no maintenance windows and offers a familiar PostgreSQL interface. It combines the benefits of relational databases with the unmatched scalability and availability of non-relational databases. 

As organizations modernize and simplify their tech stack, Spanner provides a unique opportunity to transform the way they think about and use databases as part of building new applications and customer experiences.

But choosing a database for your workload can be challenging; there are so many options in the market and each one has a different onboarding and operating experience. At Google Cloud we know it’s hard to navigate this choice and are here to help you. In this blog post, I want to bust the seven most common misconceptions that I regularly hear about Spanner so that you can confidently make your decision.

Myth #1 Only use Spanner if you have a massive workload

The truth is that Spanner powers Google’s most popular, globally available products, like YouTube, Drive, and Gmail, and has enabled many large scale transformations including that of Uber, Niantic and Sharechat. It is also true that Spanner processes more than 1 Billion queries per second at peak.

At the same time, many customers also use Spanner for their smaller workloads (both in terms of transactions per second and storage size) for availability and scalability reasons. For example, Google Password Manager has small workloads that run on Spanner. These customers cannot tolerate downtime, require high availability to power their applications and seek scale insurance for future growth scenarios.

Limitless scalability with the highest availability is critical in many industry verticals such as gaming and retail, especially when a newly launched game goes viral and becomes an overnight success or when a retailer has to handle a sudden surge in traffic due to a  Black Friday/Cyber Monday sale.  

Regardless of workload size, every customer on the journey to the cloud wants the benefits of scalability and availability while reducing the operational burden and the costs associated with patching, upgrades and other maintenance.

Myth #2 Spanner is too expensive

The truth is, when looking at the cost of a database, it is better to consider Total Cost of Ownership (TCO) and the value it offers rather than the raw list price. We deliver significant value to our customers starting at this price including critical things like availability, price performance, and reduced operational costs. 

• Availability: Spanner provides high availability and reliability by synchronously replicating data. When it comes to Disaster Recovery, Spanner offers 0-RPO and 0-RTO for zonal failures in case of a regional instance and regional failure in case of multi-regional instances. Less downtime, more revenue!
• Price-performance: Spanner offers one of the industry’s leading price-performance ratios which makes it a great choice if you are running a demanding, performance sensitive application. Great customer experiences require consistent, optimal latencies!
• Reduced operational cost: With Spanner, customers enjoy zero downtime upgrades and schema changes, and no maintenance windows. Sharding is automatically handled so the challenges associated with scaling up traditional databases don’t exist. Spend more time innovating, and less time administering!
• Security & Compliance: By default, Spanner already offers encryption for data-in-transit via its client libraries and for data-at-rest using Google-managed encryption keys. CMEK support for Spanner lets you now have complete control of the encryption keys. Spanner also provides VPC Service Controls support and has compliance certifications and necessary approvals so that it can be used for workloads requiring ISO 27001, 27017, 27018, PCI DSS, SOC1|2|3, HIPAA and FedRAMP.

With Spanner, you have peace of mind knowing that your data’s security, availability and reliability won’t be compromised.

And best of all, with the introduction of Granular Instance Sizing, you can now get started for as little as $65/month and unlock the tremendous value spanner offers.

Pro tip : Use the auto-scaler to right size your Spanner instances. Take advantage of TTL to reduce the amount of data stored.

Myth #3 You have to make a trade off between scale, consistency, and latency

The truth is, depending on the use case and instance configuration, users can use Spanner such that they don’t have to pick between consistency, latency and scale.

To provide strong data consistency, Spanner uses a synchronous, Paxos-based replication scheme, in which replicas acknowledge every write request. A write is committed when a majority of the replicas (e.g 2 out of 3), called a quorum, agree to commit the write. In the case of regional instances, the replicas are within the region and hence the writes are faster than in the case of multi-region instances, where the replicas are distributed across multiple regions. In the latter case, forming a quorum on writes can result in slightly higher latency. Nevertheless, Spanner multi-regions are carefully designed in geographical configurations that ensure that the replicas can communicate fast enough and write latencies are acceptably low.

A read can be served strong (by default) or stale. A strong read is a read at a current timestamp and is guaranteed to see all the data that has been committed up until the start of the read. A stale read is a read executed at a timestamp in the past. In case of a strong read, the serving replica ​​will guarantee that you will see all data that has been committed up until the start of the read. In some cases, this means that the serving replica has to contact the leader to ensure that it has the latest data. In case of a multi-region instance where the read is served from a non-leader replica, this would mean that read latency can be slightly higher than if it was served from a leader region. Stale reads are performed over data that was committed at a  timestamp in the past and can, therefore, be served at very low latencies by the closest replica that is caught up until that timestamp. If your application is latency sensitive, stale reads may be a good option and we recommend using a stale read value of 15 seconds. 

Myth #4 Spanner does not have a familiar interface

The truth is that Spanner offers the flexibility to interact with the database via a SQL dialect based on ANSI 2011 standard as well as via a REST or gRPC API interface, which are optimized for performance and ease-of-use. In addition to Spanner’s interface, we recently introduced a PostgreSQL interface for Spanner, that leverages the ubiquity of PostgreSQL to meet development teams using an interface that they are familiar with. The PostgreSQL interface provides a rich subset of the open-source PostgreSQL SQL dialect, including common query syntax, functions, and operators. It supports a core collection of open-source PostgreSQL data types, DDL syntax, and information schema views. You get the PostgreSQL familiarity, and relational semantics at Spanner scale. 

Learn more about our PostgreSQL interface here.

Myth #5 The only way to get observability data is via the Spanner Console

​​The truth is that Spanner client libraries support OpenCensus Tracing and Metrics, which gives insight into the client internals and aids in debugging production issues. For instance, client-side traces and metrics include sessions and transactions related information. 

Spanner also supports the OpenTelemetery receiver, which provides an easy way for you to process and visualize metrics from Cloud Spanner System tables, and export these to the Application Monitoring (APM) tool of your choice. This could be either an open source combination of a time-series database like Prometheus coupled with a Grafana dashboard, or it could be a commercial offering like Splunk, Datadog, Dynatrace, NewRelic or AppDynamics. We’ve also published reference Grafana dashboards, so that you can debug the most common user journeys such as “Why is my tail latency high” or “Why do I see a CPU spike when my workload did not change”. Here is a sample docker service, to show how the Cloud Spanner receiver can work with Prometheus exporter and Grafana dashboards.

We are continuing to embrace open standards, and continuing to integrate with our partner ecosystem. We also continue to evolve the observability experience offered by the Google console so that our customers get the best experience wherever they are. 

Myth #6 Spanner is only for global workloads requiring copies in multiple regions 

The truth is that, while Spanner offers a range of multi-region instance configurations, it also offers regional configuration in each GCP region. Each regional node is replicated in 3 zones within the region, while a multi-regional node is replicated at least 5 times across multiple regions. A regional configuration offers 4 nines of availability and protection against zonal failures.

Typically, multi-regional instance configurations are indicated if your application runs workloads in multiple geographical locations or your business needs 99.999% of availability and protection against regional failures. Learn more here.

Myth #7 Spanner schema changes require expensive locks

The truth is that Spanner never has table level locks. Spanner uses a multi-version concurrency control architecture to manage concurrent versions of schema and  data allowing ad-hoc and online qualified schema changes that do not require any downtime, additional tools, migration pipelines or complex rollback/backup plans. When issuing a schema update you can continue writing and reading from the database without interruption while Spanner backfills the update, whether you have 10 rows or 10 billion rows in your table.

The same mechanism can be used for Point-in-time recovery (PITR) and snapshot queries using stale reads to restore both schema and the state of data at a given query-condition and timestamp up to a maximum of seven days.

Now that we’ve learned the truth about Cloud Spanner, I invite you to get started – visit our website.

Over the past two years, the general population has become more acquainted with cryptocurrencies and the first iterations of NFTs, which were among the earliest use cases for blockchain technology. This public awareness and participation has led to a growing interest in, and demand for, Web3 technology at the enterprise level.

But building trust in a new wave of technology, especially in large organizations, doesn’t happen overnight. That is why it’s critical for Web3 technologists to bring the broader benefits, use cases, and core capabilities of blockchain to the forefront of the conversation. If businesses don’t understand how this new technology can help them, how can they prioritize it among competing tech plans and resources? And without baseline protocols that account for privacy, confidential data, and IP, how can they future-proof a business?

Answering these questions and delivering trustworthy infrastructure is exactly why Scott Purcell and I founded Fortress Web3 Technologies — to bring about the next wave of Web3 utility. The company’s goal is to provide infrastructure that eliminates barriers to Web3 adoption with RESTful APIs and widgetized services that enable businesses to quickly launch and scale their Web3 initiatives.

Our tools include embeddable wallets for NFTs and fungible rewards tokens; NFT minting engines; and core financial services. These include payments, compliance, and crypto liquidity via our wholly-owned financial institution, Fortress Trust. Being overseen by a chartered, regulated entity ensures privacy, compliance and business continuity.

Fortress chose Google Cloud to help usher in this new-wave technology because no other cloud provider is better suited to helping regulated industries get up to scale on our Web3 infrastructure and blockchain technology. I’ll get into more specifics below, but at the highest level: IPFS (the current standard distributed storage) is going to face major resistance when it comes to industries that are heavily regulated or deal in ownership rights. By leveraging Google Cloud, which has critical certifications such as HIPPA, Department of Defense, ISO, and Motion Picture, we’re striking the appropriate balance between decentralization and centralization, using the best of both technologies.

The Fortress Vault on Google Cloud is a huge and necessary step forward as the first ever NFT-database solution to protect intellectual property, confidential documents, and other electronic records. It represents the first technology that marries privately stored content with the accessibility, privacy, portability, and provenance that blockchain provides.

Understanding Non-Fungible Tokens (NFTs)

An NFT is not an expensive jpeg. From a technical point of view, an NFT is a unique key stored in a distributed and trustless ledger we call a blockchain. This blockchain token is uniquely identifiable from any other token and acts as a digital key to authenticate ownership and unlock data held in a database.

While different blockchains have adopted different standards, Ethereum standards are a good proxy to represent overall concepts. Going back to the primitives, if you read the EIP 721 proposal, metadata is explicitly optional. While today’s NFT hype has indeed leveraged that technology to monetize and distribute digital art, the potential of blockchain is in the ability to digitally represent ownership of a wide variety of different asset classes on a decentralized ledger.

Unique, non-fungible tokens are not a new concept. We use them every day in technical systems for things like authentication, database keys, idempotency, and much more. Now, thanks to blockchain technology, you can take those out of their walled gardens and into an open platform that can lead to transformational utility and applications.

Take real estate, for example. Instead of a paper-based title documenting you as the owner of your home, imagine that the title is tokenized with an NFT on a blockchain. Any platform could cryptographically verify the authenticity of that form of title along with its provenance in real time and confirm that you’re the rightful owner of that property.

But, perhaps you don’t want the title of your property visible to others, nor the associated permits, tax documents, architectural drawings, contractor lists, and other documents. Maybe you just want banks, insurance companies, and others to be able to confirm that you are indeed the owner without revealing the details of those records. The NFT metadata records immutable public-facing provenance, while the underlying data remains private and protected using Fortress Vault on Google Cloud.

Apply that same utility to other sensitive information such as medical records, intellectual property, estate documents, corporate contracts, and other confidential information and it’s easy to see how enterprises are just now exploring how to hold traditional assets as NFTs.

Fortress Vault: Intellectual Property, Confidential Documents, and Other Electronic Records

What NFTs and Web3 have been lacking is the ability to make the tokenized data accessible exclusively by the owner — and only the owner. NFTs are a digital key to unlock everything ranging from music and event tickets, to real estate deeds and healthcare records, to estate documents, and to everything in the world that’s digital.

This is why we created the Fortress Vault. When building it, we had to make a fundamental decision: Either go with a distributed and permissionless storage protocol like IPFS, filecoin, or other blockchain-based database offerings, or work with an industry-leading cloud platform that understands data integrity and is establishing itself as the leader in the space.

Ultimately, we chose Google Cloud for its industry-leading object storage, professional management, fault tolerance, and myriad of certifications for architecture and data integrity.

Some of the challenges faced when vaulting a vast variety and quantity of digital content at scale include:

• Balancing data availability versus cost of storage
• Data redundancy
• Long term archival needs
• Business continuity
• Flexibility to meet current and future needs of the rapidly evolving Web3 industry.

Google Cloud is the clear leader across all of these pain points. The object lifecycle management of Google Cloud Storage enables efficient transition between storage classes when either the data matures to a certain point or it’s updated with newer files. Content in the Fortress Vault can range from on-demand data to long-term uses, such as estate planning documents that won’t be accessed for 30 years.

When storing NFT data, robust disaster recovery is table stakes. We quickly gravitated to the automatic redundancy options and multi-region storage buckets that let us customize where we store our data without massive devops and management overhead. By leveraging Google Cloud, we can offer industry leading retention, redundancy, and integrity for our customers’ NFT content.

Working with a leader in data storage was key to making this a reality. Additionally, Google Cloud shares our vision of bringing every industry forward into the world of Web3. We are both focused on building the critical infrastructure that allows everyone from Web3 native companies to Fortune 500 brands navigate the strategic shift to blockchain technology.

Why Web3 Matters

“Web3” is shorthand for the “third wave” of the internet and the technological innovation that brought us here.

Web 1 — the earliest internet — democratized reading and access to information, opening the doors to mass communication. Web 2 expanded on that with the ability to read and “write.” It democratized publishing by letting people directly engage in producing information through blogs, social media, gaming, and contributions to collective knowledge.

Web 3 expands our technological capabilities even more with the ability to read, write, and “own.” With blockchain, we can now establish clear provenance with visibility into the origination of ownership of any tokenized asset, and we can see the chain of ownership. We can rely on this next-generation technology to track, authenticate, protect, and keep a ledger of our assets.

With the Fortress Vault on Google Cloud, we have the capability to ensure the integrity of non-public data while making it accessible via NFTs. This is a game changer for Web3 adoption, particularly in industries like music, event ticketing, gaming, finance, transportation, real estate, and healthcare. Every industry can benefit from the ability to tokenize assets on blockchain technology without leaving the trusted safety of Google Cloud data storage.

The market for NFTs is everyone. And the Fortress Vault on Google Cloud is the technology evolution that makes it possible for Web3 innovators to confidently build, launch, and scale their initiatives across every industry imaginable.

When Blue Apron had issues running its data warehouse on another cloud provider, it built an analytics platform using Looker and Google BigQuery to enable faster business decisions about food inventory.

Blue Apron’s mission is to make incredible home cooking accessible to everyone. Launched in 2012, Blue Apron is reimagining the way that food is produced, distributed, and consumed.

Industries: Retail & Consumer Goods

Location: United States

Google Cloud Results

• Enables near real-time business decisions to better manage food inventory and delivery by reducing query times exponentially
• Helps improve customer service and optimize business processes with faster insights
• Reduces costs while reclaiming up to a week per month of engineering

The popularity of meal kit delivery services has surged in recent years as consumer attitudes toward home cooking and grocery shopping have shifted. As a pioneer in the category, Blue Apron helps its customers create incredible home cooking experiences by sending culinary-driven recipes with high-quality ingredients and step-by-step instructions straight to customers’ doors. Blue Apron also offers a monthly wine subscription service and a la carte culinary tools and products through its marketplace.

If that sounds simple, it isn’t. Ingredients for the meal kits must be sourced at the right time, quality, and price. Orders must be packed efficiently and in exactly the right proportions. Most importantly, meal kits must be delivered to the customer fresh and on time.

To meet these criteria and make data meaningful and intuitive to its managers, one of the tools Blue Apron relies on is Looker, an analytics platform that lets business users explore data and ask sophisticated questions using familiar terms.

A Google Cloud partner and winner of the 2016 Google Cloud Global Partner Award for Solution Innovation, Looker integrates its solution with Google Cloud Platform to help customers modernize their analytics.

Blue Apron previously used Looker with a single database instance hosted on another cloud provider. As data volumes grew and queries became more complex, it became difficult to scale. Blue Apron’s only options were choosing ever-larger server classes and increasing storage throughput by purchasing a higher number of provisioned IOPS. To improve speed, scalability, and cost efficiency, Blue Apron moved its data warehouse to Google BigQuery.

“The combination of Looker and Google BigQuery is powerful, allowing us to get data-hungry analysts essential information much faster,” says Sam Chase, Tech Lead, Data Operations at Blue Apron. “Because we choose to pay by the query, it’s also flexible and cost effective—plus storage is cheap, so we can just put data in and query what we need.”

The analytics platform of the future

When you’re making business decisions about a customer’s dinner, speed matters. Looker takes full advantage of the power of Google BigQuery, making it easy to build a data exploration platform. Blue Apron’s applications publish event data to Kafka—approximately 140 million events per day—and data is then streamed into Google BigQuery, which performs lightning-fast queries on both streamed and static data. Now, business users and analytics teams can make decisions based on near real-time information in Looker, instead of waiting until the next business day for results.

“After we moved to Google BigQuery, query time was reduced exponentially. It’s an astonishing difference, allowing us to run 300 queries per day,” says Sam.

Previously, Blue Apron spent up to a week out of every month optimizing its data warehouse to attempt to improve query performance. With Google BigQuery, all maintenance is handled by Google, reclaiming 25% of up to two engineers’ time. Even when multiple people are using Looker concurrently, query performance never degrades and storage never runs out.

“Because Google BigQuery is architected as a giant, shared cluster, growth is smooth,” says Lloyd Tabb, Founder and CTO of Looker. “Like a race car going from 0 to 120 mph, there are no shift points, just smooth acceleration. To us, it looks like the future.”

An empowering, integrated toolset

Looker takes advantage of aggressive caching and support for date-based table partitioning in Google BigQuery to increase performance, simplify the load process, and improve data manageability. By partitioning data by time, Blue Apron can also take advantage of better long-term storage pricing without sacrificing query performance. When using Google BigQuery with Looker, analysts can easily see how much data is going to be scanned before each query is run.

Blue Apron is also using Looker for Google BigQuery Data Transfer Service to provide actionable analytics for all of the company’s Google marketing data from Google AdWords and DoubleClick by Google in one place to understand campaign performance across channels, saving its data operations team months of work. Using Looker Blocks, marketers can quickly make sense of the data with reports and dashboards, and set alerts when campaign performance hits certain thresholds.

Looker Blocks for Google AdWords and DoubleClick by Google provide all the analysis you’d get straight from the Google console, plus additional value-add analysis that’s impossible to replicate without SQL. Complex metrics such as ROI on ad spend, flexible multi-touch attribution, and predictive lifetime value empower marketers with a better understanding of their customers and where to spend their next dollar.

In addition to these turnkey dashboards and pieces of analysis, marketers can customize views to meet their unique needs and workflows. These capabilities help the Blue Apron marketing team make decisions regarding the allocation of spend to maximize customer acquisition and retention.

“Everyone at Blue Apron is excited about using Google BigQuery with Looker,” says Sam. “Business users and marketers are more empowered to look for answers, instead of waiting for analytics teams. Because users know they can get results rapidly, our business processes are evolving and improving.”

For data cleansing and transformation, Blue Apron uses Google Cloud Dataproc to run fully managed Apache Spark clusters on Google Cloud Platform. It’s also leveraging Google BigQuery integration with Google Workspace to bring data into Google Sheets for further distribution and analysis.

“Transferring data between Google tools is fast because it all happens on the Google network,” says Sam. “We can pull data from Google BigQuery, run transformations with Spark, and then write it back to Google BigQuery. That’s very helpful in providing our business users and data analysts with the richest, most current data.”

A perfect match for better insights

As Blue Apron seeks to expand its reach and deepen its engagement with customers, it is making Google BigQuery and Looker available to more users, providing a high-quality interactive analytics experience. “Our ability to pull a lot of data in and compute fast results affects everyone in our company,” says Sam. “Using Google BigQuery and Looker to iterate quickly and build new models to make our operations more efficient will directly impact our customers.”

For Looker, Google BigQuery represents the next step in data warehouse evolution. “Google BigQuery is a perfect match for Looker, combining easy setup with near infinite scale-out and elasticity,” says Lloyd. “People can make smarter decisions faster that directly benefit their business and customers.”