Today, we’re announcing Datastream, a serverless change data capture (CDC) and replication service, available now in preview. Datastream allows enterprises to synchronize data across heterogeneous databases, storage systems, and applications reliably and with minimal latency to support real-time analytics, database replication, and event-driven architectures. You can now easily and seamlessly deliver change streams from Oracle and MySQL databases into Google Cloud services such as BigQuery, Cloud SQL, Google Cloud Storage, and Cloud Spanner, saving time and resources and ensuring your data is accurate and up-to-date.

“Global companies are demanding change data capture to provide replication capabilities across disparate data sources, and provide a real-time source of streaming data for real-time analytics and business operations,” says Stewart Bond, Director, Data Integration and Intelligence Software Research at IDC.

However, companies are finding it difficult to realize these capabilities because commonly used data replication offerings are costly, cumbersome to set up, and require significant management and monitoring overhead to run flexibly or at scale. This leaves customers with a difficult-to-maintain and fragmented architecture. 

Datastream’s differentiated approach 

Datastream is taking on these challenges with a differentiated approach. Its serverless architecture seamlessly and transparently scales up or down as data volumes shift in real time, freeing teams to focus on delivering up-to-date insights instead of managing infrastructure. It also provides the streamlined customer experience, ease of use, and security that our customers have come to expect from Google Cloud, with private connectivity options built into the guided setup experience. 

Datastream integrates with purpose-built and extensible Dataflow templates to pull the change streams written to Cloud Storage, and create up-to-date replicated tables in BigQuery for analytics. It also leverages Dataflow templates to replicate and synchronize databases into Cloud SQL or Cloud Spanner for database migrations and hybrid cloud configurations. 

Datastream also powers a Google-native Oracle connector in Cloud Data Fusion’s new replication feature for easy ETL/ELT pipelining. And by delivering change streams directly into Cloud Storage, customers can leverage Datastream to implement modern, event-driven architectures.

Customers tell us about the benefits they’ve found using Datastream. That includes Schnuck Markets, Inc., “Leveraging Datastream, we’ve been able to replicate data from our on-premises databases to BigQuery reliably and with little impact to our production workloads. This new method replaced our batch processing and allowed for insights to be leveraged from BigQuery quicker,” says Caleb Carr, principal technologist from Schnuck Markets. “Furthermore, implementing Datastream removed the need for our analytics group to reference on-premises databases to do their work and support our business users.”

Cogeco Communications, Inc. used Datastream to also realize the value of low-latency data access. “Datastream unlocked new customer interaction opportunities not previously possible by enabling low-latency access in BigQuery to our operational Oracle data.” says Jean-Lou Dupont, Senior Director, Enterprise Architecture, Cogeco Communications, Inc. “This streamlined integration process brings data from hundreds of disparate Oracle tables into a unified data hub. Datastream enabled us to achieve this with 10X time and effort efficiency.”

In addition, Major League Baseball (MLB) used Datastream’s replication capabilities to migrate their data from Oracle to Cloud SQL for PostgreSQL. “As we’re modernizing our applications, replicating the database data reliably out of Oracle and into Cloud SQL for PostgreSQL is a critical component of that process,” says Shawn O’Rourke, manager of technology at MLB. “Using Datastream’s CDC capabilities, we were able to replicate our database securely and with low latency, resulting in minimal downtime to our application. We can now standardize on this process and repeat it for our next databases, regardless of scale.”

Our partner HCL has worked with many organizations looking to get more out of their data and plan for the future. “HCL customers across every industry are looking for ways to extract more value out of their vast amounts of data,” says Siva G. Subramanian, Global Head for Data & Analytics at HCL Google Business Unit. “CDC plays a big part in the solutions we offer to our customers using Google Cloud. Datastream enables us to deliver a secure and reliable solution to our customers that’s easy to set up and maintain. CDC is a key and integrated part of Google Cloud Data Solutions.”

“Google Cloud’s new CDC offering, Datastream, is a differentiator for Google among hyperscale cloud service providers, by supporting replication of data from Oracle and MySQL databases into the Google Cloud environment using a serverless cloud-native architecture, which removes the burden of infrastructure management for organizations, and provides elastic scalability to handle real-time workloads,” says Stewart Bond, Director, Data Integration and Intelligence Software Research at IDC.

Datastream under the hood

Datastream reads CDC events (inserts, updates, and deletes) from source databases, and writes those events with minimal latency to a data destination. It leverages the fact that each database source has its own CDC log—for MySQL it’s the binlog, for Oracle it’s LogMiner—which it uses for its own internal replication and consistency purposes. Using Google-native, agentless, high-scale log reader technology, Datastream can quickly and efficiently generate change streams populated by events based on the database’s CDC log while minimizing performance impact on the source database.

Each generated event includes the entire row of data from the database, with the data type and value of each column. The original source data types, whether it’s, for example, an Oracle NUMBER type or a MySQL NUMERIC type, are normalized into Datastream unified types. The unified types represent a lossless superset of all possible source types, and the normalization means data from different sources can easily be processed and queried downstream in a source-agnostic way. Should a downstream system need to know the original source data type, it can perform a quick API call to Datastream’s Schema Registry, which stores up-to-date, versioned schemas for every data source. This also allows for in-flight downstream schema drift resolution as source database schemas change. 

The generated streams of events, referred to as “change streams,” are then written as files, either in JSON or Avro format during preview or in other formats like Parquet in the future, into a Cloud Storage bucket organized by source table and event times. Files are rotated as table schemas change, so events in a single file always have the same schema, as well as on a configurable file size or rotation frequency setting. This way customers can find the best balance between the speed of data availability and the file size that makes the most sense for their business use case.

Through its integration with Dataflow, Datastream powers up-to-date, replicated tables for analytics over BigQuery, and for data replication and synchronization to Cloud SQL and Spanner. Datastream refers to these constantly updated tables as “materialized views.” They are kept up-to-date via Dataflow template-based upserts into Cloud SQL or Spanner, or through consolidations into BigQuery. The consolidations, performed as part of the Dataflow template, take the change streams that are written into a log table in BigQuery, and push those changes into a final table, which mirrors the table from the source.

Datastream offers a variety of secure connectivity methods to sources, so your data is always safe in transit. And with its serverless architecture, Datastream can scale up and down readers and processing power to seamlessly keep up with the speed of data and ensure minimal latency end to end. As data volumes decrease, Datastream automatically scales back down—the result is a “pay for what you use” pricing model, where you never have to pay for idle machines or worry about bottlenecks and delays during data peaks.

Get started with Datastream 

Datastream, now available in preview, supports streaming change data from Oracle and MySQL sources, hosted either on-premises or in the cloud, into Cloud Storage. You can start streaming your data today for $2 per GB of data processed by Datastream. 

To get started, head over to the Datastream area of your Google Cloud console, under Big Data, and click Create Stream. There you can:

1. Initiate stream creation, and see what actions you need to take to set up your source and destination for successful streaming.
2. Define your source and destination, whose connectivity information is saved as connection profiles you can re-use for other streams. Sources support multiple connectivity options, with both private and public connectivity options to suit your business needs.
3. Select the source data you’d like to stream, and which you’d like to exclude.
4. Test your stream to ensure it will be successful when you’re ready to go.

Start your stream and your database’s CDC data will start to flow to your Cloud Storage bucket! From there you can integrate with Dataflow templates to load data into BigQuery, Spanner, or Cloud SQL. Datastream’s preview is supported in us-central1, europe-west1, and asia-east1, with additional regions coming soon.https://www.youtube.com/embed/FZG4w4Vbj38?enablejsapi=1&

Datastream will become generally available later this year, and will soon expand its support to also include PostgreSQL and SQL Server as sources, as well as out-of-the-box integration with BigQuery for easy delivery of up-to-date replicated tables for analytics, and message queues like Pub/Sub for real-time change stream access. 

For more resources to help get you started with change streaming, check out the Datastream documentation.

Your product teams might ask – “Why does it take so long to build a feature or application?” Building applications is a heavy lift due to the technical complexity, which includes the complexity of backend services that are used to manage and store data. Every moment focused on this technical complexity is a distraction from  delivering on core business value. Firestore alters this by having Google Cloud manage your backend complexity through a complete backend-as-a-service! 

Firestore is a serverless, NoSQL document database that unlocks application innovation with simplicity, speed and confidence. 

It acts as a glue that intelligently brings together the complete Google Cloud backend ecosystem, in-app services from Firebase and core UI frameworks & OS from Google.

What is Firestore?

Firestore is a serverless, fully managed NoSQL document database that scales from zero to global scale without configuration or downtime. Here’s what makes Firestore unique:

• Ideal for rapid, flexible and scalable web and mobile development with direct connectivity to the database.
• Supports effortless real time data synchronization with changes in your database as they happen. 
• Robust support for offline mode, so your users can keep interacting with your app even when the internet isn’t available or is unreliable.
• Fully customizable security and data validation rules to ensure the data is always protected
• Built-in strong consistency, elastic scaling, high performance & best in class 99.999% availability 
• Integration with Firebase and Google Cloud services like Cloud Functions and BigQuery, serverless data warehouse.
• In addition to a rich set of Google Cloud service integrations, Firestore also offers deep one-click integrations with a growing set of 3rd party partners via Firebase Extensions to help you even more rapidly build applications.

Document-model database

Firestore is a Document-model database. All of your data is stored in “documents” and then “collections”.  You can think of a document as a JSON object. It’s a dictionary with a set of key-value mappings, where the values can be several different supported data types including strings, numbers or binary values..

These documents are stored in collections. Documents can’t directly contain other documents, but they can point to subcollections that contain other documents, which can point to subcollections, and so on. This structure brings with it a number of advantages. For starters, all queries that you make are shallow, meaning that you can grab a document without worrying about grabbing all the data underneath it. And this means that you can structure your data hierarchically in a way that makes sense to you logically, without having to worry about grabbing tons of unnecessary data. 

How to use Firestore?

Firestore can be used in two modes:

• Firestore in Native Mode:  This mode is differentiated by its ability to directly connect your web & mobile app to Firestore. Native Mode supports up to 10K writes per second, and over a million connections. 
• Firestore in Datastore Mode: This mode supports only server-side usage of Firestore, but supports unlimited scaling, including writes. 

Conclusion

Whatever your application use case may be, if you want to build a feature or an application quickly using Firestore backend-as-a-service. For a more in-depth look into Firestore check out the documentation. https://www.youtube.com/embed/moglAjmwmUQ?enablejsapi=1&

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.

At HSBC, we serve 39 million customers, in-person and online, from consumers to businesses, in 66 countries. We maintain data centers in 21 countries, with more than 94,000 servers. With an on-premises infrastructure supporting our business, we kept running into capacity challenges, which really became an innovation blocker and ultimately a business constraint.

Our teams wanted to do more with data to create better products and services, but the technology tools we had weren’t letting us grow and explore. And that data was growing continually. Just one of our data warehouses had grown 300% from 2014 to 2018.

We had a huge amount of data, but what’s the point of having all that data if we couldn’t get insights and business value from it? We wanted to serve our customers flexibly, in the ways that work best for them. 

We knew moving to cloud would let us store and process more data, but as a global bank, we were moving complex systems that needed to also be secure.

It was a team effort to create the project scope and strategy up front, and it paid off in the end. Our cloud migration now enables us to use an agile, DevOps mindset, so we can fail fast and deliver smaller workloads, with automation built in along the way.

This migration also helped us eliminate technical debt and build a data platform that lets us focus on innovation, not managing infrastructure. Along the way, we invented new technology and built processes that we can use as we continue migrating.

Planning for a cloud move
We chose cloud migration because we knew we needed cloud capabilities for our business to really reach its digital potential. We picked Google Cloud, specifically BigQuery, because it’s super fast over small and large datasets, and because we could use both a SQL interface and Connected Sheets to interact with it.

We had to move our data and its schema into the cloud—without having to manually manage every detail and miss the timelines we had set. Our data warehouse is huge, complex, and mission-critical, and didn’t easily lend itself to fit into existing reference architectures. We needed to plan ahead and automate to make sure the migration was efficient, and to ensure we could simplify data and processes along the way.

The first legacy data warehouse we migrated had been built over a period of 15 years, with 30 years worth of data comprising millions of transactions and 180 TB of data.

It ran 6,500 extract, transform, load (ETL) jobs and more than 2,500 reports, getting data from about 100 sources. Cloud migration choices usually involves either re-engineering or lift-and-shift, but we decided on a different strategy for ours: move and improve.

This allowed us to take full advantage of BigQuery’s capabilities, including its capacity and elasticity, to help solve our essential problem of capacity constraints. 

Taking the first steps to cloud
We started creating our cloud strategy through a mapping exercise, which also helped start the change management process among internal teams. We chose architecture decision records as our migration approach, basing those on technical user journeys, which we mapped out using an agile board. User journeys included things like “change data capture,” “product event handling,” or “slowly changing dimensions.” 

These are typical data warehouse topics that have to be addressed when going through a migration, and we had others more specific to the financial services industry, too.

For example, we needed to make sure the data warehouse would have a consistent, golden source of data at a specific point in time. We considered business impacts as well, so we prioritized initially moving archival and historical data to immediately take load off of the old system.

We also worked to establish metrics early on and introduce new concepts, like managing queries and quotas rather than managing hardware, so that data warehouse users would be prepared for the shift to cloud.

To simplify as we went, we examined what we currently had stored in our data warehouse to see what was used or unused. We worked with stakeholders to assess reports, and identified about 600-plus reports that weren’t being used that we could deprecate. We also examined how we could simplify our ETL jobs to remove the technical debt added by previous migrations, giving our production support teams a bit more sleep at night. 

We used a three-step migration strategy for our data: first, migrating schema to BigQuery; second, migrating the reporting load to BigQuery, adding metadata tagging and performing the reconciliation process; and third, moving historical data by converting all the SQL script into data into BigQuery-compliant scripts. 

Creating new tools for migration automation
In keeping with our automation mantra, we invented multiple accelerators to speed up migration. We developed these to meet the timelines we’d set, and to eliminate human error. 

The schema parser and data reconciliation tool helped us migrate our data layer onto BigQuery. SQL parser helped migrate the data access layer onto Google Cloud Platform (GCP) without having to individually migrate 3,500 SQL instances that don’t have data lineage or documentation. This helped us to prioritize workloads.

And the data lineage tool identified components across layers to find dependencies. This was essential for finding and eliminating integration issues during the planning stage, and for identifying application owners during the migration. Finally, the data reconciliation tool reconciles any discrepancies between the data source and the cloud data target. 

Building a cloud future
We used this first migration in our UK data center as a template, so we now have a tailored process and custom tools that we’re confident using going forward.

Our careful approach has paid off for our teams and our customers. We’re enjoying better development and testing procedures. We’ve created an onboarding path for applications, we have a single source of truth in our data warehouse, and we use authorized views for secure data access. The flexibility and scalable capacity of BigQuery means that users can explore data without constraints and our customers get the information they need, faster. 

Spanner uses certain concepts differently from other enterprise database management tools, so you might need to adjust your application to take full advantage of its capabilities. You might also need to supplement Spanner with other services from Google Cloud to meet your needs.

Migration constraints

When you migrate your application to Spanner, you must take into account the different features available. You probably need to redesign your application architecture to fit with Spanner’s feature set and to integrate with additional Google Cloud services.

Stored procedures and triggers

Spanner does not support running user code in the database level, so as part of the migration, you must move business logic implemented by database-level stored procedures and triggers into the application.

Sequences

Spanner does not implement a sequence generator, and as explained below, using monotonically increasing numbers as primary keys is an anti-pattern in Spanner. An alternative way to generate a unique primary key is to use a random UUID.

If sequences are required for external reasons, then you must implement them in the application layer.

Access controls

Spanner supports only database-level access controls using IAM access permissions and roles. Predefined roles can give read-write or read-only access to the database.

If you require finer grained permissions, you must implement them at the application layer. In a normal scenario, only the application should be allowed to read and write to the database.

If you need to expose your database to users for reporting, and want to use fine-grained security permissions (such as table- and view-level permissions), you should export your database to BigQuery.

Read the full article for more, including

• Data validation constraints
• Supported data types
• Migration process
• Transferring your data from Oracle to Spanner
• Maintaining consistency between both databases
• Verifying data consistency
• and more.