Simplifying Unstructured Data Analytics with BigQuery ML and Vertex AI: A Comprehensive Guide
1532
Of your peers have already read this article.
6:30 Minutes
The most insightful time you'll spend today!
Unstructured data such as images, speech and textual data can be notoriously difficult to manage, and even harder to analyze. The analysis of unstructured data includes use cases such as extracting text from images using OCR, sentiment analysis on customer reviews and simplifying translation for analytics. All of this data needs to be stored, managed and made available for machine learning.
The new BigQuery ML inference engine empowers practitioners to run inferences on unstructured data using pre-trained AI models. The results of these inferences can be analyzed to extract insights and improve decision making. This can all be done in BigQuery, using just a few lines of SQL.
In this blog, we’ll explore how the new BigQuery ML inference engine can be used to run inferences against unstructured data in BigQuery. We’ll demonstrate how to detect and translate text from movie poster images, and run sentiment analysis against movie reviews.
BigQuery ML’s new inference engine
Google Cloud is home to a suite of pre-trained AI models and APIs. The BigQuery ML inference engine can call these APIs and manage the responses on your behalf. All you have to do is define the model you want to use and run inferences against your data. All of this is done in BigQuery using SQL. The inference results are returned in JSON format and stored in BigQuery for analysis.
Why run your inferences in BigQuery?
Traditionally, working with AI models to run inferences required expertise in programming languages like Python. The ability to run inferences in BigQuery using just SQL can make generating insights from your data using AI simple and accessible. BigQuery is also serverless, so you can focus on analyzing your data without worrying about scalability and infrastructure.
The inference results are stored in BigQuery, which allows you to analyze your unstructured data immediately, without the need to move or copy your data. A key advantage here is that this analysis can also be joined with structured data stored in BigQuery, giving you the opportunity to deepen your insights. This can simplify data management and minimize the amount of data movement and duplication required.
Which models are supported?
For now, the BigQuery ML inference engine can be used with these pre-trained Vertex AI models:
- Vision AI API: This model can be used to extract features from images managed by BigQuery Object Tables and stored on Cloud Storage. For example, Vision AI can detect and classify objects, or read handwritten text.
- Translation AI API: This model can be used to translate text in BigQuery tables into over one hundred languages.
- Natural Language Processing API: This model can be used to derive meaning from textual data stored in BigQuery tables. For example, features like sentiment analysis can be used to determine whether the emotional tone of text is positive or negative.
So, how does this work in practice? Let’s look at an example using images of movie posters
- We will define our pre-trained models for Vision AI, Translation AI and NLP AI in BigQuery ML.
- We’ll then use Vision AI to detect the text from some classic movie posters images.
- Next, we’ll use Translation AI to detect any foreign posters and translate them to a language of our choosing – English in this case.
- Finally, we’ll combine our unstructured data with structured data in BigQuery.
We’ll use the extracted movie titles from our movie posters to look up the viewer reviews from the BigQuery IMDB public dataset. We can then run sentiment analysis against these reviews using NLP AI.
Note: The BigQuery ML inference engine is currently in Preview. You will need to complete this enrollment form to have your project allowlisted for use with the BQML Inference Engine.
We’ll give examples of the BigQuery SQL needed to define your models and run your inferences. You’ll want to check out our notebook for a detailed guide on how to get this up and running in your Google Cloud project.
1. Define your AI Models in BigQuery
You will need to enable the APIs listed below, and also create a Cloud resource connection to enable BigQuery to interact with these services.
| API | Model Name |
| Vision AI API | Cloud_ai_vision_v1 |
| Translation AI API | Cloud_ai_translate_v3 |
| NLP AI API | Cloud_ai_natural_language_v1 |
You can then run the CREATE MODEL query for each AI service to create your pretrained models, replacing the model_name as required.
CREATE OR REPLACE MODEL
`{PROJECT_ID}.{DATASET_ID}.{VISION_MODEL_NAME}`
REMOTE WITH
CONNECTION `{PROJECT_ID}.{REGION}.{CONN_NAME}`
OPTIONS ( remote_service_type = '<model_name>' );2. Use the Vision AI API to detect text in images stored in Cloud Storage
You will need to create an object table for your images in Cloud Storage. This read-only object table provides metadata for images stored in Cloud Storage:
CREATE OR REPLACE EXTERNAL TABLE
`{PROJECT_ID}.{DATASET_ID}.{OBJECT_TABLE_NAME}`
WITH
CONNECTION `{REGION}.{CONN_NAME}`
OPTIONS (object_metadata = 'SIMPLE', uris = ['{BUCKET_LOCATION}/*']);To detect the text from our posters, you can then use ML.ANNOTATE_IMAGE and specify the text_detection feature.
SELECT
ml_annotate_image_result.full_text_annotation.text AS text_content,
*
FROM
ML.ANNOTATE_IMAGE(
MODEL `{PROJECT_ID}.{DATASET_ID}.{VISION_MODEL_NAME}`,
TABLE `{DATASET_ID}.{OBJECT_TABLE_NAME}`,
STRUCT(['TEXT_DETECTION'] AS vision_features));A JSON response will be returned to BigQuery that includes the text content and language code of the text. You can parse the JSON to a scalar result using the dot annotation highlighted above.
3. Use the Translation AI API to translate foreign movie titles
ML.TRANSLATE can now be used to translate the foreign titles we’ve extracted from our images into English. You just need to specify the target language and the table of the movie posters for translation:
SELECT
text_content,
STRING(ml_translate_result.translations[0].detected_language_code)
as original_language,
STRING(ml_translate_result.translations[0].translated_text)
as translated_title
FROM
ML.TRANSLATE(
MODEL `{PROJECT_ID}.{DATASET_ID}.{TRANSLATE_MODEL_NAME}`,
TABLE `{DATASET_ID}.image_results`,
STRUCT('TRANSLATE_TEXT' as translate_mode, "en" as target_language_code));Note: The table column with the text you want to translate must be named text_content:
The table of results will include json that can be parsed to extract both the original language and the translated text. In this case, the model has detected that title text is in French and has translated it to English:
4. Finally, use natural language processing (NLP) to run sentiment analysis against movie reviews
You can easily join inference results from your unstructured data with other BigQuery datasets to bolster your analysis. For example, we can now join the movie titles we extracted from our posters with thousands of movie reviews stored in BigQuery’s IMDB public dataset `bigquery-public-data.imdb.reviews`.
You can use ML.UNDERSTAND_TEXT with the analyze_sentiment feature to run sentiment analysis against some of these reviews to determine whether they are positive or negative:
SELECT
primary_title, start_year, text_content AS review,
FLOAT64(ml_understand_text_result.document_sentiment.score) AS score,
FLOAT64(ml_understand_text_result.document_sentiment.magnitude) AS magnitude,
FROM
ML.UNDERSTAND_TEXT(
MODEL `{PROJECT_ID}.{DATASET_ID}.{NLP_MODEL_NAME}`,
(
SELECT
primary_title, start_year, review AS text_content
FROM
`bigquery-public-data.imdb.title_basics` titles
JOIN
`bigquery-public-data.imdb.reviews` reviews
ON
reviews.movie_id = titles.tconst
WHERE
UPPER(titles.primary_title) = 'THE LOST WORLD' AND
start_year = 1925
),
STRUCT("analyze_sentiment" AS nlu_option)) ;Note: The table column with the text you want to analyze must be named text_content:
The JSON response will include a score and magnitude. The score indicates the overall emotion of the text while the magnitude indicates how much emotional content is present:
So, how did the Lost World compare with other movies that year?
To wrap up, we’ll compare the average review score of the 1925 Lost World movie to other movies released that year to see which was more popular. This can be done using familiar SQL analysis:
SELECT
primary_title, start_year,
AVG(FLOAT64(ml_understand_text_result.document_sentiment.score))AS av_score,
AVG(FLOAT64(ml_understand_text_result.document_sentiment.magnitude)) AS av_magnitude
FROM
ML.UNDERSTAND_TEXT(
MODEL `{PROJECT_ID}.{DATASET_ID}.{NLP_MODEL_NAME}`,
(
SELECT
primary_title, start_year, movie_id, review AS text_content
FROM
`bigquery-public-data.imdb.title_basics` titles
JOIN
`bigquery-public-data.imdb.reviews` reviews
ON
reviews.movie_id = titles.tconst
WHERE
start_year = 1925
),
STRUCT("analyze_sentiment" AS nlu_option))
GROUP BY
primary_title, start_year
ORDER BY
av_score DESC;
It looks like The Lost World narrowly missed out on the top spot to Sally of the Sawdust!
Want to learn more?
Check out our notebook for a step by step guide on using the BQML inference engine for unstructured data in Google Cloud. You can also check out our Cloud AI service table-valued functions overview page for more details. Curious about pricing? The BQML Pricing page gives a breakdown of how costs are applied across these services.
Cloud Bigtable brings database stability and performance to Precognitive
5706
Of your peers have already read this article.
3:40 Minutes
The most insightful time you'll spend today!
At Precognitive, we were able to start with a blank technology slate to support our fraud detection software products. When we started building the initial version of our platform in 2017, we had some decisions to make: What coding language to use? What cloud infrastructure provider to choose? What database to use? The majority of the decisions were straightforward, but we struggled to decide upon a database. We had plenty of collective experience with relational databases, but not with a wide-column database like Cloud Bigtable—which we knew we’d need to scale our behavior and device workloads. At launch, our products were supported by a self-managed database, but we quickly migrated to Cloud Bigtable, and we love it.
To efficiently support our bursty, real-time fraud detection workloads, we needed a cloud database that could satisfy the following key requirements:
- Stability to keep up with increased adoption of our products
- Intelligent scaling that avoids bottlenecks
- Native integrations with BigQuery and Cloud Dataproc
- Managed services that free up our engineers’ time to work on our products
Adding Cloud Bigtable as our performance database
As we scaled our services and added customers, our data collection services for our Device Intelligence and Behavioral Analytics products were seeing thousands of events per second. Cloud Bigtable provided a stable managed database that could handle the volume we were receiving during peak hours. We weren’t always able to handle this scale, as an early version of our product utilized a self-managed database.
Every month, two or three engineers spent hours managing the database instances. Whenever the instances crashed, it would cost at least one engineer a day or two of productivity attempting to restore the instances and recovering any data from our backup database. Managing this database internally was taking precious time away from product development.
We circled back to Cloud Bigtable. After two weeks of R&D, we decided to switch the Device Intelligence and Behavioral Analytics services to Cloud Bigtable.
Cloud Bigtable solved our scaling issues. Cloud Bigtable had been attractive to us from the start because it was fully managed, and offered regional replication and other features we were lacking in our own managed instances. Cloud Bigtable provides horizontal scaling and automatically rebalances row keys (equivalent to a shard key) over time to prevent “hot” nodes. In addition, Cloud Bigtable provides a connector to BigQuery and Cloud Dataproc that allows us to analyze the terabytes of data we are processing and use that data for unsupervised machine learning.
The perks of using Cloud Bigtable
After the migration to Cloud Bigtable, we noticed a number of additional benefits: improved I/O performance, a significant cost reduction, and a sizable decrease in hours spent on database maintenance.
We measured some of our typical metrics before and after implementing Cloud Bigtable. Our request latency dropped by about 30 ms on average (to sub-10 ms) for API requests. Prior to the change, we were seeing latencies of 40+ ms on average. This latency drop on our Behavioral Analytics and Device Intelligence products allowed us to trim about an additional 10 to 15 ms off our average response time across all dependent services.
Before we moved to Cloud Bigtable, we had to scale our database instances every time a new customer was onboarded. We were over-scaling in an attempt to avoid constantly resizing our database servers. By sunsetting our self-managed database and switching to Cloud Bigtable, we cut database infrastructure costs by approximately 35% and can now scale as needed, with a couple of clicks, during onboarding.
We have spent zero hours managing a Cloud Bigtable database since launch, and we put the time we are saving every month toward product development.
Moving forward with Cloud Bigtable
As an engineering team, we love working with Cloud Bigtable. We are not only seeing improved developer experience and reduced latency, which keeps the engineers happy, but also reduced costs, which keeps the business happy. We’re able to build more product, too, with the time we’ve saved by switching to Cloud Bigtable. Stay tuned to our engineering blog for more on the lessons we’ve learned and our contributions to the wider Cloud Bigtable community.
The Fantastic Story of How BMG Enables a Micropayments Strategy So Music Artists Get Paid
7419
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
The music industry is rapidly changing. Only 20 years ago, the availability of music and the infrastructure that was required to make an album a sales success were incredibly complex and expensive. With the decline of physical sales and a fundamental shift to digital, music streaming now accounts for more than half of all sales globally.
At the same time, technology has democratized music-making; in many ways, it has made the musical landscape more diverse. Artists can upload their music with the click of a button. But while it’s easier for creators to share their songs with audiences, getting paid has become more fragmented.
Although music is booming, people no longer buy it outright. Instead, listeners download music digitally or subscribe to streaming services to have their libraries with them at all times. To monetize digital content effectively, artists need to know when, where, and how often their songs are played on each service. To help them navigate this complicated royalties landscape and maximize their profits, Berlin-based international music company BMG provides customized, transparent, and fair services to songwriters and artists.
With publishing and recording divisions under one roof, the subsidiary of international media giant Bertelsmann works with both emerging artists and established stars, including John Legend, Kylie Minogue, Mick Jagger, and Keith Richards. With the MyBMG web and mobile application, clients can view and analyze their royalty details in real time and collect payment. When a new record is released, BMG uses data to maximize its impact and revenue for its creators.
“We make sure that everyone who uses our clients’ music knows who needs to be paid the associated royalties, then we collect these royalties and share them out quickly and transparently,” explains Sebastian Hentzschel, Chief Information Officer at BMG. “When our artists release new music, we make sure that it’s marketed and promoted effectively around the world.”
“We needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship. With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”
—Gaurav Mittal, Vice President Group Technology, BMG
Getting up to speed with a new way of paying artists
In this digital world, artists aren’t just paid every time a fan buys an album—they’re paid a small amount, or royalty, for each song downloaded or streamed by a listener. So, when the industry shifted to digital, the volume of data that BMG needed to handle grew exponentially. “One CD sale is equivalent to about 1,500 streamed songs or plays,” says Gaurav Mittal, Vice President Group Technology at BMG. “That means IT departments have to process 1,500 times the amount of data to calculate payments for artists, and this makes scalable micropayment processing very important.”
Until 2019, BMG’s infrastructure was entirely hosted on-premises. Hardware limitations made it challenging to scale on-demand, making it harder to handle the data peaks that royalty processing can bring. “With our on-premises infrastructure, we were going to hit a ceiling in a few years,” says Gaurav. “We still managed to process royalty payments for our clients, but it was increasingly time consuming and expensive. To keep focusing on our clients, rather than our infrastructure, we decided to migrate to Google Cloud.”
From the outset, Gaurav and his team had a clear vision for the partnership: “Most importantly, we needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship,” he says. “With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”
Keeping artists happy with business-as-usual payouts during migration
To move applications to the cloud while keeping payment cycles on track for its artists, BMG teamed up with Google Cloud partner Rackspace Technology. “We selected Rackspace Technology because it combines strong technical muscle and a global footprint, with the customer service of a local boutique firm,“ shares Gaurav.
BMG’s own technology team put together the outline for the Google Cloud architecture, which they passed on to Rackspace Technology for optimizations and the ultimate stamp of approval. Whenever Gaurav and his developers needed support, Rackspace Technology was ready to go. “So far, we’ve migrated 17 applications successfully, and Rackspace Technology has been 100% spot-on with each suggestion,” says Gaurav. “I can’t recall a single flaw in a Rackspace Technology-approved architecture, and that really says something.”
When BMG began its migration in August 2019, the team developed an ambitious two-year plan. Only 14 months later, however, the project is more than 75% complete. Among the solutions that BMG is using today are Cloud Storage to securely store 130 TB of data, and Cloud SQL as its standard database technology. The web applications run on Compute Engine, App Engine, and Google Kubernetes Engine.
“After successfully moving a few applications, it was clear that with the strong teamwork of BMG and Rackspace Technology, together with the ease of use of Google Cloud, we could speed up the project without sacrificing quality,” says Gaurav. “We’re set to complete our migration six months before schedule, helping us to quickly move out of our hybrid environment.”
“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT. Our teams are much more productive.”
—Gaurav Mittal, Vice President Group Technology at BMG
Royalty reporting and processing with BigQuery and Dataproc
So far, all of BMG’s critical workloads are up and running on Google Cloud. Royalty calculations, for example, which require incredible processing power and the collection of many micropayments to ensure full and timely payout, run entirely on Dataproc with output stored on BigQuery for downstream integration and reporting.
Enabling more harmonious workflows through self-serve analytics
As the new beating heart of BMG’s royalty reporting, BigQuery changed the rhythm of collaboration company-wide. In the past, income tracking teams had to contact IT departments if they needed deeper data insights for their work. By integrating Data Catalog with BigQuery, BMG has made the data more accessible to all teams. This helps them detect missing income and new revenue streams independently, maximizing profits for artists.
“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT,” says Gaurav. “Our teams are much more productive.”
“Google Cloud enables us to be more client focused and deliver better features faster. We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”
—Sebastian Hentzschel, Chief Information Officer, BMG
With a leaner IT environment, BMG can focus its effort on the needs of its clients. Beyond improvements in royalty processing, it can concentrate on app development, releasing new features and enhancements more frequently. By hosting applications on Google Kubernetes Engine, App Engine, and Compute Engine, BMG has built a CI/CD pipeline with automated deployments and testing to significantly speed up workflows.
“In our old system, it could take several weeks to set up an environment,” says Gaurav. “With Google Kubernetes Engine, any of our developers can complete the process in a few clicks. Having that autonomy makes our developers more motivated and self-driven.”
“Google Cloud enables us to be more client focused and deliver better features faster,” adds Sebastian. “We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”
With the migration almost complete, BMG is looking forward to its next technology project. It plans to leverage AutoML to further scale and automate royalty tracking with machine learning. On the marketing side, advanced analytics will help BMG determine the effectiveness of promotional campaigns around the world, further increasing profits for artists. By connecting Google Data Studio to BigQuery, BMG will increase the quality of its analyses, helping musicians better understand the reach of their music around the world.
In the end, Gaurav shares, helping musicians is what it all comes down to. “We’re a new kind of music company because we build our services around our artist, songwriter, and publisher clients, not the other way around,” he says. “Google Cloud is helping us maintain strong relationships with our clients, and that’s music to our ears.”
ActivStat: Enhancing Live Sports Broadcast with Real-time Stats and Metrics
5084
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Millions of people annually view esports, as top players and teams compete in thrilling, fast-paced tests of reflexes, strategy, and teamwork. Fans are a diverse group, sharing a passion for action. A new, revolutionary project we’re working on with Call of Duty League just made that action a lot better.
ActivStat brings fans, players, and commentators the power of competitive statistics in real-time—stats that matter not only to the game at hand, but also for a full roster of competitors globally. Using ActivStat, live broadcasts will soon be enhanced with more depth and color-of-play while they’re happening, building excitement and adding to the overall experience.
Technically, ActivStat is an entirely new capability for esports. It’s a constantly updating catalog of statistics that is sourced, analyzed, updated, and delivered in an easy-to-consume way across global-scale computing systems, with a latency of milliseconds or seconds, rather than minutes or hours. By comparison, many of these stats today are available to fans after a day or more of processing.
Call of Duty League plans to begin rolling out ActivStat during the 2021 season. The initial rollout will include critical information like player and team standings and winning ratios across multiple aspects of virtual combat—including ultimately what these numbers mean for rankings. ActivStats are delivered both in raw statistics and via visualizations and graphics, providing commentators with fast access to the types of insights fans crave.
For engineers at both companies, building the service has been an epic success all its own. Due to a sponsorship with Call of Duty signed last February, our dedicated game engineering team quickly innovated a new solution incorporating high-speed networking, data pipelines from multiple cloud sources, and data warehousing to create a user-friendly dashboard. Real data was flowing into commentator dashboards by April.
Esports are more complex to cover in many ways than regular sports. Instead of a well-defined physical playing field (often a simple rectangular space), multiplayer games involve complex and sprawling virtual environments that can be the size of a large campus or airport with multiple levels and hidden locations.
Gameplay between competitors happens across many of these locations simultaneously. In addition, competitors also each choose their own configurations of equipment, known as “loadouts,” which can dramatically affect gameplay and strategy. All of this additional complexity in online gaming involves data that needs to be captured, analyzed, and communicated to the fans in insightful ways.
Two Google Cloud technology capabilities play central roles in the operations of ActivStat. BigQuery—a planet-scale data warehouse that can store and query petabytes of information in real time—is the foundation of the ActivStat platform for gathering and summarizing millisecond-level statistics. Looker, an intuitive analytic dashboard, surfaces those insights to commentators in an easy-to-use, real-time dashboard that enables the commentators to speak to compelling insights and statistics in sync with the gameplay as it is happening in the live broadcast.
While the initial release of ActivStat for this season of Call of Duty League provides compelling and powerful capabilities, it’s only the beginning of this Call of Duty League-Google Cloud co-innovation partnership. Our future vision is mapping gameplay hotspots on the field, and predicting where to place cameras with machine learning as the match evolves–enabling broadcast producers to build excitement for fans by always being in the middle of the best action. Call of Duty League and Google will also look to drive statistics and metrics directly into the broadcast feed.
The implications of the real-time statistical capabilities of ActivStat go beyond gaming and esports. Historically, gaming has been at the leading edge of what computer processing, computer graphics, wide-area networking, data analysis, and insight can do. The real-time data ingestion and output used in ActivStat could one day be useful powering live broadcasts in other types of sporting events, as well as blending live video feeds and data to support use cases in media & entertainment, healthcare, finance, manufacturing, and other verticals. We’re excited about the potential for this bold new solution and partnership between Call of Duty League and Google in esports and beyond.
To learn more about Call of Duty League-Google Cloud partnership, visit our press release.
No More ‘Tab Game’ with Easy Tutorials on Google Cloud Console
3524
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
When it comes to learning how to implement some technology, we all have our own version of what I call the “tab game”—that is, your setup for all the tabs and windows you need open at once. You may have several monitors so you can see documentation, your IDE, and terminal windows at the same time. You may have several guides and references open at once in one window to get all the information you need.
Personally, I like to work just from my laptop because I like to move around and work from various comfy spots. I think my tab game would probably enrage most devs because it involves a lot of swiping back and forth between windows *and* toggling tabs. It’s not pretty. That is, it wasn’t pretty until I discovered tutorials in the Google Cloud Console!
Jen really didn’t know about tutorials in the Google Cloud Console?
Yes, I honestly didn’t know about them! I’m sharing about it because if I can work for Google and not know, then I can’t be the only one, and it would be a shame to miss out on this because it’s a brilliant idea. Also I wrote some pretty sweet tutorials for the console, but I swear that the main reason I’m telling you is because it’s a cool thing!
There are several reasons that these tutorials are great:
- You can view the instructions and the console at the same time. No more playing the tab game!
- The tutorials include links and highlights, making it easy to find the screens and buttons you’re looking for
- You can run code from Cloud Shell, so you don’t need a separate window for an IDE
- You can use the demo data provided to try things out, or you can apply the steps to your existing projects using data that suits your app’s needs
Firestore tutorials
I’m developing a series of tutorials in the Google Cloud Console designed to take you through everything you need to know about Firestore–from manually adding data in the Google Cloud Console to triggering Cloud Functions to make changes for you. Below are links and summaries for the currently available tutorials. Check back regularly to find the latest additions as they’re released!
- Enable Firestore on a project
- Learn about the Firestore data model
- Add a collection of documents
- Add fields to a document
- Delete documents and collections
Updating Data in Firestore using Node.js or using Python
- Add a collection of documents
- Explore available data types
- Replace the data of document
- Replace fields in a document
- Handle special cases: incrementing, timestamps, and arrays
Reading Data from Firestore using Node.js or using Python
- Add a collection of documents
- Explore available data types
- Read a collection
- Read a single document
- Order documents
- Query documents
Transactions in Firestore using Node.js
- Add a collection of documents
- Update data without a transaction to observe issue
- Complete a transaction
- Complete a batched write
Batched Writes in Firestore using Node.js or using Python
- Use Cloud Shell and Cloud Shell Editor to write a Node.js or Python app
- Complete a batched write
Firestore triggers for Cloud Functions
- Initialize Cloud Functions using the Firebase CLI
- Write a Cloud Function triggered by a new document write to Firestore
- Add data to Firestore in the Cloud console Firestore dashboard
- Create a web app that uses Firestore using the Firebase SDK
- Deploy Firestore security rules that enable access to the required data
- Enable data persistence in the web app
- Observe app behavior with and without network connection
Chime in
Is there a particular action or concept in Firestore that you’d like to see a tutorial for? Is there another Google Cloud product that you want to learn more about? Tweet @ThatJenPerson and you may just see your suggestion come to life in the Google Cloud Console!
Maximizing Storage Efficiency: A Guide to Reducing Point-in-Time Recovery Impact
1457
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Point-In-Time Recovery (PITR) is a critical capability for enterprise applications. It allows database administrators to recover from accidental data deletion by restoring their production databases to a time before the incident.
Cloud SQL for PostgreSQL launched support for PITR in July 2020, allowing you to recover from disasters like data corruption or accidental deletion by restoring your Cloud SQL instance to a previous time. We’re excited to announce an additional enhancement to PITR for Cloud SQL for PostgreSQL that makes enabling PITR an even easier decision: for instances with Point-in-Time Recovery newly-enabled, the write-ahead logs being stored for PITR operations (which are the transaction logs that are used to go back in time) will no longer consume disk storage space. Instead, when you enable PITR for new instances, Cloud SQL will store transaction logs collected during the retention window in Google Cloud Storage, and retrieve them when you perform a restore. Because transaction logs can grow rapidly when your database experiences a burst of activity, this move will help reduce the impact these bursts have on your provisioned disk storage. These logs will be stored for up to seven days in the same Google Cloud region as your instance at no additional cost to you.
PITR is enabled by default when you create a new Cloud SQL for PostgreSQL instance from the Google Cloud console, and transaction logs will no longer be stored on the instance for instances that have PITR newly enabled. If you have already enabled PITR on your PostgreSQL instances, this enhancement will be rolled out to your instances at a later point. If you want to take advantage of this change sooner, you can first disable and then re-enable PITR on your instance (which will reset your ability to perform a point-in-time restore to the time at which PITR was re-enabled). On instances with this feature enabled, you’ll notice that consumed storage on your instance will reduce relative to the volume of write-ahead logs (WAL) generated by your instance. The actual amount of storage your logs consume will vary by instance and by database activity – during busy times for your database, log size may shrink or grow. However, these logs will now only be stored on your instance long enough to successfully replicate to any replicas of the instance and to ensure that they are safely written to Cloud Storage; afterwards, they will be removed from your instance.
We’re excited to continue to enhance Cloud SQL for PostgreSQL to ensure that disaster recovery is easy to enable, cost effective, and seamless to use. Learn more about this change in our documentation.
More Relevant Stories for Your Company
Predictive Model Built on Google Cloud Helps You Get a 7-day Mosquito Forecast Report!
Mosquitoes aren’t just the peskiest creatures on Earth; they infect more than 700 million people a year with dangerous diseases like Zika, Malaria, Dengue Fever, and Yellow Fever. Prevention is the best protection, and stopping mosquito bites before they happen is a critical step. SC Johnson—a leading developer and manufacturer
Google and AI Researchers Work towards Building Data-centric AI
AI researchers and engineers need better data to enable better AI solutions. The quality of an AI solution is determined by both the learning algorithm (such as a deep-neural network model) and the datasets used to train and evaluate that algorithm. Historically, AI research has focused much more on algorithms
Unlocking the Potential of Advanced Analytics with BigQuery and Connected Vehicle Data
As software-defined vehicles continue to advance and the quantity of digital services grows to meet consumer demand, the data required to provide these services continue to grow as well. This makes automotive manufacturers and suppliers look for capabilities to log and analyze data, update applications, and extend commands to in-vehicle
Skincare Firm Scales 4X in Minutes with SAP on Google Cloud
As the number one skincare brand in the United States, Rodan + Fields must support its team of over 300,000 of independent contractors as well as work to ensure a really personalized experience for customers. To keep pace with the company’s growth, Rodan + Fields realized it needed a more
