Explore the Innovations and Architecture Powering Spanner and BigQuery - Build What's Next

6076

Of your peers have already watched this video.

1:30 Minutes

The most insightful time you'll spend today!

Blog

Explore the Innovations and Architecture Powering Spanner and BigQuery

Previously, databases had architectures with tightly coupled storage and compute. This resulted in higher latency, and with faster networks these constraints no longer surface. With Google Cloud’s BigQuery and CloudSpanner, the storage and compute architecture have been separated, allowing for better scalability and availability to address businesses’ high throughput data needs.

Watch the video to understand how these database and analysis products leverage Google’s distributed storage system, in-house custom network hardware and software, internal cluster management system and more!

Blog

Transforming Software Development Education with CourseMatix

1478

Of your peers have already read this article.

3:00 Minutes

The most insightful time you'll spend today!

Embrace a future of innovative and engaging learning! Explore how CourseMatix is revolutionizing computer science education and uncover the transformative potential of this platform as it shapes the next generation of tech-savvy professionals.

As the world increasingly relies on software, businesses have struggled to find enough developer talent to build and maintain their applications. IDC estimates the global shortage of software developers reached 1.4 million in 2021 and expects that number to balloon to 4 million by 2025.

Higher education is a great starting point to solve the software development skills shortage. One institution — Spokane Community College — has made strides in enhancing student learning and serving more students with the right mix of software development and IT education while saving educators time. 

Building a modern IT education platform

As a first step to improve computer science education and scale the number of students it could teach, Spokane Community College needed a platform that could help instructors more efficiently manage the teaching process. That’s where CourseMatix came in.

CourseMatix is an education platform that delivers the required tools, platforms, and frameworks such as automated grading, assignment tracking, and testing to save teachers time and enhance students’ learning. 

CourseMatix is entirely made up of microservices, with significant compute demands because it creates and runs a unique dedicated environment for each student. The platform is complex by nature as it supports a range of technologies that computer science professors need to teach varying coursework. This led to issues with scalability, reliability, and uptime when running on legacy infrastructure.

“We really wanted to no longer manage our infrastructure, leaving us to focus entirely on our platform,” says Aleks Korn, chief executive officer of CourseMatix. “We engaged Google Cloud partner CDW to see how we could simplify and improve our IT environment. The decision was quickly made to migrate to Google Kubernetes Engine (GKE).”

GKE provides CourseMatix with automatic horizontal and vertical scaling to minimize costs and downtime, while removing management and maintenance requirements. This enables CourseMatix to expand the reach of its platform at the lowest possible price and maintain a lean team. CDW helped CourseMatix make this decision to migrate to GKE, managed the migration, and provided guidance on a storage solution. 

Because CourseMatix has high data storage volume requirements (given the size of software development assignments) and needs to be fully reliable to ensure work is not lost, it also chose to work with NetApp. NetApp Cloud Volumes Service for Google Cloud perfectly complements GKE in its highly expandable, reliable, and user-friendly capabilities. The platform was now ready for its U.S. launch at Spokane Community College. 

Transforming software development education 

Spokane Community College needed a solution to address the challenges facing its professors, particularly with respect to time management.

“Each software development assignment can take an hour to grade,” says Bret Dickey, professor of software development at Spokane Community College. “When you have 100 students, you’re talking weeks of turnaround time to grade those projects. CourseMatix provides immediate feedback, which is a game changer for us as professors and our students.”

Students now complete and submit their assignments with real-world tools, using source control, creating branches to complete their work, and merging those branches to push up to the server where pipelines run the applications.

“It’s making our teaching better overall, including using real-world concepts to enhance teaching and outcomes,” says Dickey.

CourseMatix also provides the tooling to upskill on various programming languages and software packages required for coursework, helping Spokane Community College manage server-side frameworks and client-side languages that each have their own tools. 

“Students can access their cloud-based coding environment and all the tools they may need for an assignment are ready to go,” says Karmen Blake, another software development professor at Spokane Community College. “Having a central system for tooling within CourseMatix has been a huge time saver, meaning more time for learning and practice that prepares students for careers as professional software developers.”

Karmen and Dickey have seen marked improvements in their ability to teach and in student learning by using CourseMatix. For example, the platform provides automated hints and varying error messages based on the number of attempts a student has made to solve a problem to help each student along without making the assignment too easy or too difficult. 

As a cloud-based platform, CourseMatix also empowers students from a wide range of socioeconomic backgrounds to learn computer science. 

“As long as students can access the internet on their laptops, they have everything to complete their assignments,” says Karmen. “The cloud-based CourseMatix offering improves efficiency and equity across all students in our computer science programs.”

With a more fluid experience for students and immense amounts of time saved on grading, Spokane Community College is in a better position to increase enrollment in software development classes.

“Now that we have automated our grading and optimized our assignment processes, I am beginning to take on more students and teach more classes,” says Dickey. “I could not have imagined this outcome only a couple of years ago, and it’s all possible thanks to CourseMatix.”

To learn more about how partners help organizations make the most of Google Cloud, you can visit our partner page.

4230

Of your peers have already watched this video.

54:00 Minutes

The most insightful time you'll spend today!

Case Study

Behind the Scenes: How eBay Provides its Customers New Shopping Experiences

“If it exists in the world, you are likely to find it on eBay.” So they say. With 180 million buyers and a global presence in over 190 markets, it’s probably true. A company that emerged out of the ashes of the dot-com bubble, eBay today is one of the most renowned organizations in the world. 

eBay catalog has over 1 billion listings available worldwide at any given point in time and over 15 million listings are added on a daily basis. That essentially means many datasets from different sources. And that was a challenge.

The company needed a scalable and flexible platform to efficiently enable new shopping experiences globally, showcasing unique eBay inventory. It needed to iterate quickly and integrate data-heavy AI technologies.

eBay turned to Cloud Bigtable from Google. Today, Cloud Bigtable handles eBay’s global catalog with billions of listings, scaling to hundreds of terabytes with many millions of reads and writes, and gigabytes of data transferred in and out every second.

This helped eBay synchronize its entire catalog over GCP in near real-time with low latency and high consistency. It also enabled the company to serve catalog data to other services on GCP with high read volume and low latency and keep and serve all listing images in GCP along with AI vision signatures.

Find out how.

Blog

Gartner Positions Google as the Leader for Cloud Database Management Systems!

3605

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

Second time in the row, Google earns Gartner® Magic Quadrant™ for Cloud Database Management Systems (DBMS) for its unified capabilities across transactional and analytics use cases and innovations in data management, security and more!

We are thrilled that Gartner has positioned Google as a Leader for the second year in a row in the 2021 Gartner® Magic Quadrant™ for Cloud Database Management Systems (DBMS).

gartner DBMS MQ 2021.jpg

We believe the report evaluated Google’s unified capabilities across both transactional and analytical use cases, and showcases innovation progress in areas like data management consistency, high speed processing and ingestion, security, elasticity, advanced analytics, and more. 

With the recent announcement of Dataplex, organizations can centrally manage, monitor, and govern their data across data lakes, data warehouses, and data marts with consistent controls.  Solutions like BigQuery ML provide a “built-in” approach for advanced analytics capabilities and Analytics Hub offer the infrastructure customers need to share data analytics solutions securely and at scale in ways never before achieved. For example, over a seven-day period in April, more than 3,000 different organizations shared over 200 petabytes of data using BigQuery. 

Research shows 90% of organizations have a multicloud strategy, which is why we have invested in a cross-cloud data analytics solution for Google Cloud, AWS, and Azure with BigQuery Omni. Additionally, our progress with Anthos and our Distributed Cloud this past year further advance our ability to support multi and hybrid cloud scenarios. To gain a competitive advantage using data, organizations need a data platform that transcends transactional and analytical workloads and can be run with the highest level of reliability, availability and security. Cloud Spanner, our globally distributed relational database has redefined the scale, global consistency, and availability of Online Transaction Processing (OLTP) systems. Spanner processes over 1 billion requests per second at peak, and has been battle-tested with some of the most-demanding applications, including Google services such as Search, YouTube, Gmail, Maps, and Payments. And what’s unique about our core services Spanner and BigQuery, is that they leverage common infrastructure such as our highly durable distributed file system (Colossus), our large-scale cluster management system (Borg), and Jupiter, our high-performance networking infrastructure, enabling features such as federation between Spanner and BigQuery.

We remain focused on the integration of Google Trends, Maps, Search, Ads, and have increased industry domain expertise in areas such as retail, financial services, healthcare, and gaming. We’re continuing to develop industry white papers such as this one – How to develop Global Multiplayer Games using Cloud Spanner – and we’re proud of the work the team has done to create and share industry and horizontal architecture patterns built from industry leaders to serve as solution accelerators for customer use cases. 

Innovation momentum continues with a unified and open data cloud

We continue to innovate across our data cloud portfolio, especially with the innovations we announced at Google Cloud NEXT’21. BigQuery Omni is now available for AWS and Azure, supporting cross-cloud analytics for customers. We’ve added additional capabilities for enterprise data management and governance with Dataplex, which recently went GA. We’ve made migrations to Cloud SQL easier and faster with the Database Migration Service. More than 85% of all migrations are underway in under an hour, with the majority of customers migrating their databases from other clouds. We are embracing openness with Spanner by adding a PostgreSQL interface, allowing enterprises to take advantage of Spanner’s unmatched global scale, 99.999% availability, and strong consistency using skills and tools from the popular PostgreSQL ecosystem. We are also automating data processing with Spark on Google Cloud, which enables developers to spend less time on infrastructure management and more time on data science, modeling, and delivering business value. Finally, we announced Google Earth Engine on Google Cloud, allowing customers to integrate Earth Engine with BigQuery, Google Cloud’s ML technologies, and the Google Maps Platform. 

With these innovations, enterprises like PayPal, Deutsche Bank, and Equifax use Google Cloud to solve for their end-to-end data lifecycle use cases. Organizations like Telefonica use Google Cloud to deliver new customer experiences. Telefónica has transformed every aspect of how they store, share, and analyze data while doubling processing power and lowering costs.

We continue to support an open ecosystem of data partners including Informatica, Tableau, MongoDB, Neo4j, C3.ai, and Databricks, giving customers the flexibility of choice to build their data clouds without being locked into a single approach. 

We are honored to be a Leader in the 2021 Gartner Magic Quadrant for Cloud Database Management Systems (DBMS), and look forward to continuing to innovate and partner with you on your digital transformation journey. 

Download the complimentary 2021 Gartner Magic Quadrant for Cloud Database Management Systems report.

Learn more about how organizations are building their data clouds with Google Cloud solutions. 


Gartner, Magic Quadrant for Cloud Database Management Systems, Henry CookMerv AdrianRick GreenwaldAdam RonthalPhilip Russom, 14 December 2021.

Gartner and Magic Quadrant are registered trademarks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.

Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s Research & Advisory organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Google Cloud.

Blog

Explore Google Cloud SQL’s 3 Fault Tolerance Mechanism to Ease Data Pro

5217

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

Grow your organization's fault tolerance with Cloud SQL's three fault tolerance mechanism, and ensure data protection as well as recovery human errors, primary instance failures and zonal outages, and regional outages.

If you’re managing a crucial application that has to be fully fault-tolerant, you need your system to be able to handle every fault, no matter the type and scope of failure, with minimal downtime and data loss. Protecting against these faults means juggling numerous variables that can impact performance as well as recovery time and cost.

Today’s managed database services take over the operational complexity that used to exist for database administrators. Growing your organization’s tolerance required adding machines, compute, and storage, plus the operational costs of IT management: performing backups, writing scripts, creating dashboards, and carrying out testing to make sure your platform is ready when problems arise–all in a secure way.

At Google, our Cloud SQL managed database service offers three fault tolerance mechanisms —backup, high availability, and replication—and there are three major factors to consider for each of them:

  • RTO (recovery time objective): When a failure happens, how much time can be lost before significant harm occurs?
  • RPO (recovery point objective): When a failure happens, how much data can be lost before significant harm occurs?
  • Cost: How cost-effective is this solution?  

We’ve heard from customers like Major League BaseballHSBC, and Equifax that they have strict data-protection needs and require highly fault-tolerant multi-region applications—and they’ve all chosen Cloud SQL to meet those needs. 

Let’s take a closer look at how the decision-making process plays out for each recovery solution.  

High availability (HA)

If your application is business critical, you require minimum RTO and zero RPO— a high availability configuration ensures that you and your customers are protected. If the primary instance fails, there’s another standby instance ready to take over with no data loss. There’s an additional cost here, but doing this manually brings a great operational cost, since you have to detect and verify the fault, do the failover, and make sure it’s correct—you can’t have two primary instances or you risk data corruption—then finally connect the application to the new database.

Cloud SQL removes all that complexity. Choose high availability for a given instance and we’ll replicate the data across multiple zones, synchronously, to each zone’s persistent disk. If an HA instance has a failure, you don’t have to think about when to fail over because Cloud SQL detects the failure and automatically initiates failover, for a full recovery and no data loss within minutes. Cloud SQL also moves the IP address during failover so your application can easily reconnect. MLB, for example, uses Cloud SQL high availability to serve prediction data to live games with minimal downtime. Dev/test instances don’t need those same guarantees, but can use local backups to recover from any potential failure.

Cross-region replica

If a whole Google Cloud region goes down you still need your business to continue to run. That’s where cross-region replication comes in, a hot standby replica in another Google Cloud region provides RTO of minutes and RPO typically less than a minute . If you create a read replica in a region separate from your primary instance and you get hit with a regional outage, your application and database can start serving customers from another region within minutes. But this solution can be complex and enabling it yourself can be difficult and time-consuming. Securing cross-geography traffic demands end-to-end encryption and can bring connectivity issues too. 

This is where the fully managed Cloud SQL solution shines. We offer MySQL, PostgreSQL and SQL Server database engines as a cross-region replication solution that’s easily configured and bolstered by Google’s interconnected global network. Just say, “I’m in U.S. East, I want to create a replica in U.S. West,” and it’s done, reliably and securely.

create read replica.jpg

Backup

When you suffer data loss because of an operations error (for example, a bug in a script dropped your tables) or human error (for example, someone dropped the wrong table by accident), backups help you restore lost data to your Cloud SQL instance. Our low cost backup mechanism features point-in-time, granular recovery, meaning that if you accidentally delete data or something else goes wrong, you can ask for recovery of, for example, the state of that database down to the millisecond, such as Monday at 12:53pm. Your valuable data is replicated multiple times in multiple geographic locations automatically. This enables the automatic handling of failover in cases of major failure. You can always rest assured that your database is available and data is secure, even in the times of major failure crises. 

Cloud SQL provides automated and on-demand backups. With automated backups, Google manages the backups so that you can easily restore them when required. Also, the scheduled backing is automatically taken by default. With on-demand backup, you can create a backup at any time. This could be useful if you are about to perform a risky operation on your database, as Cloud SQL lets you select a custom location for your backup data. When the backup is stored in multiple regions, and there’s an outage in the region that contains the source instance, you can restore a backup to a new or existing instance in a different region. This is also useful if your organization needs to comply with data residency regulations that require you to keep your backups within a specific geographic boundary.

cloud sql backups.jpg

Putting it all together

For critical workloads, MLB configures their Cloud SQL instances with backups, high availability, and cross-region replication. Doing so ensures they can recover from many failure types.

  • To recover from human error (“Oops, I didn’t mean to delete that”), MLB uses backups and point-in-time recovery to recovery to a millisecond or specific database transaction
  • To automatically recover from primary instance failures and zonal outages, MLB uses Cloud SQL’s high availability configuration
  • To protect against regional outages, MLB uses cross-region replication

Creating a robust configuration, like MLB did, takes just a few minutes. Get started in our Console or review documentation.

Case Study

What Swiggy and You Can Learn From This Company’s Use of ML to Engage Customers

9650

Of your peers have already read this article.

1:45 Minutes

The most insightful time you'll spend today!

Just Eat, which is similar to Swiggy, uses Google Cloud's machine learning to power sophisticated consumer recommendations on both its app and website. It enables them to create an “Adventurous Index”, for instance, something we haven't seen in Indian ordering apps.

The app economy has enabled a huge range of unique business models to flourish. One such model is online food ordering and delivery services, in which apps leverage geo-location data to aggregate local food choices and offer personalized options to consumers.

A leading company in this space is Just Eat. Launched in the UK in 2001 with a vision of ‘serving the world’s greatest menu. Brilliantly.’ The company has capitalized on the popularity of online food delivery and grown its presence across 12 markets. 

Just Eat acts as an intermediary between take-out food outlets and hungry customers, giving local restaurants access to a broader base of potential diners, while providing consumers with an easy and secure way to order and pay for food from their favourite restaurants.

Today the company helps 27 million customers find food from more than 112,000 restaurants—everything from homemade Italian pasta, to Chinese noodle bowls, to fish-and-chips. 

Data is the fuel of Just Eat’s rapid growth, but it wasn’t always looked at that way. In its early days, Just Eat struggled with the deluge of information and faced fragmentation across its systems. In fact, the company realized its legacy data vendor wasn’t capable of ingesting 90 percent of the data produced by its food platform. This was incredibly frustrating for Just Eat’s analysts and data scientists, who had to waste time cleaning up sources instead of leveraging the data to create a better user experience. 

Just Eat turned to Google Cloud, and now uses machine learning (ML) to power sophisticated consumer recommendations on both its app and website. It also makes heavy use of features offered by Google Cloud Platform, including BigQuery for running analytics on its customer data set and Cloud Pub/Sub for messaging app users with relevant offers in real-time. 

Having all of Just Eat’s data in one platform has translated into real value for its customers. With Google Cloud tools, Just Eat has created its own proprietary Customer Ontology framework, which today contains 5.5 billion features that better understand consumers’ behavior and food habits, and provides insights into previous visits.

Just Eat recently created an “Adventurous Index” to map its customers according to their ordering habits, enabling them to tailor their marketing and user experiences. For example, mid-adventurous customers are shown a choice of restaurants that serve their most ordered cuisine, while adventurous customers can choose from restaurants that serve a wider variety. This not only has prompted consumers to be more adventurous with their choices, but also has led to more business at a more diverse set of restaurants.

Matt Cresswell, Director of Customer Platforms at Just Eat said that Google Cloud has become integral to its product delivery: “Consumer food choice is a hugely nuanced topic. We know that individuals have their own unique journeys when they use Just Eat. We’ve sought to create a truly one-to-one relationship with every customer. The changes we’ve made to the platform mean they can access the dishes they enjoy at the touch of a fingertip, and find inspiration to discover new dishes they’ll love. We’re grateful to Google Cloud for helping us support our customers on their culinary explorations.”

More Relevant Stories for Your Company

Blog

Principles to Make Organizations Data Engineering Driven

In the “What type of data processing organisation” paper, we examined that you can build a data culture whether your organization consists mostly of data analysts, or data engineers, or data scientists. However, the path and technologies to become a data-driven innovator are different and success comes from implementing the right

How-to

Migrate Your Microsoft SQL Server Workloads to Google Cloud

Enterprise database workloads are the backbone of many of your applications and ecosystems. Also, guaranteed availability is critical when choosing a cloud provider. Many enterprises built their mission-critical applications on Microsoft SQL Server 2008, and it’s common still to run into older versions of SQL Server as you’re working toward

Case Study

Insurer Uses Google Cloud AI to Battle Slow Growth: It Improves Sales by 5% in 8 Weeks

For a business to succeed in the long term, it needs to learn not just to adapt to inevitable change, but to harness it. South Africa-based PPS has been an insurance company since 1941 and today is the biggest mutual insurance provider in the country. As a mutual company, PPS is owned

How-to

Architecting Multi-region Database Disaster Recovery for MySQL

Enterprises expect extreme reliability of the database infrastructure that’s accessed by their applications. Despite your best intentions and careful engineering, database errors happen, whether that’s machine crashes or network partitioning. Good planning can help you stay ahead of problems and recover more quickly when issues do occur. This blog shows one approach

SHOW MORE STORIES