How to Create, Manage and Run SQL Instances in Google Cloud SQL - Build What's Next

4488

Of your peers have already watched this video.

4:00 Minutes

The most insightful time you'll spend today!

How-to

How to Create, Manage and Run SQL Instances in Google Cloud SQL

Database admins and application developers can easily create a database on Google Cloud SQL, which helps manage mundane administrative tasks so that they can focus on what matters the most. From MySQL to Postgres databases, they can spin up an instance in just a few simple steps.

An instance can easily be created in just a few clicks by navigating to Cloud SQL in the GCP console, selecting the database of choice, and naming it and setting a password. Users can configure options like instance size, the number of cores, the amount of memory, and storage amount and type, from the same place. In addition, users can also isolate the database from failures by selecting the high availability option.

Once they click on create, the database instance spins up in just a few minutes. Once the instance is created, all the users would have to do is to connect to the database. Users can connect using Google App Engine, Compute Engine, or Container Engine or from anywhere else by authorizing the IP address or by using the Cloud SQL Proxy.

In just a few clicks and in a matter of minutes, users can easily spin up a database instance and manage routine administrative tasks.

3680

Of your peers have already watched this video.

61:30 Minutes

The most insightful time you'll spend today!

How-to

How to Move From Redshift to BigQuery Easily

Enterprise data warehouses are getting more expensive to maintain. Traditional data warehouses are hard to scale and often involve lots of data silos. Business teams need data insights quickly, but technology teams have to grapple with managing and providing that data using old tools that aren’t keeping up with demand. Increasingly, enterprises are migrating their data warehouses to the cloud to take advantage of the speed, scalability, and access to advanced analytics it offers. 

With this in mind, we introduced the BigQuery Data Transfer Service to automate data movement to BigQuery, so you can lay the foundation for a cloud data warehouse without writing a single line of code. Earlier this year, we added the capability to move data and schema from Teradata and S3 to BigQuery via the BigQuery Data Transfer Service. To help you take advantage of the scalability of BigQuery, we’ve now added a service to transfer data from Amazon Redshift, in beta, to that list. 

Data and schema migration from Redshift to BigQuery is provided by a combination of the BigQuery Data Transfer Service and a special migration agent running on Google Kubernetes Engine (GKE), and can be performed via UI, CLI or API. In the UI, Redshift to BigQuery migration can be initiated from BigQuery Data Transfer Service by choosing Redshift as a source. 

The migration process has three steps: 

  1. UNLOAD from Redshift to S3—The GKE agent initiates an UNLOAD operation from Redshift to S3. The agent extracts Redshift data as a compressed file, which helps customers minimize the egress costs. 
  2. Transfer from S3 to Cloud Storage—The agent then moves data from Amazon S3 to a Cloud Storage bucket using Cloud Storage Transfer Service. 
  3. Load from Cloud Storage to BigQuery—Cloud Storage data is loaded into BigQuery (up to 10 million files).
GCP bigquery.png
The BigQuery Data Transfer Service, showing Redshift as a source.

Watch the video to learn more!

Blog

What Differences in Functionalities Can I Expect Between Google Cloud’s SQL Database and Standard MySQL?

3959

Of your peers have already read this article.

5:30 Minutes

The most insightful time you'll spend today!

Keen to try out Google’s fully-managed database service, Cloud SQL, but aren’t sure if it provides the same service and experience you get from your current SQL database? We break down the differences.

In general, the MySQL functionality provided by a Google Cloud’s SQL Database (Cloud SQL) instance is the same as the functionality provided by a locally-hosted MySQL instance.

However, there are a few differences between a standard MySQL instance and a Cloud SQL for MySQL instance.

Unsupported Features

Unsupported Statements

Sending any of the following types of SQL statements will generate an error with the message “Error 1290: The MySQL server is running with the google option so it cannot execute this statement”:

  • LOAD DATA INFILE
  • Note that LOAD DATA LOCALINFILE is supported.
  • SELECT … INTO OUTFILE
  • SELECT … INTO DUMPFILE
  • INSTALL PLUGIN …
  • UNINSTALL PLUGIN
  • CREATE FUNCTION … SONAME …

Unsupported Statements for Second Generation Instances

The following statements are not supported because Second Generation instances use GTID replication:

  • CREATE TABLE … SELECT statements
  • CREATE TEMPORARY TABLE statements inside transactions
  • Transactions or statements that update both transactional and nontransactional tables

For more information, see the MySQL documentation.

Unsupported Functions

  • LOAD_FILE()

Unsupported Client Program Features

  • mysqlimport without using the –local option. This is because of the LOAD DATA INFILE restriction. If you need to load data remotely, use the Cloud SQL import function.
  • mysqldump using the –tab option or options that are used with –tab. This is because the FILE privilege is not granted for instance users. All other mysqldump options are supported.
  • If you want to import databases with binary data into your Cloud SQL for MySQL instance, you must use the –hex-blob option with mysqldump.
  • Although hex-blob is not a required flag when you are using a local MySQL server instance and the mysql client, it is required if you want to import any databases with binary data into your Cloud SQL instance. For more information about importing data, see Importing Data.
  • Not all MySQL options and parameters are enabled for editing as Cloud SQL flags.
  • For Second Generation instances, InnoDB is the only supported storage engine. For help with converting tables from MyISAM to InnoDB, see the MySQL documentation.
  • You cannot import or export triggers, functions, stored procedures, or views into Cloud SQL. However, you can create and use these elements on a Cloud SQL instance.

Notable MySQL Options

Cloud SQL runs MySQL with a specific set of options. If an option might impact how your applications work, we note it here for your information.

skip-name-resolve

This flag impacts how hostnames are resolved for client connections. Learn more.

Cloud SQL for PostgreSQL

Features

  • Fully managed PostgreSQL databases in the cloud, based on the Cloud SQL Second Generation platform.
  • Custom machine types with up to 416 GB of RAM and 64 CPUs.
  • Up to 10TB of storage available, with the ability to automatically increase storage size as needed.
  • Create and manage instances in the Google Cloud Platform Console.
  • Instances available in US, EU, or Asia.
  • Customer data encrypted on Google’s internal networks and in database tables, temporary files, and backups.
  • Support for secure external connections with the Cloud SQL Proxy or with the SSL/TLS protocol.
  • Data replication between multiple zones with automatic failover.
  • Import and export databases using SQL dump files.
  • Support for PostgreSQL client-server protocol and standard PostgreSQL connectors.
  • Automated and on-demand backups.
  • Instance cloning.
  • Integration with Stackdriver logging and monitoring.

Some features are not yet available for Cloud SQL for PostgreSQL:

  • Point-in-time recovery (PITR)
  • Import/export in CSV format using GCP Console or the gcloud command-line tool.

Supported Extensions

Cloud SQL for PostgreSQL supports many PostgreSQL extensions. For a complete list, see PostgreSQL Extensions.

Supported Procedural Languages

Cloud SQL for PostgreSQL supports the PL/pgSQL SQL procedural language.

Supported Languages

You can use Cloud SQL for PostgreSQL with App Engine applications running in the flexible environment that are written in Java, Python, PHP, Node.js, Go, and Ruby. You can also use Cloud SQL for PostgreSQL with external applications using the standard PostgreSQL client-server protocol.

Differences between Cloud SQL and standard PostgreSQL functionality

In general, the PostgreSQL functionality provided by a Cloud SQL instance is the same as the functionality provided by a locally-hosted PostgreSQL instance. However, there are a few differences between a standard PostgreSQL instance and a Cloud SQL for PostgreSQL instance.

Unsupported Features

  • Any features that require SUPERUSER privileges. An exception to this rule is made for the CREATE EXTENSION statement, but only for supported extensions.
  • Custom background workers
  • The psql client in Cloud Shell does not support operations that require a reconnection, such as connecting to a different database using the \c command.

Notable Differences

There are a number of PostgreSQL options and parameters that are not enabled for editing as Cloud SQL flags.

To request the addition of a configurable Cloud SQL flag, use the Cloud SQL Discussion group.

Case Study

The Divercity Story: Using Google Cloud to Achieve a More Inclusive and Sustainable Workforce

4737

Of your peers have already read this article.

2:30 Minutes

The most insightful time you'll spend today!

Divercity is using Google Cloud to build more inclusive and sustainable workforces. This blog discusses how the company is leveraging technology to create a diverse and environmentally responsible workplace.

Despite a growing number of diversity, equity, and inclusion (DEI) initiatives, Black and Latinx people remain highly underrepresented in tech. Although comprising 12.6% and 18% of the U.S. labor force respectively, Black professionals hold only 5% of tech positions, while Latinx professionals fill just 6% of tech roles.

Long-standing biases in hiring practices and non-inclusive work environments are the primary contributors to this lack of diversity. Even when underrepresented professionals are successfully recruited, invisible barriers to promotion and layoffs that disproportionally impact Black and Latinx employees make it extremely challenging for tech firms to retain top talent.

We founded Divercity to help employers build diverse workforces that are more inclusive and sustainable. With our comprehensive recruiting and retention platform, tech companies can accurately measure employee diversity and gender parity, seamlessly connect with underrepresented talent, and significantly reduce turnover.

As Divercity continues to grow, we’ll introduce new services and solutions that empower the tech world to build inclusive companies while improving compliance with state and federal equal opportunity laws. We also hope to expand the reach of Divercity to support DEI initiatives in non-tech industries and bolster recruiting underrepresented professionals in other countries as well.

Divercity Founding Team

Scaling Divercity with the help of the Google for Startups Black Founders Fund

Shortly after founding Divercity, we participated in the 2021 Techstars Workforce Development Accelerator. The incredible support and guidance we received during and after the program highlighted the importance of long-term collaboration with reliable technology partners who actively champion diversity and inclusion.

That’s why we became part of the Google for Startups Cloud Program. After completing the program, we used Google Cloud credits and Google for Startups Black Founders Fund funding to cost-effectively trial, deploy, and scale key Google Cloud solutions. In just months, we rolled out new inclusion tracking and recruiting tools on the highly secure-by-design infrastructure of Google Cloud to expedite the sourcing and hiring of underrepresented talent in the tech industry.

With the support and mentorship of the Google for Startups Cloud Program and the Black Founders Fund, Divercity is well on its way to becoming one of the industry’s most trusted sites for DEI measurement, recruitment, and retention.

Delivering predictive diversity analyses with a 99% accuracy rate

We rely on the expansive Google Cloud ecosystem to power all the services offered on the Divercity platform. Specifically, we leverage Colab to write and execute the sophisticated TensorFlow machine learning (ML) models that deliver our predictive diversity analyses with a 99% accuracy rate.

We also use BigQuery to democratize insights and run analytics at scale with 27% lower three-year TCO than cloud data warehouse alternatives. BigQuery seamlessly integrates with Looker and Data Studio to display company diversity and recruitment data on interactive dashboards—and automatically populate reports with detailed demographic information.

We also accelerate the development, launch, and management of new Divercity tools with Firebase, while taking advantage of key features such as A/B testing and messaging campaigns to boost user engagement.

In the future, we plan to explore how Google Cloud AI and machine learning products such as Vertex AI and AutoML can further refine our diversity score analyses and applicant recruiting pipeline. We’ll also continue leveraging the many resources provided by the Google Black Founders Fund, including opportunities for technical project partnerships, early access to new Google Cloud products and tools, and collaboration with dedicated Google experts.

The Google for Startups Cloud Program and Google Black Founders Fund have been invaluable to our success. Since completing the Black Founders Accelerator program, we’ve been named a top 10 HR tech product by HR Tech Outlook, significantly increased our subscriber base, and received positive feedback from investors. We can’t wait to see what we accomplish next as we empower tech companies to build more diverse, inclusive, and sustainable workforces.

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

Case Study

Held Back by Database Scalability, This Financial Services Company Switches to Google Cloud and Cloud Spanner

11557

Of your peers have already read this article.

4:30 Minutes

The most insightful time you'll spend today!

Financial services provider, Azimut Group, turns to Google Cloud and Google Cloud Spanner. The result? It can scale up databases in two minutes instead of one day, and it saves 35% on cloud provider costs.

Azimut Group operates an international network of companies handling investment and asset management, mutual funds, hedge funds, and insurance. Founded in Milan, Italy in 1988, Azimut Group today has branches in fifteen countries, including Brazil, China, and the USA.

“We have subsidiaries and manage funds all over the world,” explains Simone Bertolotti, IT Manager at Azimut Holding S.p.a. “That means that any technology that we put in place has to cover needs from many different countries.”

“When complicated analysis has to be executed, we have to increase our table space in a couple of minutes so that the AI can drill down into the data and deliver the information we need.”

Simone Bertolotti, IT Manager, Azimut Holding S.p.a.

Azimut manages its funds with investment advisors who use information sourced from Bloomberg, Reuters and others. “They use a huge amount of data,” says Simone. “They work with spreadsheets, algorithms, formulae and they analyse data in minutes.” In finance, every second is crucial, which is why Azimut decided to develop a risk management dashboard that can process information even more quickly, then distribute it worldwide.

“When an advisor manages data, that data is used to make immediate decisions on funds, capital movements or whether to sell stock,” says Simone. “They have to be ready to make recommendations for any amount of data that comes to them. For our dashboard, that means that when additional information arrives or complicated analysis has to be executed, we have to increase our table space in a couple of minutes so that the AI can drill down into the data and deliver the information we need.”

Generating insights at speed

Investors and investment managers make decisions based on the most accurate, up-to-date information possible. For Azimut Group, information sourced through financial data vendors such as Bloomberg and Reuters provided only part of the data that the group required.

“We looked to collect information from a range of different providers,” explains Simone, “then analyse it to develop a predictive algorithm that could work faster than an advisor stationed at the terminal. We set ourselves the challenge to try to manipulate that data to add new insights into our matrix, so that every one of our branches across the world can see risk information about the funds in real-time.”

“We compared Google Cloud Platform’s performance with our previous cloud provider, and saw huge benefits of switching to Google. For me, the key performance issue is scaling. With Google Cloud Platform I know that I can increase and decrease my infrastructure quickly, when I need it.

Simone Bertolotti, IT Manager, Azimut Holding S.p.a.

The first cloud provider Azimut used to build its system struggled to scale quickly to meet different kinds of data challenges. “If we wanted to add more cores, that was fine,” says Simone. “But the previous cloud provider made it complicated to raise the amount of space in a database infrastructure and scale up to demand. Scaling up for more in-depth analysis would take a day, and our need was immediate.”

That’s why Azimut switched one year ago to Google Cloud Platform to run the 150 VMs on its risk analysis platform. “We compared Google Cloud Platform’s performance with our previous cloud provider, and saw huge benefits of switching to Google. For me, the key performance issue is scaling,” says Simone. “With Google Cloud Platform I know that I can increase and decrease my infrastructure quickly, when I need it. Instead of waiting a day to scale up infrastructure, we can request and add space to our database in a couple of minutes.”

The infrastructure of Azimut’s solution handles around 800TB of data per month, and Google’s global network of servers and high-speed connections ensure that it gets to where it’s most needed by the most direct route. Impressed by the speed, security and availability of Google Cloud Platform, Azimut has moved its intranet on to Google Cloud Platform, too, eliminating the need for staff to login with VPNs.

“Instead of waiting a day to scale up infrastructure, we can request and add space to our database in a couple of minutes.”

Simone Bertolotti, IT Manager, Azimut Holding S.p.a.

Driving ahead with Noovle

For Azimut, migrating the risk management dashboard is the latest of many Google product collaborations with cloud consultancy Noovle. “Everything started five years ago,” says Simone, “when Noovle assisted us in migrating to Gmail from our on-premise email solution. From G Suite to Google Cloud Platform, we’ve had a great relationship. Noovle provides consultancy services, support for mobility, and external advisors who work on our premises, such as when they trained us how to broadcast our meetings on Google Hangouts. As an independent company, we know we can trust them for transparent advice. All they care about is the best way to get a job done and to help us reach our goals.”

New app, new customers

In a business case comparison, Google Cloud Platform cost Azimut 35% less to run than the previous cloud provider. Now the group is building a major new mobile application on Google App Engine to be released in 2018.

“The new mobile application will allow customers to trade directly, without human advisors, by proposing different investment solutions depending on targets the customers set,” says Simone. “So if a customer aims to make money with investments, they enter their relevant personal information and we carry out the necessary regulatory checks and suggest what they could buy. The entire project will be based on Google Cloud Platform, so customers can control their investments through the app while we manage the fund, using Google Cloud Spanner on the backend.”

Blog

Gartner Positions Google as the Leader for Cloud Database Management Systems!

3597

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

Second time in the row, Google earns Gartner® Magic Quadrant™ for Cloud Database Management Systems (DBMS) for its unified capabilities across transactional and analytics use cases and innovations in data management, security and more!

We are thrilled that Gartner has positioned Google as a Leader for the second year in a row in the 2021 Gartner® Magic Quadrant™ for Cloud Database Management Systems (DBMS).

gartner DBMS MQ 2021.jpg

We believe the report evaluated Google’s unified capabilities across both transactional and analytical use cases, and showcases innovation progress in areas like data management consistency, high speed processing and ingestion, security, elasticity, advanced analytics, and more. 

With the recent announcement of Dataplex, organizations can centrally manage, monitor, and govern their data across data lakes, data warehouses, and data marts with consistent controls.  Solutions like BigQuery ML provide a “built-in” approach for advanced analytics capabilities and Analytics Hub offer the infrastructure customers need to share data analytics solutions securely and at scale in ways never before achieved. For example, over a seven-day period in April, more than 3,000 different organizations shared over 200 petabytes of data using BigQuery. 

Research shows 90% of organizations have a multicloud strategy, which is why we have invested in a cross-cloud data analytics solution for Google Cloud, AWS, and Azure with BigQuery Omni. Additionally, our progress with Anthos and our Distributed Cloud this past year further advance our ability to support multi and hybrid cloud scenarios. To gain a competitive advantage using data, organizations need a data platform that transcends transactional and analytical workloads and can be run with the highest level of reliability, availability and security. Cloud Spanner, our globally distributed relational database has redefined the scale, global consistency, and availability of Online Transaction Processing (OLTP) systems. Spanner processes over 1 billion requests per second at peak, and has been battle-tested with some of the most-demanding applications, including Google services such as Search, YouTube, Gmail, Maps, and Payments. And what’s unique about our core services Spanner and BigQuery, is that they leverage common infrastructure such as our highly durable distributed file system (Colossus), our large-scale cluster management system (Borg), and Jupiter, our high-performance networking infrastructure, enabling features such as federation between Spanner and BigQuery.

We remain focused on the integration of Google Trends, Maps, Search, Ads, and have increased industry domain expertise in areas such as retail, financial services, healthcare, and gaming. We’re continuing to develop industry white papers such as this one – How to develop Global Multiplayer Games using Cloud Spanner – and we’re proud of the work the team has done to create and share industry and horizontal architecture patterns built from industry leaders to serve as solution accelerators for customer use cases. 

Innovation momentum continues with a unified and open data cloud

We continue to innovate across our data cloud portfolio, especially with the innovations we announced at Google Cloud NEXT’21. BigQuery Omni is now available for AWS and Azure, supporting cross-cloud analytics for customers. We’ve added additional capabilities for enterprise data management and governance with Dataplex, which recently went GA. We’ve made migrations to Cloud SQL easier and faster with the Database Migration Service. More than 85% of all migrations are underway in under an hour, with the majority of customers migrating their databases from other clouds. We are embracing openness with Spanner by adding a PostgreSQL interface, allowing enterprises to take advantage of Spanner’s unmatched global scale, 99.999% availability, and strong consistency using skills and tools from the popular PostgreSQL ecosystem. We are also automating data processing with Spark on Google Cloud, which enables developers to spend less time on infrastructure management and more time on data science, modeling, and delivering business value. Finally, we announced Google Earth Engine on Google Cloud, allowing customers to integrate Earth Engine with BigQuery, Google Cloud’s ML technologies, and the Google Maps Platform. 

With these innovations, enterprises like PayPal, Deutsche Bank, and Equifax use Google Cloud to solve for their end-to-end data lifecycle use cases. Organizations like Telefonica use Google Cloud to deliver new customer experiences. Telefónica has transformed every aspect of how they store, share, and analyze data while doubling processing power and lowering costs.

We continue to support an open ecosystem of data partners including Informatica, Tableau, MongoDB, Neo4j, C3.ai, and Databricks, giving customers the flexibility of choice to build their data clouds without being locked into a single approach. 

We are honored to be a Leader in the 2021 Gartner Magic Quadrant for Cloud Database Management Systems (DBMS), and look forward to continuing to innovate and partner with you on your digital transformation journey. 

Download the complimentary 2021 Gartner Magic Quadrant for Cloud Database Management Systems report.

Learn more about how organizations are building their data clouds with Google Cloud solutions. 


Gartner, Magic Quadrant for Cloud Database Management Systems, Henry CookMerv AdrianRick GreenwaldAdam RonthalPhilip Russom, 14 December 2021.

Gartner and Magic Quadrant are registered trademarks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.

Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s Research & Advisory organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Google Cloud.

More Relevant Stories for Your Company

Case Study

Cloud Bigtable Helps Fraud-detection Company Meet Scalability Demands and Secure Customer Data

Editor’s note: Today we are hearing from Jono MacDougall , Principal Software Engineer at Ravelin. Ravelin delivers market-leading online fraud detection and payment acceptance solutions for online retailers. To help us meet the scaling, throughput, and latency demands of our growing roster of large-scale clients, we migrated to Google Cloud and its

Blog

Want to Code for the Cloud? Get Started with the Native App Development Track

Earlier this year, we launched the Google Cloud skills challenge, which provides 30 days of free access to training to build your cloud knowledge and an opportunity to earn skill badges that showcase your Google Cloud competencies. Today, we’re adding a Native App Development track to the skills challenge, joining the Getting Started,

Case Study

Combining IoT and Analytics to Warn Manufacturers of Line Break Downs and Increase Profitability

Oden Technologies is using the Internet of Things (IoT) to improve the factories of today. The giant network of “things” (including people) connected to each other via the Internet has the potential to reduce waste, increase efficiency, and improve safety across all walks of life. Oden is leading IoT innovation in

Case Study

Customer Voices: How Firms from Across Industries Leverage Google Cloud

From powering everyday operations and accelerating application innovation, to providing tools for specific business needs and executing on big ideas, to advancing the security of technology solutions, companies from across industries have leveraged Google Cloud for business benefits. Companies from across industries have turned to Google Cloud for transforming their

SHOW MORE STORIES