With summer just around the corner, things are really heating up. But you’re in luck because this month BigQuery is supplying a cooler full of ice cold refreshments with this release of user-friendly SQL capabilities. 

We are pleased to announce three categories of BigQuery user-friendly SQL launches: Powerful Analytics Features, Flexible Schema Handling, and New Geospatial Tools.

Powerful Analytics Features

These powerful SQL analytics features provide greater flexibility to analysts for organizing, filtering, and rendering data in BigQuery than ever before. You can enable spreadsheet-like functionality on summarized data using PIVOT and UNPIVOT and filter irrelevant data in analytic functions using QUALIFY.

Through this section, we will become familiar with these new features through examples using the BigQuery Public dataset, usa_names.

PIVOT/UNPIVOT (Preview)

One of the most time-consuming tasks for data analytics practitioners is wrangling data into the right shape. SQL is great for wrangling data, but sometimes you want to reformat a table as you would in a spreadsheet, pivoting rows and columns interchangeably. To support this use case, we are pleased to introduce PIVOT and UNPIVOT operators in BigQuery. PIVOT creates columns from unique values in rows by aggregating values, and UNPIVOT reverses this action.The example below uses PIVOT on bigquery-public-data.usa_names.usa_1910_current to show the number of males and females born each year, representing each gender as a column. Then UNPIVOT reverses this action.

Language: SQL

  -- we start with SQL to create a simple table
-- we only include gender, year, and number. 
CREATE TABLE
  mydataset.sampletable1 AS (
  SELECT
    Gender,Year,SUM(Number) AS Number
  FROM
    `bigquery-public-data.usa_names.usa_1910_current`
  WHERE
    Year >= 2017
  GROUP BY
    Gender, Year);
-- The resulting table:
--+----------------------------------------+
--|   Gender   |    Year    |    Number    |
--+----------------------------------------+
--|      F     |    2019    |    1353716   |
--|      F     |    2017    |    1403989   |
--|      F     |    2018    |    1380382   |
--|      M     |    2018    |    1568678   |
--|      M     |    2019    |    1538056   |
--|      M     |    2017    |    1604609   |
--+----------------------------------------+
-- use PIVOT to create columns for “female” and “male” 
CREATE TABLE
  mydataset.Pivoted AS
SELECT
  year, male, female
FROM
  mydataset.sampletable1 
PIVOT( SUM(Number) FOR gender IN ('M' AS male,
  'F' AS female))
ORDER BY
  year;
-- The resulting pivoted table:
--+----------------------------------------+
--|    Year    |   female   |     male     |
--+----------------------------------------+
--|    2017    |   1403989   |    1604609  |
--|    2018    |   1380382   |    1568678  |
--|    2019    |   1353716   |    1538056  |
--+----------------------------------------+
-- UNPIVOT reverses the row/column rotation of PIVOT.
SELECT
  *
FROM
  mydataset.Pivoted 
UNPIVOT(number FOR gender IN (male AS 'M',
  female AS 'F'));

QUALIFY (Preview)

More advanced users of SQL know the power of analytic functions (aka window functions). These functions compute values over a group of rows, returning a single result for each row. For example, customers use analytic functions to compute a grand total, subtotal, moving average, rank, and more. With the announcement of support for QUALIFY, BigQuery users can now filter on the results of analytic functions by using the QUALIFY clause. 

QUALIFY belongs in the family of query clauses used for filtering along with WHERE and HAVING. The WHERE clause is used to filter individual rows in a query. The HAVING clause is used to filter aggregate rows in a result set after aggregate functions and GROUP BY clauses. The QUALIFY clause is used to filter results of analytic functions. 

To show the utility of QUALIFY, the example below uses QUALIFY to return the top 3 female names from each year in the last decade using from bigquery-public-data.usa_names.usa_1910_current. 

Language: SQL

  -- QUALIFY filters the result of the RANK function
SELECT
  name,year,SUM(number) AS total,
  RANK() OVER (PARTITION BY year 
  ORDER BY SUM(number) DESC) AS rank
FROM
  `bigquery-public-data.usa_names.usa_1910_current`
WHERE
  gender = 'F'
  AND YEAR >= 2010
GROUP BY 1,2 
QUALIFY RANK <= 3
ORDER BY 2,4;

Flexible Schema Handling

New SQL for administrators and data engineers enables table renaming for data pipeline processes, as well as flexible column management.

Table Rename (GA)

In data pipeline processes, tables are often created and then renamed so that they can make way for the next iteration of the pipeline run. To accomplish this, customers need a mechanism by which they can create a table and then subsequently rename it. Now if customers want to change this name using SQL, they can. Using the simple syntax that ALTER TABLE RENAME TO provides, customers will be able to rename a table after creation to clear the way for the next iteration of tables in the data pipeline.

Language: SQL

  -- create a sample table “tablename” in “mydataset”. 
-- You will rename this table.
CREATE OR REPLACE TABLE dataset.tablename(
    col1 STRING, 
    col2 NUMERIC);
-- if this table “tablename” becomes obsolete
-- perform Table Rename to “obsoletetable”
ALTER TABLE
  mydataset.name RENAME TO obsoletetable;

DROP NOT NULL constraints on a column (GA)

While BigQuery has historically provided many tools available in the UI, CLI and APIs, we know that many administrators prefer interfacing with the database using SQL. BigQuery recently released DDL statements which enable data administrators to provision and manage datasets and tables, greatly simplifying provisioning and management. Today, we continue the next addition in this line of releases by announcing ALTER COLUMN DROP NOT NULL constraint on a column:

• ALTER COLUMN DROP NOT NULL allows the administrator to remove the NOT NULL constraint from a column in BigQuery.

Language: SQL

  -- create a table to store credit card numbers
-- the business requires this field, so 
-- include a NOT NULL constraint 
CREATE TABLE
  mydataset.customers(credit_card_number STRING NOT NULL);
-- if needs of the business no longer require this field,
-- the customer can allow null entries in this column
-- by dropping the constraint
ALTER TABLE
  mydataset.customers 
ALTER COLUMN credit_card_number DROP NOT NULL;

CREATE VIEW with column list (GA)

Views are used ubiquitously by BigQuery customers to capture business logic. Oftentimes, BigQuery users have business requirements to assign aliases to columns in views. Now BigQuery supports doing so upon view creation in a column name list format with the release of CREATE VIEW with column list syntax.

Language: SQL

  -- aliases list1 and list2 can be assigned in a list format
CREATE VIEW
  myview (list1, list2) AS
SELECT
  column_1, column_2
FROM
  mydataset.exampletable

New Geospatial Tools

ST_POINTN, ST_STARTPOINT, and ST_ENDPOINT

Geospatial data is incredibly valuable to data analytics customers dealing with data from the physical world. BigQuery has very strong geospatial function support to help customers process marketing data, track storms, or manage self-driving cars. Particularly for analyzing vehicle or location tracking data, we’re thrilled to provide three new functions to allow users to easily extract or filter on key points:  

• ST_POINTN
• ST_STARTPOINT
• ST_ENDPOINT

For example, when working with vehicle histories,  ST_POINTN, ST_STARTPOINT, and ST_ENDPOINT allow users to extract elements such as the start and the end of a trip. For identifying origin-destination pairs these functions will make that task much easier.

Language: SQL

  -- pull the first, second, penultimate and final points 
-- from a linestring
WITH linestring as (
    SELECT ST_GeogFromText('linestring(1 1, 2 1, 3 2, 3 3)') g
)
SELECT
  ST_StartPoint(g) AS first, ST_EndPoint(g) AS last,
  ST_PointN(g,2) AS second, ST_PointN(g, -2) as second_to_last
FROM
  linestring
+--------------+--------------+--------------+----------------+
| first        | last         | second       | second_to_last |
+--------------+--------------+--------------+----------------+
| POINT(1 1)   | POINT(3 3)   | POINT(2 1)   | POINT(3 2)     |
+--------------+--------------+--------------+----------------+

As sure as a hot summer day pairs well with an ice-cold beverage, these new user-friendly SQL features in BigQuery pair well with your data analytics workflows. To learn more about BigQuery, visit our website, and get started immediately with the free BigQuery Sandbox.

Many of the world’s leading companies run on SAP—and deploying it on Google Cloud extends the benefits of SAP even further. Migrating your current SAP S/4HANA deployment to Google Cloud—whether it resides on your company’s on-premises servers or another cloud service—provides your organization with a flexible virtualized architecture that lets you scale your environment to match your workloads, so you pay only for the compute and storage capacity you need at any given moment. Google Cloud includes built-in features, such as Compute Engine live migration and automatic restart, that minimize downtime for infrastructure maintenance. And it allows you to integrate your SAP data with multiple data sources and process it using Google Cloud technology such as BigQuery to drive data analytics.

SAP server-side architecture consists of two layers: the SAP HANA database, and the Netweaver application layer. In this blog post, we’ll look at the options and steps for moving the database layer to Google Cloud as a lift and shift or rehost, a straightforward approach that entails moving your current SAP environment unchanged onto Google Cloud.

Deploying an SAP HANA system on Google Cloud

Google Cloud offers SAP-certified virtual machines (VMs) optimized for SAP products, including SAP HANA and SAP HANA Enterprise Cloud, as well as dedicated servers for SAP HANA for environments greater than 12TB. (For a complete list of VM and hardware options, visit the Certified and Supported SAP HANA Hardware Directory.)

Before proceeding with a rehost migration to Google Cloud, your current (source) environment and Google Cloud (target) environments should meet these specifications:

Prerequisites:  

• The configuration of the Google Cloud environment (i.e., VM  resources, SSD storage capacity) should be identical to that of the source environment. If the underlying hardware is different, however, you must use Option 2 for your migration, detailed below.
• Both environments should be running the same operating system (SUSE or RHEL Linux).
• The HANA version, instance number, and system ID (SID) should be identical.

• Schema names must remain the same.
• Establishing the network connection between the on-premises environment and Google Cloud will be required in this phase to support rehost of the SAP application.you can use Cloud VPN or Dedicated Interconnect. Learn more about Dedicated Interconnect and Cloud VPN.

Note: Depending on your internet connection and bandwidth requirements, we recommend using a Dedicated Interconnect over Cloud VPN for production environments. 

We offer a number of automated processes to accelerate your cloud journey. To deploy the SAP HANA system on Google Cloud, you can use the Google Cloud Deployment manager or Terraform and Ansible scripts available on GitHub with configuration file templates to define your installation. For more details, see the Google Cloud SAP HANA Planning Guide.

Note: To deploy SAP HANA on Google Cloud machine types that are certified by SAP for production, please review the Certification for SAP HANA on Google Cloud page. 

Moving an SAP HANA Database to Google Cloud

There are two different options you can use to rehost your SAP HANA database to Google Cloud, and each has pros and cons that you should consider when deciding on your approach.

Option 1: Asynchronous replication uses SAP’s built-in replication tool to provide continuous data replication from the source system (also known as the primary system) to the destination or secondary system—in this case residing on Google Cloud. It’s best for mission-critical applications for which minimum downtime is a high priority, and for large databases. In addition, the high level of automation means that the process requires less manual intervention. Here’s where you can learn more on HANA Asynchronous Replication.

Option 2: Backup and restore relies on SAP’s backup utility to create an image of the database that is then transferred to Google Cloud, where it is restored in the new environment. Downtime for this method varies by database size, so large databases may require more downtime via this method vs. asynchronous replication. It also involves more manual tasks. However, it requires fewer resources to perform, making it an attractive option for less urgent use cases. Here’s where you can learn more on SAP HANA database Backup and restore.

How to migrate the SAP HANA database to Google Cloud using Asynchronous Replication

1. Create and configure Dedicated Interconnect or Cloud VPN between the current environment and Google Cloud.
2. Set up SAP HANA asynchronous replication. You can configure system replication using SAP HANA Cockpit, SAP HANA Studio, or hdbnsutil. See Setting Up SAP HANA System Replication in the SAP HANA Administration Guide.
3. Be sure to use the same instance number and HANA SID in the template as the primary instance.
4. Configure the Google Cloud instance as the secondary node for using HANA Asynchronous replication.
5. Perform data validation once full data replication is completed to the SAP HANA database in Google Cloud. To learn more: HANA System Replication overview.  
6. Perform an SAP HANA takeover on your standby database. This switches your active system from the current primary system onto the secondary system on Google Cloud. Once the takeover command runs, the system on Google Cloud becomes the new primary system.To learn more: HANA Takeover. 

How to migrate the SAP HANA database to Google Cloud using Backup and Restore

1. Create a full backup of your SAP HANA database in your current environment.
2. Create a new storage bucket in your Google Cloud environment. Visit Creating Storage Buckets in the Google Cloud Storage documentation. 
3. Download and install gsutil onto the source environment and run it to upload the HANA backup to the Google Cloud storage bucket. To install gsutil utility on any computer or server, visit Install gsutil in the Google Cloud Storage documentation.
   Note: You can run parallel multi thread/multi processing in gsutil to copy large files more quickly.
4. Recover the HANA database on Google Cloud using SAP’s RECOVER DATABASE statement. See RECOVER DATABASE Statement (Backup and Recovery) in the SAP HANA SQL Reference Guide for SAP HANA Platform.

Note: BackInt agent is an integrated SAP interface tool used for HANA database on Google Cloud.Backint agent for SAP HANA can be used to store and retrieve backups directly from Google Cloud Storage. It is supported and certified by SAP on Google Cloud. To learn more:  SAP HANA Backint Agent on Google Cloud. 

In summary, we recommend using Asynchronous Replication (Option 1) for mission-critical applications that require the lowest downtime window. For all other applications, we recommend Backup and Restore (Option 2), as this approach requires fewer resources. It’s also a great way to implement the backup and restore functionality on Google Cloud.

A rehost migration is the most straightforward path to getting your SAP on HANA system up and running on Google Cloud. And the sooner you migrate, the sooner you can take advantage of the many benefits Google Cloud brings to your SAP solution. For more information on the different migration options please review: SAP on Google Cloud: Migration strategies. 

Learn more about deploying SAP on Google Cloud. Technical resources can be found here.

The ability to gain real-time insights from IoT data can redefine competitiveness for businesses. Intelligence allows connected devices and assets to interact efficiently with applications and with human beings in an intuitive and non-disruptive way. After your IoT project is up and running, many devices will be producing lots of data. You need an efficient, scalable, affordable way to both manage those devices and handle all that information. 

IoT Core is a fully managed service for managing IoT devices. It supports registration, authentication, and authorization inside the Google Cloud resource hierarchy as well as device metadata stored in the cloud, and the ability to send device configuration from other GCP or third-party services to devices. 

Main components

The main components of Cloud IoT Core are the device manager and the protocol bridges:

• The device manager  registers devices with the service, so you can then monitor and configure them. It provides:
  • Device identity management 
  • Support for configuring, updating, and controlling individual devices
  • Role-level access control
  • Console and APIs for device deployment and monitoring
• Two protocol bridges (MQTT and HTTP) can be used by devices to connect to Google Cloud Platform for:
  • Bi-directional messaging
  • Automatic load balancing
  • Global data access with Pub/Sub

How does Cloud IoT Core work?

Device telemetry data is forwarded to a Cloud Pub/Sub topic, which can then be used to trigger Cloud Functions as well as other third-party apps to consume the data. You can also perform streaming analysis with Dataflow or custom analysis with your own subscribers.

Cloud IoT Core supports direct device connections as well as gateway-based architectures. In both cases the real time state of the device and the operational data is ingested into Cloud IoT Core and the key and certificates at the edge are also managed by Cloud IoT Core. From Pub/Sub the raw input is fed into Dataflow for transformation, and the cleaned output is populated in Cloud Bigtable for real-time monitoring or BigQuery for warehousing and machine learning. From BigQuery the data can be used for visualization in Looker or Data Studio and it can be used in Vertex AI for creating machine learning models. The models created can be deployed at the edge using Edge Manager (in experimental phase). Device configuration updates or device commands can be triggered by Cloud Functions or Dataflow to Cloud IoT Core, which then updates the device.  

Design principles of Cloud IoT Core

As a managed service to securely connect, manage, and ingest data from global device fleets, Cloud IoT COre is designed to be:

• Flexible, providing easy provisioning of device identities and enabling devices to access most of Google Cloud
• IThe industry leader in IoT scalability and performance
•  Interoperable, with supports for the most common industry-standard IoT protocols

Use cases

IoT use cases range across numerous industries. Some typical examples include:

• Asset tracking, visual inspection, and quality control in retail, automotive, industrial, supply chain and logistics
• Remote monitoring and predictive maintenance in oil & gas, utilities, manufacturing, and transportation
• Connected homes and consumer technologies.
• Vision intelligence in retail, security, manufacturing, and industrial sectors
• Smart living in commercial, residential, and smart spaces 
• Smart factories with predictive maintenance and real-time plant floor analytics

 For a more in-depth look into Cloud IoT Core check out the documentation.  

https://youtube.com/watch?v=76v16P-Wqe4%3Fenablejsapi%3D1%26

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.

Enterprises want to embrace the pace of innovation and an operational business model of the modern-day cloud, but they don’t want to disrupt their existing IT landscape or upgrade all their legacy applications. This presents a conundrum, as most legacy applications were not designed to run in the cloud, and migrating them can be challenging, risky and cost-prohibitive. 

The complexity, cost and risk of cloud migration often stand in the way of a successful digital transformation. At Google Cloud, we work to meet you where you are and help you craft a migration strategy that lets you run what you want, where you want, how you want. This is why we’re excited to introduce Bare Metal Solution: to jumpstart the migration of applications that have been holding back your cloud adoption. 

Bare Metal Solution consists of all the infrastructure you need to run your specialized workload such as Oracle Database close to Google Cloud. This infrastructure is connected with a dedicated, low-latency and highly resilient interconnect, and connects to all native Google Cloud services. Bare Metal Solution uses OEM hardware that is certified to run multiple enterprise applications, most of which can be migrated to this infrastructure with little or no change, minimizing the risk of migration while simultaneously increasing its velocity.

Bare Metal Solution also comes with automation tools to help you onboard your environment quickly—provisioning your applications, relational databases, configuring popular operating systems and setting up services such as backups and monitoring. The management interface will be familiar to your existing IT teams or systems integrator, allowing you to leverage your investments in existing tools, processes and personnel.

In addition to providing state-of-the-art infrastructure and integrated low-latency access to Google Cloud, Bare Metal Solution also provides the following:

• Completely managed hardware infrastructure: End-to-end infrastructure management such as compute, storage and networking, as well as fully managed and monitored environments such as power, cooling and facilities.
• Google Cloud support and billing: A seamless support experience with support for infrastructure, including defined SLAs for initial response. 24X7 coverage for all Priority 1 and 2 issues. Unified billing across Google Cloud and Bare Metal Solution. 
• Service Level Agreements: Defined enterprise-grade SLA for hardware uptime and interconnect availability.

As an integral part of Google Cloud, Bare Metal Solution lets you off-load provisioning, managing and monitoring of your infrastructure to Google, so you can focus on your own data center modernization. 

The nitty gritty 

Many of the workloads that run on Bare Metal Solution have demanding CPU and I/O requirements. Bare Metal Solution provides a cost-effective and scalable architecture based on state-of-the-art x86 servers and high-performance, resilient storage. The servers are offered in several fixed configurations that support most mainstream enterprise operating systems:

• Dual-socket x86 systems
  • 16 core with 384 GB DRAM
  • 24 core with 768 GB DRAM
  • 56 core with 1536 GB DRAM
• Quad-socket x86 systems
  • 56 core with 1536 GB DRAM 
  • 112 core with 3072 GB DRAM

Bare Metal Solution servers can be used for standalone applications, or configured with application native or database native clustering technologies for high availability. Storage is offered in 1TB volume increments of either hybrid or all-flash disk. For applications that require custom compute shapes or special-purpose hardware, we also offer bespoke hardware configurations. 

Bare Metal Solution uses OEM hardware that is certified for many ISV software applications as well as custom built applications, including those built on Oracle Database. These hardware configurations are offered as a subscription, billed monthly with a preferred term length of 36 months. There are no data ingress and egress charges between Bare Metal Solution and Google Cloud in the same region; customers are responsible for provisioning adequate bandwidth for their business needs.

Legacy apps are no barrier to cloud

Google Cloud is the preferred destination for organizations building cloud-native applications as well as migrating existing on-premises applications, bringing reliable infrastructure, leading data analytics capabilities, and a culture of innovation to your IT environment. Whether you’re looking to build new applications in the cloud or to overhaul your infrastructure, we’re here to help you reach your digital transformation goals. And now, with Bare Metal Solution, we’re excited to extend the power of Google Cloud to specialized, legacy workloads. To learn more please visit our website.