Whether driven by government policy, industry regulation, or geo-political considerations, the evolution of cloud computing has led organizations to want even more control over their data and more transparency from their cloud services. At Google Cloud, one of the best tools for achieving that level of control and transparency is a bit of technological magic we call Cloud External Key Manager (EKM).

Cloud EKM can help you protect your cloud data at rest with encryption keys which are stored and managed in a third-party key management system that’s outside Google Cloud’s infrastructure, and ultimately outside Google’s control. This can help you achieve full separation between your encryption keys and your data stored in the cloud. Cloud EKM works with symmetric and asymmetric encryption keys, and offers organization policies that allow for fine-grained control over what types of keys are used. Via Key Access Justification (KAJ) it also offers a way for clients to control each key use.

At their core, many cloud security and cloud computing discussions are about the kinds of trust that Cloud EKM specifically and encryption more broadly can help create. While the concept of digital trust is much bigger than cybersecurity and its tripartite components of security, privacy, and compliance, one of the most crucial themes of cloud computing is the cloud trust paradox. In order to trust the cloud more, you must be able to trust it less, and external control of keys and their use can help reduce concerns over unauthorized access to sensitive data.

How it works

From our Cloud EKM documentation, you can use keys that you manage within a supported external key management partner to protect data within Google Cloud. You can protect data at rest in services that support CMEK, or by calling the Cloud Key Management Service API directly.

Cloud EKM provides several benefits:

• Key provenance: You control the location and distribution of your externally-managed keys. Externally-managed keys are never cached or stored within Google Cloud. Google cannot see them. Instead, Cloud EKM communicates directly with the external key management device for each request.
• Access control: You manage access to your externally-managed keys. Before you can use an externally-managed key to encrypt or decrypt data in Google Cloud, you must grant the Google Cloud project access to use the key. You can revoke this access at any time.
• Centralized key management: You can manage your keys and access policies from a single location and user interface, whether the data they protect resides in the cloud or on your premises. The system that managed the keys is entirely outside Google control.

In all cases, the key resides on the external system, and is never sent to Google.

Here’s how it works:

1. Create or use an existing key in a supported external key management partner system. This key has a unique URI.
2. Grant your Google Cloud project access to use the key, in the external key management partner system.
3. Create a Cloud EKM key in your Google Cloud project, using the URI for the externally-managed key.

The Cloud EKM key and the external key management partner key work together to protect your data. The external key is never exposed to Google and cannot be accessed by Google employees. Furthermore, Cloud EKM can be combined with Key Access Justifications (KAJ) to establish cryptographic control over data access.

KAJ with Cloud EKM can give customers the ability to deny Google Cloud administrators access to their data at rest for any reason, even in situations typically exempted from customer control, such as outages or responses to third-party data requests. KAJ does this by providing customers a clear reason why data is being decrypted, which they can use to programmatically decide whether to permit decryption and thus allow access to their data.

Previously, we’ve discussed three patterns where keeping the keys off the cloud may in fact be truly necessary or outweighs the benefits of cloud-based key management. Here’s a brief summary of those three scenarios where Cloud EKM can help solve these Hold Your Own Key dilemmas.

Scenario 1: The last data to go to the cloud

As organizations complete their digital transformations by migrating data processing workloads to the cloud, there is often a pool of data that can not be moved to the cloud. Perhaps it’s the most sensitive data, the most regulated data, or the data with the toughest internal security control requirements.

Finance, healthcare, manufacturing and other heavily-regulated organizations face myriad risk, compliance, and policy reasons that may make it challenging to send some of their data to a public cloud provider. However, the organization may be willing to migrate this data set to the cloud as long as it is encrypted and they have sole possession of the encryption keys.

Scenario 2: Regional regulations and concerns

Regional requirements are playing a larger role in how organizations migrate to and operate workloads in the public cloud.

Some organizations are already facing situations where they are based in one country and want to use a cloud provider based in a different country, but they aren’t comfortable with or legally allowed to give the provider access to encryption keys for their stored data. Here the situations are more varied, and can include an organization’s desire to stay ahead of evolving regulatory demands or industry-specific mandates.

Ultimately, this scenario allows organizations to utilize Google Cloud while keeping their encryption keys in the location of their choice, and under their physical and administrative control.

Scenario 3: Centralized encryption key control

The focus here is on operational efficiency. Keeping all the keys within one system to cover multiple cloud and on-premise environments can help reduce overhead and attack surface, thus helping to improve security. As Gartner researchers concluded in their report, “Develop an Enterprisewide Encryption Key Management Strategy or Lose the Data1,” organizations are motivated to reduce the number of key management tools.

“By minimizing the number of third-party encryption solutions being deployed within an environment, organizations can focus on establishing a cryptographic center of excellence,” Gartner researchers said

Given that few organizations are 100% cloud-based today for workloads that require encryption, keeping keys on-prem can streamline key management. Centralizing key management can give the cloud user a central location to enforce policies around access to keys and access to data-at-rest, while a single set of keys can help reduce management complexity. A properly implemented system with adequate security and redundancy outweighs the need to have multiple systems.

Do I need Cloud EKM?

Whether protecting highly sensitive data, retaining key control to address geopolitical and regional concerns, or supporting hybrid and multi-cloud architectures, Cloud EKM is best suited for those Google Cloud customers who must keep their encryption keys off of the cloud and always under their full control.

To learn more about Cloud EKM, please review these resources:

• Our research explaining why Google Cloud users can benefit from Cloud EKM
• The most recent updates to Cloud EKM
• Take a deeper dive into the cloud trust paradox

1. Gartner, Develop an Enterprisewide Encryption Key Management Strategy or Lose the Data, David Mahdi, Brian Lowans, March 2022.

Many of the world’s leading companies run on SAP—and deploying it on Google Cloud extends the benefits of SAP even further. Migrating your current SAP S/4HANA deployment to Google Cloud—whether it resides on your company’s on-premises servers or another cloud service—provides your organization with a flexible virtualized architecture that lets you scale your environment to match your workloads, so you pay only for the compute and storage capacity you need at any given moment. Google Cloud includes built-in features, such as Compute Engine live migration and automatic restart, that minimize downtime for infrastructure maintenance. And it allows you to integrate your SAP data with multiple data sources and process it using Google Cloud technology such as BigQuery to drive data analytics.

SAP server-side architecture consists of two layers: the SAP HANA database, and the Netweaver application layer. In this blog post, we’ll look at the options and steps for moving the database layer to Google Cloud as a lift and shift or rehost, a straightforward approach that entails moving your current SAP environment unchanged onto Google Cloud.

Deploying an SAP HANA system on Google Cloud

Google Cloud offers SAP-certified virtual machines (VMs) optimized for SAP products, including SAP HANA and SAP HANA Enterprise Cloud, as well as dedicated servers for SAP HANA for environments greater than 12TB. (For a complete list of VM and hardware options, visit the Certified and Supported SAP HANA Hardware Directory.)

Before proceeding with a rehost migration to Google Cloud, your current (source) environment and Google Cloud (target) environments should meet these specifications:

Prerequisites:  

• The configuration of the Google Cloud environment (i.e., VM  resources, SSD storage capacity) should be identical to that of the source environment. If the underlying hardware is different, however, you must use Option 2 for your migration, detailed below.
• Both environments should be running the same operating system (SUSE or RHEL Linux).
• The HANA version, instance number, and system ID (SID) should be identical.

• Schema names must remain the same.
• Establishing the network connection between the on-premises environment and Google Cloud will be required in this phase to support rehost of the SAP application.you can use Cloud VPN or Dedicated Interconnect. Learn more about Dedicated Interconnect and Cloud VPN.

Note: Depending on your internet connection and bandwidth requirements, we recommend using a Dedicated Interconnect over Cloud VPN for production environments. 

We offer a number of automated processes to accelerate your cloud journey. To deploy the SAP HANA system on Google Cloud, you can use the Google Cloud Deployment manager or Terraform and Ansible scripts available on GitHub with configuration file templates to define your installation. For more details, see the Google Cloud SAP HANA Planning Guide.

Note: To deploy SAP HANA on Google Cloud machine types that are certified by SAP for production, please review the Certification for SAP HANA on Google Cloud page. 

Moving an SAP HANA Database to Google Cloud

There are two different options you can use to rehost your SAP HANA database to Google Cloud, and each has pros and cons that you should consider when deciding on your approach.

Option 1: Asynchronous replication uses SAP’s built-in replication tool to provide continuous data replication from the source system (also known as the primary system) to the destination or secondary system—in this case residing on Google Cloud. It’s best for mission-critical applications for which minimum downtime is a high priority, and for large databases. In addition, the high level of automation means that the process requires less manual intervention. Here’s where you can learn more on HANA Asynchronous Replication.

Option 2: Backup and restore relies on SAP’s backup utility to create an image of the database that is then transferred to Google Cloud, where it is restored in the new environment. Downtime for this method varies by database size, so large databases may require more downtime via this method vs. asynchronous replication. It also involves more manual tasks. However, it requires fewer resources to perform, making it an attractive option for less urgent use cases. Here’s where you can learn more on SAP HANA database Backup and restore.

How to migrate the SAP HANA database to Google Cloud using Asynchronous Replication

1. Create and configure Dedicated Interconnect or Cloud VPN between the current environment and Google Cloud.
2. Set up SAP HANA asynchronous replication. You can configure system replication using SAP HANA Cockpit, SAP HANA Studio, or hdbnsutil. See Setting Up SAP HANA System Replication in the SAP HANA Administration Guide.
3. Be sure to use the same instance number and HANA SID in the template as the primary instance.
4. Configure the Google Cloud instance as the secondary node for using HANA Asynchronous replication.
5. Perform data validation once full data replication is completed to the SAP HANA database in Google Cloud. To learn more: HANA System Replication overview.  
6. Perform an SAP HANA takeover on your standby database. This switches your active system from the current primary system onto the secondary system on Google Cloud. Once the takeover command runs, the system on Google Cloud becomes the new primary system.To learn more: HANA Takeover. 

How to migrate the SAP HANA database to Google Cloud using Backup and Restore

1. Create a full backup of your SAP HANA database in your current environment.
2. Create a new storage bucket in your Google Cloud environment. Visit Creating Storage Buckets in the Google Cloud Storage documentation. 
3. Download and install gsutil onto the source environment and run it to upload the HANA backup to the Google Cloud storage bucket. To install gsutil utility on any computer or server, visit Install gsutil in the Google Cloud Storage documentation.
   Note: You can run parallel multi thread/multi processing in gsutil to copy large files more quickly.
4. Recover the HANA database on Google Cloud using SAP’s RECOVER DATABASE statement. See RECOVER DATABASE Statement (Backup and Recovery) in the SAP HANA SQL Reference Guide for SAP HANA Platform.

Note: BackInt agent is an integrated SAP interface tool used for HANA database on Google Cloud.Backint agent for SAP HANA can be used to store and retrieve backups directly from Google Cloud Storage. It is supported and certified by SAP on Google Cloud. To learn more:  SAP HANA Backint Agent on Google Cloud. 

In summary, we recommend using Asynchronous Replication (Option 1) for mission-critical applications that require the lowest downtime window. For all other applications, we recommend Backup and Restore (Option 2), as this approach requires fewer resources. It’s also a great way to implement the backup and restore functionality on Google Cloud.

A rehost migration is the most straightforward path to getting your SAP on HANA system up and running on Google Cloud. And the sooner you migrate, the sooner you can take advantage of the many benefits Google Cloud brings to your SAP solution. For more information on the different migration options please review: SAP on Google Cloud: Migration strategies. 

Learn more about deploying SAP on Google Cloud. Technical resources can be found here.

Since its initial release in 1995, MySQL has not only been the de-facto database for many data storage needs, but has gathered much attention over the years as one of the most well-recognized databases around when it comes to Relational Database Management Systems and transactional data processing.

Any application in retail, e-commerce, or banking has likely had its fair share of business depending on a relational database for its transactional needs. Many of these applications are built on MySQL due to its flexibility, open source nature, and strong community support.

Whether in the context of a migration from a relational database (like MySQL or PostgreSQL) , migration from a NoSQL database (like Cassandra), or a green grass workload, many companies turn to Cloud Spanner seeking a high availability SLA (99.99% for regional instances and 99.999% for multi region instances),  unlimited scale,  and low operational overhead – no patching required, no maintenance or other planned downtimes, just to name a few benefits.

The tooling and open source ecosystem around Spanner has evolved and grown ever since the service was introduced. HarbourBridge is a part of this ecosystem, and it’s meant to help customers port their existing MySQL or PostgreSQL schema to a Cloud Spanner schema. 

Application Migration Tips

Despite helpful tools like HarbourBridge, database migrations are never trivial. Here are a few things to pay attention to when migrating from MySQL to Spanner, and how to update your application logic to address them.

Note – The following snippets use the PHP client for Cloud Spanner. A couple of the snippets reference a partial Magento port that our friends over at Searce have been working on. Once you understand the operations, you should be able to implement the same in any of the other languages that Cloud Spanner supports.

Cloud spanner enforces strict data types  

In a MySQL query, the value of an attribute can be referenced as either a string or an integer. Example:

• select * from catalog_eav_attribute where attribute_id = 46;
• select * from catalog_eav_attribute where attribute_id = “46”;

Both are valid and equivalent. 

In Cloud Spanner, the query will return an error if you try to reference an integer type by using a string representation.

Here is an example of a working query from Cloud Spanner:

• select * from catalog_eav_attribute where attribute_id = 46; — this will work
• Select * from catalog_eav_attribute where attribute_id = “46” — will fail, since we are supplying a string with “46” 

You may use a function like the following to assist you with such transformations.

  /**
    * Formats the SQL for Cloud Spanner
    * Example
    * Input SQL : <select statement> WHERE 
    *    (`product_id` = '340') ORDER BY position  ASC
    * Output SQL : <select statement> WHERE 
    *    (`product_id` = 340) ORDER BY position  ASC
    * In the above example integer 
    *`340` is sanitized by removing single quotes.
    * Sanitization is required since Cloud Spanner
    * has strict typing
    * @param string $sql
    * @return string $sql
    */
   public function sanitizeSql(string $sql)
   {
       if (preg_match_all("/('[^']*')/", $sql, $m)) {
           $matches = array_shift($m);
           for($i = 0; $i < count($matches); $i++) {
               $curr =  $matches[$i];
               $curr = filter_var($curr, 
                   FILTER_SANITIZE_NUMBER_INT);
               if (is_numeric($curr)) {
                   $sql = str_replace($matches[$i], 
                       $curr, $sql);
               }
           }
       }
       return $sql;
   }

Using sequences in primary keys is not a Spanner best practice

Cloud Spanner does not implement a sequence generator,  and it is not a Spanner best practice to use sequential IDs because doing so can cause hotspotting.

An alternative mechanism of generating a unique primary key is to use a UUID or any other similar mechanisms that result in non-sequential values. For more information, please refer to this article.

In order to convert all existing primary keys to a UUID pattern, you can change the schema using the snippet here

You may use this code snippet to modify the application code to generate the UUID for the auto increment. There are other ways to do this as well, such as using PHP built-in uniqid function.

  /**
    * Generate UUID.
    * @return string
    */
   public function getAutoIncrement()
   {
       if (function_exists('com_create_guid') === true) {
           return trim(com_create_guid(), '{}');
       }
       return sprintf('%04X%04X-%04X-%04X-%04X-%04X%04X%04X',
           mt_rand(0, 65535), mt_rand(0, 65535),
           mt_rand(0, 65535), mt_rand(16384, 20479), 
           mt_rand(32768, 49151),  mt_rand(0, 65535),
           mt_rand(0, 65535), mt_rand(0, 65535));   
   }

Implicit casting and the need for explicit casting of field data types 

Both MySQL and Cloud Spanner follow the SQL standard, hence much of the query syntax is the same. One notable difference is that the Cloud Spanner field types are implicitly cast to an appropriate data type out of the ones mentioned here. When implicit casting of the field type fails, Spanner returns a read error, so it would be safer to perform the casting to the appropriate type when issuing the SELECT statement.

Modify application code to cast to respective data type before execution of query.

  $con = $this->getSpannerConnection();
 /**
 * Cloud Spanner follows strict type so cast the columns appropriately
*/
$select = $con->addCast($select, "`t_d`.`value`", 'string');
$select = $con->addCast($select, "`t_s`.`value`", 'string');
$select = $con->addCast($select, "IF(t_s.value_id IS NULL, 
    t_d.value, t_s.value)", 'string');
$values = $con->fetchAll($select);

Modify the application code to cast the column to its respective type 

  /**
    * Cast the column with type
    * @param string $sql
    * @param string $col
    * @param string $type
    * @return string
    */
   public function addCast(string $sql,
       string $col, string $type)
   {
      $cast = "cast(".$col." as ".$type.")";
      return str_replace($col, $cast, $sql);
   }

Using interleaved tables to improve read performance

Cloud Spanner’s table interleaving is a great choice for many parent-child relationships where the child table’s primary key includes the parent table’s primary key columns. Interleaving ensures that child rows are collocated with their parent rows, which can significantly improve query performance.

Refer to the statements here for a few samples of creating interleaved tables. To learn more about table interleaving, visit the documentation.

Conclusion

Database migrations are complicated.  Hopefully, using HarbourBridge and the  list of tips in this article can make that task easier.  For additional tips, please take a look at this migration guide and read more about HarbourBridge.

Whether with the cloud or within their own data centers, enterprises have undergone a period of remarkable consolidation and centralization of their compute resources. But with the rise of ever more powerful mobile devices, and increasingly capable cellular networks, application architects are starting to think beyond the confines of the data center, and looking out to the edge. 

What exactly do we mean by edge? Think of the edge as distributed compute happening on a wide variety of non-traditional devices — mobile phones of course, but also equipment sensors in factories, industrial equipment, or even temperature and reaction monitoring in a remote lab. Edge devices are also connected devices, and can communicate back to the mothership over wireless or cellular networks. 

Equipped with increasingly powerful processors, these edge devices are being called upon to perform tasks that have thus far been outside the scope of traditional IT. For enterprises, this could mean pre-processing incoming telemetry in a vehicle, collecting video in kiosks at a mall, gathering quality control data with cameras in a warehouse, or delivering interactive media to retail stores. Enterprises are also relying on edge to ingest data from outposts or devices that have even more intermittent connectivity, e.g., oil rigs or farm equipment, filtering that data to improve quality, reducing it to right-size information load, and processing it in the cloud. New data and models are then pushed back to the edge; in addition, we can also push configuration, software, and media updates and decentralize processing workload.

Edge isn’t all about enabling new use cases – it’s also about right-sizing environments and improving resource utilization. For example, adopting an edge model can also relieve load on existing data centers. 

But while edge computing is full of promise for enterprises, there are many pieces that are still works in progress. Further, developing edge workloads is very different from developing traditional applications, which enjoy the benefits of persistent data connections and run on well-resourced hardware platforms. As such, cloud architects are still in the early days of figuring out how to use and implement edge for their organizations. 

Fortunately, there are tools you can use to help ease the transition to edge computing — and that likely fit into your organization’s existing computing systems. Kubernetes, of course, but also higher level management tools like Anthos, which provides a consistent control plane across cloud, private data center and edge locations. Other parts of the Anthos family – Anthos Config Management and Anthos Service Mesh — go one step further and provide consistent, centralized management to your edge and cloud deployments. And there’s more to come.

For the remainder of this blog post, we’ll dive deeper into the past and current state of edge computing, and the benefits that architects and developers can expect to see from edge computing. In a next post, we’ll take a deeper look at some of the challenges that designing for edge introduces, and some of the advantages the average enterprise has in adopting the edge model. Finally, we’ll look at the Google Cloud tools that are available today to help you build out your edge environment, and look at some early customer examples that highlight what’s possible today — and that will spark your imagination for what to do tomorrow. 

The evolution of edge computing 

The edge is not a new concept. In fact, it’s been around for the last two decades, spanning many use cases that are prevalent today. One of the first applications for edge was to use content delivery networks (CDN) to cache and serve daily static website pages near clients, for example, web servers in California data centers serving financial data to European customers. 

As connectivity has improved and software evolved, the edge has evolved too, and the focus has shifted towards using edge to distribute services. First, simple services expanded from static HTML to javascript libraries or image repositories. Common functions like image transformation, credit and address validation support services followed. Soon, organizations were deploying more complex cloudlet and clustered microservices installations, as well as distributed and replicated datasets. The term “endpoint” became ubiquitous, and APIs profilerated. 

In parallel, there’s been an explosion of creativity in hardware, microcontrollers and dedicated edge devices. Fit-for-purpose products were deployed globally. Services like Google Cloud IoT Core extended our ability to manage and securely connect these dispersed devices, allowing platform managers to register tools and leverage managed services like Pub/Sub and Dataflow for data ingestion. And with Kubernetes, large remote clusters — mini private clouds in and of themselves — operate as self-healing, autoscaling services across the broader internet, opening the door to new models for applications and architectural patterns. In short, both distributed asynchronous systems and economies have blossomed.

What does this mean for enterprises? For the purposes of this series, edge means you can now go beyond the corporate network, beyond cloud VPCs, and beyond hybrid. The modern edge is not sitting at a major remote data center, nor is it a CDN, cloud provider, or in a corporate data center rack — it’s just as likely to look like 100 of these attached to a thousand sensors.

Edge, in short, is about having hardware and devices installed at remote locations that can process and communicate back the information they collect and generate. The edge management challenge, meanwhile, is being able to push configuration and software/model/media updates to these remote locations when they are connected.

Enable new use cases

Today, we have reached a new threshold for edge computing — one where micro-data-processing centers are deployed as the edge of a fractal arm, as it were. Together, they form a broad, geographically distributed, always-on framework for streaming, collecting, processing and serving asynchronous data. This big, loosely coupled application system lives, breathes and grows. Always changing, always learning from the data it collects — and always pushing out updated models when the tendrils are connected. 

Right now, the rise of 5G is pushing the limits of edge even further. Devices enabled with 5G can transmit using a mobile network — no ISP required — enabling connectivity anywhere within reach of a cell tower. Granted, these networks have lower bandwidth, but they are often more than adequate for certain types of data, for example fire sensors in forests bordering remote towns that emit temperature or carbon monoxide data periodically. Recently, Google Cloud partnered with AT&T to enhance business use of 5G edge technology but there is so much more that can be done. 

Reduce data center investments

In addition to enabling the digitization of a broad range of new use cases, adopting edge can also benefit your existing data center.

Let’s face it: data centers are expensive to maintain. Moving some data center load to edge locations can reduce your data center infrastructure investment, as well as compute time spent there. Edge services tend to have much lower service level objectives (SLOs) than data center services, driving lower levels of hardware investment. Edge installations also tend to tolerate disconnectedness, and thus function perfectly well with lower SLOs — and lower costs. 

Let’s look at an example of where edge can really reduce costs: big data. Back in the day, we used to build monolithic serial processors — state machines — that had to keep track of where they were in processing in case of failure. But time and again, we’ve seen that smaller, more distributed processing can break down big, expensive problems into smaller, more cost-effective chunks. 

Starting with the explosion of MapReduce almost 20 years ago, big-data workloads were parallelized across clusters on a network, and state management was simplified with intermediate output to share, wait for, or restart processing from checkpoints. Those monolithic systems were replaced by cheaper, smarter, networked clusters and data repositories where parallel work could be executed and rendered into workable datasets. 

Flash forward to today, and we are seeing those same concepts applied and distributed to edge data-collection points. In this evolution of big data processing, we are scaling up and out to the point where observation data is so massive that it must first be prefiltered, and then preprocessed down to a manageable size and still be actionable. Only then should it be written back to the main data repositories for more resource-intensive processing and model building.

In short, data collection, cleanup, and potentially initial aggregation happens at the edge location, which reduces the amount of junk data sitting in costly data stores. This increases performance of the core data warehouse, and reduces the size and cost of network transfers and storage! 

The edge is a huge opportunity for today’s enterprises. But designing environments that can make effective use of the edge isn’t without its challenges. Stick around for part two of this series, where we look at some of the architectural challenges typically encountered while designing for the edge and how we begin to address them.