When you choose to run your business on Google Cloud you benefit from the same planet-scale infrastructure that powers Google’s products such as Maps, YouTube, and Workspace. 

We have picked 10 ways in which Google Cloud Infrastructure services outshine alternatives in the market in how they simplify your operations, save money, and secure your data. 

1. Custom Machine Types means no wasted resources

Compute Engine offers predefined machine types that you can use when you create a VM instance. A predefined machine type has a preset number of vCPUs and a preset amount of memory; each type is billed at a set price as described on the Compute Engine pricing page. 

If predefined machine types don’t meet your needs, you can create a VM instance with a custom number of vCPUs and custom amount of memory, effectively building a custom machine type. Custom machine types are available only for general-purpose machine families. When you create a custom machine type, you are deploying a custom machine type from the E2, N2, N2D, or N1 machine family on GCP.  No other leading cloud vendor offers custom machine types so extensively.

Custom machine types are a good idea for workloads that aren’t a good fit for the predefined machine types and for workloads that require more processing power or memory but don’t need all of the upgrades provided by the next machine type level. This translates into lower operating costs.   They are also useful for controlling software licensing costs that are based on the number of underlying compute cores. 

Jeremy Lloyd, Infrastructure and Application Modernization Lead at Appsbroker, a Google partner: 

“Custom machine types coupled with Google’s StratoZone data center discovery tool provides Appsbroker with the flexibility we need to provide cost efficient virtual machines matched to a virtual machine’s actual utilization. As a result, we are able to keep our customers’ operating costs low while still providing the ability to scale as needed.”

2. Compute Engine Virtual Machines are optimized for scale-out workloads 

For scale-out workloads, T2D, the first instance type in the Tau VM family, is based on 3rd Gen AMD EPYC processors and leapfrogs VMs for scale-out workloads of any leading public cloud provider today, both in terms of performance and price-performance. Tau VMs offer 56% higher absolute performance and 42% higher price-performance compared to general-purpose VMs from any leading public cloud vendor (source). The x86 compatibility provided by these AMD EPYC processor-based VMs gives you market-leading performance improvements and cost savings, without having to port your applications to a new processor architecture. Sign up here  if you are interested in trying out T2D instances in Preview. 

For SAP HANA, Google Cloud has demonstrated with SAP how we can run the world’s largest scale-out HANA system in the public cloud (96TB).   With such innovation, you are covered as your business grows exponentially.

3. Largest single node GPU-enabled VM

Google is the only public cloud provider to offer up to 16 NVIDIA A100 GPUs in a single VM, making it possible to train very large AI models. Users can start with one NVIDIA A100 GPU and scale to 16 GPUs without configuring multiple VMs for single-node ML training, without crossing the VM layer. 

Additionally, customers can choose smaller GPU configurations—1, 2, 4 and 8 GPUs per VM—providing the flexibility to scale their workload as needed. 

The A2 VM family was designed to meet today’s most demanding applications—workloads like CUDA-enabled machine learning (ML) training and inference, for example. This family is built on the A100 GPU which offers up to 20x the compute performance compared to the previous generation GPU and comes with 40 GB of high-performance HBM2 GPU memory. To speed up multi-GPU workloads, the A2 VMs use NVIDIA’s HGX A100 systems to offer high-speed NVLink GPU-to-GPU bandwidth that delivers up to 600 GB/s. A2 VMs come with up to 96 Intel Cascade Lake vCPUs, optional Local SSD for workloads requiring faster data feeds into the GPUs and up to 100 Gbps of networking. A2 VMs provide full vNUMA transparency into the architecture of underlying GPU server platforms, enabling advanced performance tuning. Google Cloud offers these GPUs globally. 

4. ​​Non-disruptive maintenance means you worry less about planned downtime

Compute Engine offers live migration (non-disruptive maintenance) to keep your virtual machine instances running even when a host system event, such as a software or hardware update, occurs. Google’s Compute Engine live migrates your running instances to another host in the same zone without requiring your VMs to be rebooted. Live migration enables Google to perform maintenance that is integral to keeping infrastructure protected and reliable without interrupting any of your VMs. When a VM is scheduled to be live-migrated, Google provides a notification to the guest that a migration is imminent. 

Live migration keeps your instances running during:

• Regular infrastructure maintenance and upgrades
• Network and power grid maintenance in the data centers
• Failed hardware such as memory, CPU, network interface cards, disks, power, and so on. This is done on a best-effort basis; if a hardware component fails completely or otherwise prevents live migration, the VM crashes and restarts automatically and a hostError is logged.
• Host OS and BIOS upgrades
• Security-related updates
• System configuration changes, including changing the size of the host root partition, for storage of the host image and packages

Live migration does not change any attributes or properties of the VM itself. The live migration process transfers a running VM from one host machine to another host machine within the same zone. All VM properties and attributes remain unchanged, including internal and external IP addresses, instance metadata, block storage data and volumes, OS and application state, network settings, network connections, and so on. This has the benefit of reducing operational and maintenance overhead, helps you build a more robust security posture where infrastructure can be consciously revamped from a known good state and minimizes risks for advanced persistent threats. 

Refer to Lessons learned from a year of using live migration in production on Google Cloud from the Google engineering team.

5. Trusted Computing: Shielded VMs guard you against advanced, persistent attacks

Establishing trust in your environment is multifaceted, involving hardware and firmware, as well as host and guest operating systems. Unfortunately, threats like boot malware or firmware rootkits can stay undetected for a long time, and an infected virtual machine can continue to boot in a compromised state even after you’ve installed legitimate software. 

Shielded VMs can help you protect your system from attack vectors like:

• Malicious guest OS firmware, including malicious UEFI extensions
• Boot and kernel vulnerabilities in the guest OS
• Malicious insiders within your organization

To guard against these kinds of advanced persistent attacks, Shielded VMs use:

• Unified Extensible Firmware Interface (UEFI) BIOS: Helps ensure that firmware is signed and verified
• Secure and Measured Boot: Helps ensure that a VM boots an expected, healthy kernel
• Virtual Trusted Platform Module (vTPM): Establishes root-of-trust, underpins Measured Boot, and prevents exfiltration of vTPM-sealed secrets
• Integrity Monitoring: Provides tamper-evident logging, integrated with Stackdriver, to help you quickly identify and remediate changes to a known integrity state

The Google approach allows customers to deploy Shielded VMs with only a simple click, thereby easing implementation. 

6. Confidential Computing encrypts data while in use

Google Cloud was a founding member of the Confidential Computing Consortium. Along with encryption of data in transit and at rest using customer-managed encryption keys (CMEK) and customer-supplied encryption keys (CSEK), Confidential VM adds a “third pillar” to the end-to-end encryption story by encrypting data while in use. Confidential Computing uses processor-based technology that allows data to be encrypted in use while it is being processed in the public cloud. Confidential VM allows you to to encrypt memory in use on a Google Compute Engine VM by checking a single checkbox. 

All Confidential VMs support the previously mentioned Shielded VM features under the covers—you can think of Shielded VM as helping to address VM integrity, while Confidential VM addresses the memory encryption aspect which relies on CPU features. With the confidential execution environments provided by Confidential VM and AMD Secure Encrypted Virtualization (SEV), Google Cloud keeps customers’ sensitive code and other data encrypted in memory during processing. Google does not have access to the encryption keys. In addition, Confidential VM can help alleviate concerns about risk related to either dependency on Google infrastructure or Google insiders’ access to customer data in the clear. 

See what Google Cloud partners say about Confidential Computing here. 

7. Advanced networking delivers full-stack networking and security services with fast, consistent, and scalable performance

Google Cloud’s network delivers low latency, reduces operational costs and ensures business continuity, enabling organizations to seamlessly scale up or down in any region to meet business needs. Our planet-scale network uses advanced software-defined networking and security with edge caching services to deliver fast, consistent, and scalable performance. With 28 regions, 85 zones, and 146 PoPs connected by 16 subsea fiber cables around the world, Google Cloud’s network offers a full stack of layer 1 to layer 7 services for enterprises to run their workloads anywhere. Enterprises can be assured that they have best-in-class networking and security services connecting their VMs, containers, and bare metal resources in hybrid and multi-cloud environments with simplicity, visibility, and control. 

Google Cloud’s network has protected customers from one of the world’s largest DDoS attacks at 2.54 Tbps. With our multi-layer security architecture and products such as Cloud Armor, our customers ran their business with no disruptions. Furthermore, our recent integration of Cloud Armor with reCAPTCHA Enterprise adds best-in-class bot and fraud management to prevent volumetric attacks. Cloud Armor is deployed with our Cloud Load Balancer and Cloud CDN, extending the secure benefits at the network edge for traffic coming into Google Cloud so customers have security, performance, and reliability all built in. Furthermore, we are excited to offer Cloud IDS in preview, which was co-developed with security industry leader, Palo Alto Networks, to run natively in Google Cloud. 

Our advanced networking capabilities also extends to GKE and Anthos networking. With the GKE Gateway controller, customers can manage internal and external HTTPS load balancing for a GKE cluster or a fleet of GKE clusters with multi-tenancy while maintaining centralized admin policy and control. Unlike other Kubernetes offerings, we offer eBPF dataplane which brings powerful tooling such as Kubernetes network policy and logging to GKE. eBPF is known to kernel engineers as a “superpower” for its unique architecture to load and unload modules in kernel space, and now this capability is built in with Google Cloud networking. 

For observability and monitoring, our customers deploy Network Intelligence Center, Google Cloud’s comprehensive network monitoring, verification and optimization platform. With four key modules in Network Intelligence Center, and several more to come, we are working towards realizing our vision of proactive network operations that can predict and heal network failures, driven by AI/ML recommendations and remediation. Network Intelligence Center provides unmatched visibility into your network in the cloud along with proactive network verification. Centralized monitoring cuts down troubleshooting time and effort, increases network security and improves the overall user experience.  

8. Regional Persistent Disk for High Availability

Regional Persistent Disk is a storage option that provides synchronous replication of data between two zones in a region. Regional Persistent Disks can be a great building block if you need to ensure high availability of your critical applications as they offer cost-effective durable storage and replication of data between two zones in the same region. 

Regional Persistent Disks are also easy to set up within the Google Cloud Console. If you are designing robust systems or high availability services on Compute Engine, Regional Persistent Disks combined with other best practices such as backing up your data using snapshots enable you to build an infrastructure that is highly available and recoverable in a disaster. Regional Persistent Disks are also designed to work with regional managed instance groups. In the unlikely event of a zonal outage, Regional Persistent Disks allow continued I/O through failover of your workloads to another zone. Regional Persistent Disks can help meet zero RPO and near-zero RTO requirements and other stringent SLAs that your critical applications might require by maximizing application availability and protection of data during events such as host/VM failures and zonal outages. 

9. Cloud Storage’s single namespace for dual-region and multi-region means managing regional replication is incredibly simple

Similar to how Persistent Disk makes data more available by replicating data across zones, Cloud Storage provides similar benefits for object storage. Cloud Storage within a region is cross-zone by definition, reducing the risk that a zonal outage would take down your application. Cloud Storage adds to this by also providing a cross-region option that can protect against a regional outage and gets your data closer to distributed users. This comes in the form of Dual-region or Multi-region settings for a bucket. These are the simplest to implement cross-region replication offerings in the industry—just a simple button or API call to enable them. In addition to being simple to implement, they offer an added advantage of using a single bucket name that spans regions. 

This is unique in the industry. Competitive offerings currently require setting up and managing two distinct buckets, one in each region and they don’t offer the strong consistency properties Cloud Storage offers across regions. Operations and app development are burdened by this design. Google’s single namespace approach dramatically simplifies application development (the app runs on single region or dual/multi-region without any changes), and provides simpler application restarts and testing for DR.

10. Predictive autoscaling 

Customers use predictive autoscaling to improve response times for applications with long initialization times or for applications with workloads that vary predictably with daily or weekly cycles. When you enable predictive autoscaling, Compute Engine forecasts future load based on your Managed Instance Group’s history and scales out the MIG’s in advance of predicted load, so that new instances are ready to serve when the load arrives. Without predictive autoscaling, an autoscaler can only scale a group reactively, based on observed changes in load in real time. 

With predictive autoscaling enabled, the autoscaler works with real-time data as well as with historical data to cover both the current and forecasted load. Forecasts are refreshed every few minutes (faster than competing clouds) and consider daily and weekly seasonality, leading to more accurate forecasts of load patterns.

For more information, see How predictive autoscaling works and Checking if predictive autoscaling is suitable for your workload.

These are just a few examples of customer-centric innovation that set Google Cloud infrastructure apart.  Bring your applications and let the platform work for you.   

Get started by learning about your options for migration, or talk to our sales team to join the thousands of customers who have embarked upon this journey.

Acknowledgement

Special thanks to Dheeraj Konidena (Google) for contributing to this article.

When customers migrate to Google Cloud Platform (GCP), their first step is often to adopt Compute Engine, which makes it easy to procure and set up virtual machines (VMs) in the cloud that provide large amounts of computing power. Launched in 2012, Compute Engine offers multiple machine types, many innovative features, and is available in 20 regions and 61 zones! 

Compute Engine’s predefined and custom machine types make it easy to choose VMs closest to your on-premises infrastructure, accelerating the workload migration process cost effectively. Cloud allows you the pricing advantage of ‘pay as you go’ and also provides significant savings as you use more compute with Sustained Use Discounts. 

As Technical Account Managers, we work with large enterprise customers to analyze their monthly spend and recommend optimization opportunities. In this blog, we will share the top recommendations that we’ve developed based on our collective experience working with GCP customers. 

Getting ready to save

Before you get started, be sure to familiarize yourself with the VM instance pricing page—required reading for anyone who needs to understand the Compute Engine billing model and resource-based pricing. In addition to those topics, you’ll also find information about the various Compute Engine machine types, committed use discounts and how to view your usage, among other things. 

Another important step to gain visibility into your Compute Engine cost is using Billing reports in the Google Cloud Console and customizing your views based on filtering and grouping by projects, labels and more. From there you can export Compute Engine usage details to BigQuery for more granular analysis. This allows you to query the datastore to understand your project’s vCPU usage trends and how many vCPUs can be reclaimed. If you have defined thresholds for the number of cores per project, usage trends can help you spot anomalies and take proactive actions. These actions could be rightsizing the VMs or reclaiming idle VMs.

Now, with these things under your belt, let’s go over the five ways you can optimize your Compute Engine resources that we believe will give you the most immediate benefit. 

1. Apply Compute Engine rightsizing recommendations

Compute Engine’s rightsizing recommendations feature provides machine type recommendations that are generated automatically based on system metrics gathered by Stackdriver Monitoring over the past eight days. Use these recommendations to resize your instance’s machine type to more efficiently use the instance’s resources. It also recommends custom machine types when appropropriate. Compute Engine makes viewing, resizing and other actions easier right from the Cloud Console as shown below. 

Recently, we expanded Compute Engine rightsizing capabilities from just individual instances to managed instance groups as well. Check out the documentation for more details.

For more precise recommendations, you can install the Stackdriver Monitoring agent which collects additional disk, CPU, network, and process metrics from your VM instances to better estimate your resource requirements. You can also leverage the Recommender API for managing recommendations at scale.

2. Purchase Commitments

Our customers have diverse workloads running on Google Cloud with differing availability requirements. Many customers follow a 70/30 rule when it comes to managing their VM fleet—they have constant year-round usage of ~70%, and a seasonal burst of ~30% during holidays or special events. 

If this sounds like you, you are probably provisioning resources for peak capacity. However, after migrating to Google Cloud, you can baseline your usage and take advantage of deeper discounts for Compute workloads. Committed Use Discounts are ideal if you have a predictable steady-state workload as you can purchase a one or three year commitment in exchange for a substantial discount on your VM usage.

We recently released a Committed Use Discount analysis report in the Cloud Console that helps you understand and analyze the effectiveness of the commitments you’ve purchased. In addition to this, large enterprise customers can work with their Technical Account Managers who can help manage their commitment purchases and work proactively with them to increase Committed Use Discount coverage and utilization to maximize their savings.

3. Automate cost optimizations

The best way to make sure that your team is always following cost-optimization best practices is to automate them, reducing manual intervention.

Automation is greatly simplified using a label—a key-value pair applied to various Google Cloud services. For example, you could label instances that only developers use during business hours with “env: development.” You could then use Cloud Scheduler to schedule a serverless Cloud Function to shut them down over the weekend or after business hours and then restart them when needed. Here is an architecture diagram and code samples that you can use to do this yourself. 

Using Cloud Functions to automate the cleanup of other Compute Engine resources can also save you a lot of time and money. For example, customers often forget about unattached (orphaned) persistent disk, or unused IP addresses. These accrue costs, even if they are not attached to a virtual machine instance. VMs with the “deletion rule” option set to “keep disk” retain persistent disks even after the VM is deleted. That’s great if you need to save the data on that disk for a later time, but those orphaned persistent disks can add up quickly and are often forgotten! There is a Google Cloud Solutions article that describes the architecture and sample code for using Cloud Functions, Cloud Scheduler, and Stackdriver to automatically look for these orphaned disks, take a snapshot of them, and remove them. This solution can be used as a blueprint for other cost automations such as cleaning up unused IP addresses, or stopping idle VMs. 

4. Use preemptible VMs

If you have workloads that are fault tolerant, like HPC, big data, media transcoding, CI/CD pipelines or stateless web applications, using preemptible VMs to batch-process them can provide massive cost savings. In fact, customer Descartes Labs reduced their analysis costs by more than 70% by using preemptible VMs to process satellite imagery and help businesses and governments predict global food supplies.

Preemptible VMs are short lived— they can only run a maximum of 24 hours, and they may be shut down before the 24 hour mark as well. A 30-second preemption notice is sent to the instance when a VM needs to be reclaimed, and you can use a shutdown script to clean up in that 30-second period. Be sure to fully review the full list of stipulations when considering preemptible VMs for your workload. All machine types are available as preemptible VMs, and you can launch one simply by adding “-preemptible” to the gcloud command line or selecting the option from the Cloud Console. 

Using preemptible VMs in your architecture is a great way to scale compute at a discounted rate, but you need to be sure that the workload can handle the potential interruptions if the VM needs to be reclaimed. One way to handle this is to ensure your application is checkpointing as it processes data, i.e., that it’s writing to storage outside the VM itself, like Google Cloud Storage or a database. As an example, we have sample code for using a shutdown script to write a checkpoint file into a Cloud Storage bucket. For web applications behind a load balancer, consider using the 30-second preemption notice to drain connections to that VM so the traffic can be shifted to another VM. Some customers also choose to automate the shutdown of preemptible VMs on a rolling basis before the 24-hour period is over, to avoid having multiple VMs shut down at the same time if they were launched together. 

5. Try autoscaling 

Another great way to save on costs is to run only as much capacity as you need, when you need it. As we mentioned earlier, typically around 70% of capacity is needed for steady-state usage, but when you need extra capacity, it’s critical to have it available. In an on-prem environment, you need to purchase that extra capacity ahead of time. In the cloud, you can leverage autoscaling to automatically flex to increased capacity only when you need it. 

Compute Engine managed instance groups are what give you this autoscaling capability in Google Cloud. You can scale up gracefully to handle an increase in traffic, and then automatically scale down again when the need for instances is lowered (downscaling). You can scale based on CPU utilization, HTTP load balancing capacity, or Stackdriver Monitoring metrics. This gives you the flexibility to scale based on what matters most to your application. 

High costs do not compute

As we’ve shown above, there are many ways to optimize your Compute Engine costs. Monitoring your environment and understanding your usage patterns is key to understanding the best options to start with, taking the time to model your baseline costs up front. Then, there are a wide variety of strategies to implement depending on your workload and current operating model. 

For more on cost management, check out our cost management video playlist. And for more tips and tricks on saving money on other GCP services, check out our blog posts on Cloud Storage, Networking and BigQuery cost optimization strategies. We have additional blog posts coming soon, so stay tuned!

Editor’s note: Today we are hearing from Jono MacDougall , Principal Software Engineer at Ravelin. Ravelin delivers market-leading online fraud detection and payment acceptance solutions for online retailers. To help us meet the scaling, throughput, and latency demands of our growing roster of large-scale clients, we migrated to Google Cloud and its suite of managed services, including Cloud Bigtable, the scalable NoSQL database for large workloads.

As a fraud detection company for online retailers, each new client brings new data that must be kept in a secure manner and new financial transactions to analyze. This means our data infrastructure must be highly scalable and constantly maintain low latency. Our goal is to bring these new organizations on quickly without interrupting their business. We help our clients with checkout flows, so we need latencies that won’t interrupt that process—a critical concern in the booming online retail sector. 

We like Cloud Bigtable because it can quickly and securely ingest and process a high volume of data. Our software accesses data in Bigtable every time it makes a fraud decision. When a client’s customer places an order, we need to process their full history and as much data as possible about that customer in order to detect fraud, all while keeping their data secure. Bigtable excels at accessing and processing that data in a short time window. With a customer key, we can quickly access data, bring it into our feature extraction process, and generate features for our models and rules. The data stays encrypted at rest in Bigtable, which keeps us and our customers safe.

Bigtable also lets us present customer profiles in our dashboard to our client, so that if we make a fraud decision, our clients can confirm the fraud using the same data source we use.

We have configured our bigtable clusters to only be accessible within our private network and have restricted our pods access to it using targeted service accounts. This way the majority of our code does not have access to bigtable and only the bits that do the reading and writing have those privileges.

We also use Bigtable for debugging, logging, and tracing, because we have spare capacity and it’s a fast, convenient location. 

We conduct load testings against Bigtable.  We started at a low rate of ~10 Bigtable requests per second and we peaked at ~167000 mixed read and write requests per second  at absolute peak. The only intervention that was done to achieve this was pressing a single button to increase the number of nodes in the database. No other changes were made.

In terms of real traffic to our production system, we have seen ~22,000 req/s (combined read/write) on Bigtable in our live environment as a peak within the last 6 weeks.

Migrating seamlessly to Google Cloud 

Like many startups, we started with Postgres, since it was easy and it was what we knew, but we quickly realized that scaling would be a challenge, and we didn’t want to manage enormous Postgres instances. We looked for a kind of key value store, because we weren’t doing crazy JOINS or complex WHERE clauses. We wanted to provide a customer ID and get everything we knew about it, and that’s where key value really shines.  

I used Cassandra at a previous company, but we had to hire several people just for that chore. At Ravelin we wanted to move to managed services and save ourselves that headache. We were already heavy users and fans of BigQuery, Google Cloud’s serverless, scalable data warehouse, and we also wanted to start using Kubernetes. This was five years ago, and though quite a few providers offer Kubernetes services now, we still see Google Cloud at the top of that stack with Google Kubernetes Engine (GKE). We also like Bigtable’s versioning capability that helped with a use case involving upserts. All of these features helped us choose Bigtable.

Migrations can be intimidating, especially in retail where downtime isn’t an option. We were migrating not just from Postgres to Bigtable, but also from AWS to Google Cloud. To prepare, we ran in AWS like always, but at the same time we set up a queue at our API level to mirror every request over to Google Cloud. We looked at those requests to see if any were failing, and confirmed if the results and response times were the same as in AWS. We did that for a month, fine tuning along the way. 

Then we took the big step and flipped a config flag and it was 100% over to Google Cloud. At the exact same time, we flipped the queue over to AWS so that we could still send traffic into our legacy environment. That way, if anything went wrong, we could fail back without missing data. We ran like that for about a month, and we never had to fail back. In the end, we pulled off a seamless, issue-free online migration to Google Cloud.

Flexing Bigtable’s features

For our database structure, we originally had everything spread across rows, and we’d use a hash of a customer ID as a prefix. Then we could scan each record of history, such as orders or transactions. But eventually we got customers that were too big, where the scanning wasn’t fast enough. So we switched and put all of the customer data into one row and the history into columns. Then each cell was a different record, order, payment method, or transaction. Now, we can quickly look up the one row and get all the necessary details of that customer. Some of our clients send us test customers who place an order, say, every minute, and that quickly becomes problematic if you want to pull out enormous amounts of data without any limits on your row size. The garbage collection feature makes it easy to clean up big customers.  

We also use Bigtable replication to increase reliability, atomicity, and consistency. We need strong consistency guarantees within the context of a single request to our API since we make multiple bigtable requests within that scope. So within a request we always hit the same replica of Bigtable and if we have a failure, we retry the whole request. That allows us to make use of the replica and some of the consistency guarantees, a nice little trade-off where we can choose where we want our consistency to live.https://www.youtube.com/embed/0-eH5u7rrQQ?enablejsapi=1&

We also use BigQuery with Bigtable for training on customer records or queries with complicated WHERE clauses. We put the data in Bigtable, and also asynchronously in BigQuery using streaming inserts, which allows our data scientists to query it in every way you can imagine, build models, and investigate patterns and not worry about query engine limitations. Since our Bigtable production cluster is completely separate, doing a query on BigQuery has no impact on our response times. When we were on Postgres many years ago, it was used for both analysis and real time traffic and it was not the optimal solution for us. We also use Elasticsearch for powering text searches for our dashboard.

If you’re using Bigtable, we recommend three features:

• Key visualizer. If we get latency or errors coming back from Bigtable, we look at the key visualizer first. We may have a hotkey or a wide row, and the visualizer will alert us and provide the exact key range where the key lives, or the row in question. Then we can go in and fix it at that level. It’s useful to know how your data is hitting Bigtable and if you’re using any anti-patterns or if your clients have changed their traffic pattern that exacerbated some issue.
• Garbage collection. We can prevent big row issues by putting size limits in place with the garbage collection policies.  
• Cell versioning. Bigtable has a 3d array, with rows, columns, and cells, which are all the different versions. You can make use of the versioning to get history of a particular value or to build a time series within one row. Getting a single row is very fast in Bigtable so as long as you can keep the data volume in check for that row, making use of cell versions is a very powerful and fast option. There are patterns in the docs that are quite useful and not immediately obvious. For example, one trick is to reverse your timestamps (MAXINT64 – now) so instead of the latest version, you can get the oldest version effectively reversing the cell version sorting if you need it.

Google Cloud and Bigtable help us meet the low-latency demands of the growing online retail sector, with speed and easy integration with other Google Cloud services like BigQuery. With their managed services, we freed up time to focus on innovations and meet the needs of bigger and bigger customers. 

Learn more about Ravelin and Bigtable, and check out our recent blog, How BIG is Cloud Bigtable?

Backup and DR data is a valuable business asset, and ensuring that it’s safe and accessible is essential. In particular, you want to be able to monitor your backups to ensure that the data is indeed protected and that you can quickly recover it in the event of a disaster, user error, or failed upgrade.

Users of Google Cloud’s Backup and DR service often ask:

1. What are the best practices for monitoring my backup and recovery jobs?
2. How can I proactively identify issues and troubleshoot? 
3. How can I create alerts and be notified about important events? 

The good news is that Google Cloud Backup and DR now integrates with Cloud Logging and Cloud Monitoring tools. Now, you can monitor backup events, jobs, appliance health, resource consumption and user actions in the same way you monitor other Google Cloud workloads and services. 

With this integration, you can now:

• Monitor events related to backup and restore – With detailed Google Cloud Backup and DR service event logs in Cloud Logging, you can now use log based alerts to create a customisable notification, in near-real time. This lets you monitor important events such as  backup job failures, jobs that did not run, local backup storage saturation, and network failures.
• Configure fine grained alerting – You can write custom event queries on a wide variety of dimensions such as event severity, application type, application name, job type, job name, error message and many more. This gives a lot of flexibility to define alerts for specific events or conditions within a system, thus reducing noise and helping you identify and fix issues easily. 
• Get notified on your preferred notification channel – Google Cloud offers seven pre-built notification channels that let you receive customisable notifications directly over email, SMS, Slack or the Google Cloud mobile app. Alternatively, you can integrate with your own monitoring and event management tools using webhooks or Pub/Sub. 
• Derive useful insights to troubleshoot issues – With Log Analytics, you can use BigQuery to query your data using SQL queries and generate operational insights, which can help you reduce time spent troubleshooting. For example, you can analyze the key reasons that backup and restore jobs fail for a given application or application type. 

The integration also lets you proactively discover deeper issues related to backup and recovery, such as capacity-related issues or certain job failures that happen periodically, by monitoring recurring events in the logs over time. With log-based metrics, you can: 

• Create trend charts to track important metrics such as the number of backup or restore jobs that failed during a day/week/month 
• Receive a notification when the number of occurrences crosses a threshold, for example receive a notification if a snapshot pool saturates more than five times a week 
• Monitor trends in data, such as latency values in logs, and receive a notification if the values change in an unacceptable way.

Getting started

The time to learn that your backup failed should never be when you go to restore. By integrating our Backup and DR service with Cloud Monitoring and Cloud Logging, you can get valuable assurances about the health of your backup and business continuity processes with the same tools that you use to manage other Google Cloud workloads. To get started, check out the Backup and DR event logs page. You can also watch a video that shows you how to set up and use the service and configure some custom alerts.

Businesses that operate in complex cloud environments, large fleets, or sophisticated security operations all require visibility into their cloud assets in order to keep their teams nimble and their data secure. Cloud Asset Inventory (CAI) helps these teams understand their Google Cloud  and Anthos environments by providing complete visibility, real-time monitoring, and powerful asset analysis capabilities. Today, Cloud Asset Inventory gets four new capabilities that help you understand your environment more clearly and easily than ever before.

New user interface eases asset and insight discovery 

Cloud Asset Inventory console preview is now publicly available for GCP and Anthos customers. This preview provides insights into your cloud footprint, history and details of resource usage with powerful filtering and search capabilities. For example, you can view your global distribution of resources and policies, how your GCE VM footprint has been changing over time, as well as full metadata and change history for all your assets. The CAI console can be filtered at the organization, folder, or project-level, so each user can view the resources they have permissions for down to project level granularity.

Asset discovery and Datadog integration

A new asset list service in CAI provides quick and comprehensive asset discovery, including asset history, without needing to export the data to a storage destination. Datadog, a leading multi-cloud monitoring and security service provider, relies on deep integration with CAI for service and asset discovery. Datadog has been piloting and taking full advantage of the newly released asset list service. Datadog Product Manager, Steve Harrington, commented: 

“Google’s new Cloud Asset Inventory API provides us with an immensely valuable, single source of truth for determining the resources present in a given GCP environment. Along with its rich metadata, this enables us to enhance multiple aspects of our integration with GCP, including streamlined metric collection and ingestion of custom labels. We plan to continue building around Cloud Asset Inventory in the future to improve existing features, and are envisioning ways it could help us provide entirely new insights to our customers.”

Answer “who can access what resources?”

Determining authoritative answers to security-related questions like “Who can read data from my storage bucket that contains PII?” or “Does a terminated employee still have any remaining access to my system?” can be difficult and time consuming. This is why access management and identity certification is one of the top security priorities for enterprises running workloads in the cloud. To help alleviate this challenge, the new Policy Analyzer capability in CAI thoroughly analyzes the relationship between IAM policies and resources. The analysis includes powerful and efficient group expansion, service account impersonation, conditional access analysis, resource expansion, and more. You can even export the results to a BigQuery table or Cloud Storage bucket for further analysis and record keeping. CAI’s enhanced UI makes it even easier for you to build your own flexible queries and quickly get to a comprehensive answer.

Create asset posture visibility 

Cloud Asset Inventory now provides seven types of Asset Insights through the Active Assist platform. These new asset insights help proactively detect anomalies within your organization’s IAM policies, which may be opportunities to improve your security posture. The insights can be aggregated at the Organization, Folder or Project level. 

The seven new Asset Insights include: 

• External members in IAM policies. 
• External users that impersonate your service accounts.
• External members as policy editors.
• External users who can view cloud storage buckets.
• Terminated users/groups that are still in IAM policies
• IAM policies containing all users or all authenticated users.
• Projects with only terminated users as owners.

As a Google Cloud customer you can get started and use all the recently released capabilities and features immediately; check out our documentation to see how. We’d love to hear your feedback; email us with any questions or concerns!