Best Practices for Cost Optimization in the Cloud
5423
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
When customers migrate to Google Cloud Platform (GCP), their first step is often to adopt Compute Engine, which makes it easy to procure and set up virtual machines (VMs) in the cloud that provide large amounts of computing power. Launched in 2012, Compute Engine offers multiple machine types, many innovative features, and is available in 20 regions and 61 zones!
Compute Engine’s predefined and custom machine types make it easy to choose VMs closest to your on-premises infrastructure, accelerating the workload migration process cost effectively. Cloud allows you the pricing advantage of ‘pay as you go’ and also provides significant savings as you use more compute with Sustained Use Discounts.
As Technical Account Managers, we work with large enterprise customers to analyze their monthly spend and recommend optimization opportunities. In this blog, we will share the top recommendations that we’ve developed based on our collective experience working with GCP customers.
Getting ready to save
Before you get started, be sure to familiarize yourself with the VM instance pricing page—required reading for anyone who needs to understand the Compute Engine billing model and resource-based pricing. In addition to those topics, you’ll also find information about the various Compute Engine machine types, committed use discounts and how to view your usage, among other things.
Another important step to gain visibility into your Compute Engine cost is using Billing reports in the Google Cloud Console and customizing your views based on filtering and grouping by projects, labels and more. From there you can export Compute Engine usage details to BigQuery for more granular analysis. This allows you to query the datastore to understand your project’s vCPU usage trends and how many vCPUs can be reclaimed. If you have defined thresholds for the number of cores per project, usage trends can help you spot anomalies and take proactive actions. These actions could be rightsizing the VMs or reclaiming idle VMs.
Now, with these things under your belt, let’s go over the five ways you can optimize your Compute Engine resources that we believe will give you the most immediate benefit.
1. Apply Compute Engine rightsizing recommendations
Compute Engine’s rightsizing recommendations feature provides machine type recommendations that are generated automatically based on system metrics gathered by Stackdriver Monitoring over the past eight days. Use these recommendations to resize your instance’s machine type to more efficiently use the instance’s resources. It also recommends custom machine types when appropropriate. Compute Engine makes viewing, resizing and other actions easier right from the Cloud Console as shown below.
Recently, we expanded Compute Engine rightsizing capabilities from just individual instances to managed instance groups as well. Check out the documentation for more details.
For more precise recommendations, you can install the Stackdriver Monitoring agent which collects additional disk, CPU, network, and process metrics from your VM instances to better estimate your resource requirements. You can also leverage the Recommender API for managing recommendations at scale.
2. Purchase Commitments
Our customers have diverse workloads running on Google Cloud with differing availability requirements. Many customers follow a 70/30 rule when it comes to managing their VM fleet—they have constant year-round usage of ~70%, and a seasonal burst of ~30% during holidays or special events.
If this sounds like you, you are probably provisioning resources for peak capacity. However, after migrating to Google Cloud, you can baseline your usage and take advantage of deeper discounts for Compute workloads. Committed Use Discounts are ideal if you have a predictable steady-state workload as you can purchase a one or three year commitment in exchange for a substantial discount on your VM usage.
We recently released a Committed Use Discount analysis report in the Cloud Console that helps you understand and analyze the effectiveness of the commitments you’ve purchased. In addition to this, large enterprise customers can work with their Technical Account Managers who can help manage their commitment purchases and work proactively with them to increase Committed Use Discount coverage and utilization to maximize their savings.
3. Automate cost optimizations
The best way to make sure that your team is always following cost-optimization best practices is to automate them, reducing manual intervention.
Automation is greatly simplified using a label—a key-value pair applied to various Google Cloud services. For example, you could label instances that only developers use during business hours with “env: development.” You could then use Cloud Scheduler to schedule a serverless Cloud Function to shut them down over the weekend or after business hours and then restart them when needed. Here is an architecture diagram and code samples that you can use to do this yourself.
Using Cloud Functions to automate the cleanup of other Compute Engine resources can also save you a lot of time and money. For example, customers often forget about unattached (orphaned) persistent disk, or unused IP addresses. These accrue costs, even if they are not attached to a virtual machine instance. VMs with the “deletion rule” option set to “keep disk” retain persistent disks even after the VM is deleted. That’s great if you need to save the data on that disk for a later time, but those orphaned persistent disks can add up quickly and are often forgotten! There is a Google Cloud Solutions article that describes the architecture and sample code for using Cloud Functions, Cloud Scheduler, and Stackdriver to automatically look for these orphaned disks, take a snapshot of them, and remove them. This solution can be used as a blueprint for other cost automations such as cleaning up unused IP addresses, or stopping idle VMs.
4. Use preemptible VMs
If you have workloads that are fault tolerant, like HPC, big data, media transcoding, CI/CD pipelines or stateless web applications, using preemptible VMs to batch-process them can provide massive cost savings. In fact, customer Descartes Labs reduced their analysis costs by more than 70% by using preemptible VMs to process satellite imagery and help businesses and governments predict global food supplies.
Preemptible VMs are short lived— they can only run a maximum of 24 hours, and they may be shut down before the 24 hour mark as well. A 30-second preemption notice is sent to the instance when a VM needs to be reclaimed, and you can use a shutdown script to clean up in that 30-second period. Be sure to fully review the full list of stipulations when considering preemptible VMs for your workload. All machine types are available as preemptible VMs, and you can launch one simply by adding “-preemptible” to the gcloud command line or selecting the option from the Cloud Console.
Using preemptible VMs in your architecture is a great way to scale compute at a discounted rate, but you need to be sure that the workload can handle the potential interruptions if the VM needs to be reclaimed. One way to handle this is to ensure your application is checkpointing as it processes data, i.e., that it’s writing to storage outside the VM itself, like Google Cloud Storage or a database. As an example, we have sample code for using a shutdown script to write a checkpoint file into a Cloud Storage bucket. For web applications behind a load balancer, consider using the 30-second preemption notice to drain connections to that VM so the traffic can be shifted to another VM. Some customers also choose to automate the shutdown of preemptible VMs on a rolling basis before the 24-hour period is over, to avoid having multiple VMs shut down at the same time if they were launched together.
5. Try autoscaling
Another great way to save on costs is to run only as much capacity as you need, when you need it. As we mentioned earlier, typically around 70% of capacity is needed for steady-state usage, but when you need extra capacity, it’s critical to have it available. In an on-prem environment, you need to purchase that extra capacity ahead of time. In the cloud, you can leverage autoscaling to automatically flex to increased capacity only when you need it.
Compute Engine managed instance groups are what give you this autoscaling capability in Google Cloud. You can scale up gracefully to handle an increase in traffic, and then automatically scale down again when the need for instances is lowered (downscaling). You can scale based on CPU utilization, HTTP load balancing capacity, or Stackdriver Monitoring metrics. This gives you the flexibility to scale based on what matters most to your application.
High costs do not compute
As we’ve shown above, there are many ways to optimize your Compute Engine costs. Monitoring your environment and understanding your usage patterns is key to understanding the best options to start with, taking the time to model your baseline costs up front. Then, there are a wide variety of strategies to implement depending on your workload and current operating model.
For more on cost management, check out our cost management video playlist. And for more tips and tricks on saving money on other GCP services, check out our blog posts on Cloud Storage, Networking and BigQuery cost optimization strategies. We have additional blog posts coming soon, so stay tuned!
3359
Of your peers have already downloaded this article
10:00 Minutes
The most insightful time you'll spend today!
Operational resilience continues to be a key focus for financial services firms. Regulators from around the world are refocusing supervisory approaches on operational resilience to support the soundness of financial firms and the stability of the financial ecosystem. Our new white paper discusses the continuing importance of operational resilience to the financial services sector, and the role that a well-executed migration to Google Cloud can play in strengthening it.
Create and Protect Admin Accounts
4862
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Setting up your new cloud infrastructure is scary. Extra scary when you realize that someone (is it gonna be you?) gets to have phenomenal cosmic power over the whole thing.
Yes, I’m talking about the admin account, and today we’ll dig into why they are important, dangerous and different.
When the team at pistach.io got their nuts-as-a-service business growing fruitfully, they knew they needed to think carefully about admin accounts. These people would have tremendous control over their use of Google Cloud, and they could potentially cause very big problems if any were compromised. Definitely resources that require a protective shell.
Early in pistach.io’s development, an employee named Walter Nutt wanted to play a prank by changing his co-worker’s profile photo from an almond to a peanut (pretty devious, since a peanut is actually a legume). He didn’t have access to his friend’s computer, but he did have access to the company’s Cloud Storage bucket. While searching for the profile photo in question, Wally inadvertently deleted the entire contents of the bucket!
As he searched for ways to restore its contents, Wally modified access to two other pistach.io buckets. The company was ground to a halt for a week while teams worked to crack through the permissions issues.
Time to rethink permissions a bit, so this couldn’t spoil their buttery smooth operations in the future.
Following the resource manager guide, the team made a super admin email address that wasn’t tied to a particular individual or Workspace account, and secured it with strong multi-factor authentication. This would be their backup in case an admin account were to be compromised, so they could recover and repair.
The team already uses Google Workspace, so they have an organization set up already. That creation process established initial super administrators, allowing them to create and modify all other resources inside the organization. As they looked toward using Google Cloud, the super administrators could:
- Give the admin role to people, for Cloud
- Act as a point of contact for account recovery
- Modify or delete the organization if needed
Making admin users for the organization allows other people to then flesh out the resources and policies for pistach.io, before they go nuts and give everyone all the permissions. While that would speed things up, it would make it easy for an attacker to crack through the security shell because any account compromise could give ousize access. Yikes!
Instead, the IT leads specified certain people to act as organization admins, and then gave them permissions to:
- Define Identity and Access Management policies
- Structure the Resource Hierarchy
- Delegate control of specific Cloud elements to others on the team
Once those organization admins were set up, they could give management and oversight of Compute, Storage, Networking and other resource types to the relevant leads, making sure each person had just the right amount of permission for the role they needed to perform. The organization admins don’t have permissions themselves to make these resources. They just delegate.
Now each person can accomplish the job they’re responsible for, but doesn’t have overly permissive access. Delegating like this keeps the entire organization safer, and limits the blast radius if someone does manage to break in.
You can go through these steps yourself with this tutorial.
By default the creation of an organization resource for the domain gives everyone the ability to create projects and billing accounts. Once they set up their Organization Admin at pistach.io they decided to remove some of these wide permissions and, in a nutshell, bring everything down to a much finer control. So people could get permissions for a folder or a project, but not the entire organization!
Remember to take care of your admin roles, as they have the power, and responsibility, to cause serious harm if not used safely. Be safe with your Identity and Access Management. And keep your data yours!
Next time we join you we’ll take a crack at creating and provisioning an app to run inside the policies and resource management frameworks created today.
Turning the Tide: How PrestaShop Regained Trust in Data
4273
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Since 2007, PrestaShop has helped companies unlock the power of e-commerce through its open-source platform. Over 300,000 merchants worldwide use the PrestaShop platform to grow their business and serve online shoppers.
“Our open-source strategy to ecommerce enablement sets us apart,” says Rémi Paulin, Ph.D., Data Architect at PrestaShop. “Customization is becoming more crucial to retailers, and our open-source platform allows companies to continually evolve their sites and services to stand out from competitors.”
As PrestaShop grew, it wished to derive more value from its data, but the company ran into issues caused by a legacy, siloed architecture that negatively impacted data consistency and accessibility.
Let’s look at how PrestaShop works with Google Cloud and partners Fivetran and Hightouch to gain more control over data, enable a beyond-BI data strategy, and increase employee engagement from less than 10% to more than 40%.
Improving trust in data
Core systems at PrestaShop, including SQL and NoSQL databases, and SaaS Applications, were siloed; each presenting its own data, often captured from different sources such as support tickets, marketing engagement, purchase activity, and product usage. This setup made data overall inconsistent as no single system would contain a source of truth, resulting in many inefficiencies, poor collaboration across teams, and a reluctance to use data to support key decisions.
“Not long ago, less than 10% of the company regularly relied on data, so we were missing opportunities to make more data-driven decisions,” says Paulin. “Data was underutilized, and people were rapidly losing trust in data.”
PrestaShop set out to design a new architecture to address past challenges, such as lack of data consistency, and improve data accessibility.
“Google Cloud, along with Hightouch and Fivetran, allowed us to build a modern stack to solve these challenges and support our beyond-BI data strategy.”
Building a modern data stack
The first step was to build a robust data ingestion pipeline. After considering several vendors, PrestaShop chose to work with Fivetran to extract data from SaaS applications, including Zendesk, HubSpot, and GitHub, to load into BigQuery. They also use Datastream to stream Change Data Capture (CDC) data from transactional databases into BigQuery in real-time.
“Fivetran and Datastream are no-ops, efficient and highly reliable, and relieve our Data Engineers of management tasks. This brings us a high degree of confidence to build the rest of the stack atop these services,” says Paulin.
PrestaShop relies on several Google Cloud solutions, including Dataflow, and a managed Spark service by Ascend.io, for data transformation. It also uses Looker for its semantic modeling capacities and as a self-serve data platform.
As the company continued on its journey to transform how it manages and benefits from data, it engaged Hightouch to enable data accessibility through activation. Sitting on top of Looker, Hightouch unlocks all data models for operational intelligence. For example, in just a few days, the team built a customer knowledge model combining data from multiple sources and used Hightouch to sync data from the semantic layer to Zendesk via Reverse ETL. This allowed the care team to make more data-informed decisions, speeding up the time to resolve support tickets submitted through Zendesk by 33%.
“Hightouch feels like a natural extension of Looker and reinforces the position of the semantic data model as the single source of truth,” says Paulin. “It powers a variety of Data Activation use cases, supporting our beyond-BI strategy by providing teams with access to data when and where they need it to improve everyday operations. This has a big impact on the company, bolstering employee trust in available data.”
Becoming data-driven
In less than six months, PrestaShop managed to get the entire data stack up and running, build over 30 data models and engage over 120 employees with a small team of only two Data Engineers.
“Data is now accessible to every stakeholder within the company, regardless of their technical abilities,” says Paulin.
PrestaShop has already seen much progress in its shift to a more data-driven company and is excited to roll out more self-service intelligence capabilities in the future.
“Google Cloud drives home a culture of simplicity around our data stack, which is essential for us, especially given the small size of our engineering team,” says Paulin. “Fivetran and Hightouch share this culture of simplicity. Together, they offer strong foundations to support our data needs.”
Dashboards, which the company had always had an appetite for, are seamlessly created today. Before moving to Looker, a full-fledged dashboard would take an average of six weeks to develop. Now, it takes less than two days – and a simple dashboard can be created autonomously by business users in as little as 15 minutes.
Furthermore, data usage goes beyond dashboards. Thanks to Looker’s self-service exploration capabilities, many stakeholders can now glean insights surrounding product issues and business opportunities. Thanks to Hightouch, teams can activate their data to make better and smarter operational decisions.
“This is a big leap forward and one of many to come as we continue to add new models, activate our data, and onboard more users,” says Paulin. “Given our global reach and unique approach to e-commerce enablement, we know this is just the start of the great things we can accomplish with Google Cloud, Fivetran, and Hightouch.”
Check out Fivetran on Google Cloud Marketplace, or sign up for a free Hightouch workspace to learn more about what partners can do for your business.
SevenRooms: Building a profitable, sustainable future in food and beverage with Google Cloud
2942
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Finding ways to increase customer loyalty and profitability in a post-COVID world is top of mind for hotels, bars, and restaurants. Unfortunately, many food and beverage services providers struggle to deliver the personalized experiences that keep guests coming back for more. The reality is that most traditional hospitality apps offer only limited insight into guest activities, preferences, underlying operating costs, and other essential details.
We built SevenRooms to help food and beverage operators create truly memorable experiences by managing and personalizing every step of the guest journey. With SevenRooms, restaurants in more than 250 cities globally leverage technology and data to provide unforgettable hospitality experiences with a human touch. That includes seating guests at their favorite table, pouring complimentary glasses of wine from a preferred vintage, offering special menu options, and personalized experiences for special occasions.
Scaling guest experience and retention on Google Cloud
When developing SevenRooms, we needed a technology partner that would enable our small team to securely scale services, automate manual tasks, and accelerate time to market while reducing IT costs. That’s why we started working with Google App Engine—and later became more involved with the Google for Startups Cloud Program.
We soon realized many traditional apps lacked the integrations and capabilities needed to respond to the unique challenges facing food and beverage operators that the Google for Startups Cloud Program services provide. With guidance from experts on the Google Startups Success team, we quickly transformed SevenRooms from a beta restaurant and nightclub reservation and guest management point solution into a full-fledged guest experience and retention platform that analyzes actionable data to automatically build and continuously update detailed guest profiles.
The combination of Google App Engine and other Google Cloud solutions makes everything easier to build and scale. We’re seeing our total cost of ownership (TCO) decline by 10-15%, so we can shift additional resources to R&D to help customers create one-of-a-kind interactions with their guests. And importantly, we can bring new products to market 200-300% faster than competitors.
Because SevenRooms handles a lot of sensitive guest data, stringent security protocols are vital. We store all data on Google Cloud, taking advantage of its highly secure-by-design infrastructure and built-in support for international data privacy laws such as GDPR. We use BigQuery and Looker for data analysis and reporting, and power our NoSQL database with Firestore. We also scale workloads and run Elasticsearch on Google Compute Engine (GCE)—and seamlessly integrate our reservation booking engine with Google Maps and Google Search.
In the future, we’re looking to further market actionable guest data with the help of advanced machine learning (ML) models with TensorFlow and Google Cloud Tensor Processing Units (TPUs). Cloud TPUs and Google Cloud data and analytics services are fully integrated with other Google Cloud offerings, including Google Kubernetes Engine (GKE). By running ML workloads on Cloud TPUs, SevenRooms will benefit from Google Cloud’s leading networking and data analytics technologies such as BigQuery.
We’re also exploring additional Google Cloud solutions such as Anthos to unify the management of infrastructure and applications across on-premises, edge, and multiple public clouds, as well as Google Cloud Run to deploy scalable containerized applications on a fully managed serverless platform. These solutions will enable us to continue to quickly expand our services and offer customers a variety of new benefits.
Building a profitable, sustainable future in food and beverage
Our work with the Google Startups Success team has been instrumental to helping us get where we are today. Their responsiveness is incredible and stands out compared to services from other technology providers. Google Cloud gives us a highly secure infrastructure and next-level training to evolve our infrastructure using solutions such as BigQuery and Backup and Disaster Recovery.
We also work with Google Cloud partner, DoiT International, to further scale and optimize our operations. In particular, DoiT provided expert scripts to short-cut lengthy processes, while actively troubleshooting any issues or questions that came up. DoiT continues to share guidance in key areas for future products and features. They have provided expertise for architecture, infrastructure and cost management. Moving forward, we’re excited to work with Google Cloud and DoiT to handle a growing surge in users which we anticipate in 2023 and beyond.
With Google Cloud, SevenRooms is revitalizing food and beverage service delivery by enabling businesses to cultivate and maintain direct guest relationships, deliver exceptional experiences, and encourage repeat visits. We’ve compiled many case studies that demonstrate how our customers see great results by personalizing their interactions with guests, from $1.5M in cost savings, to $400K of additional revenue, to a 68% jump in email open rates.
Demand for our guest experience and retention platform keeps growing as we help our customers take a people-first approach by delivering unique and tailored dining experiences. We can’t wait to see what we accomplish next as we expand our team and reach new markets worldwide.
If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.
3150
Of your peers have already downloaded this article
10:00 Minutes
The most insightful time you'll spend today!
FinOps is a concept similar to DevOps, but with a different set of goals. Cloud FinOps is an operational framework and cultural shift that brings together technology, finance and business to drive financial accountability and accelerate business value realization. In layman terms, FinOps aims to help companies achieve most out of every dollar invested on cloud technology. It includes a broad set of existing and net-new processes or frameworks that breaks silos across functions, and build a better working model to achieve collaboration, agility, ownership and value.
Rest your anxieties about the changes in mindset and organizational behavior around current financial management practices while taking full flexibility benefits of cloud. Ease your cloud migration journey and value realization as experts at Google Cloud have shared best-practices, insights and action items to implement FinOps on Google Cloud in this whitepaper. Download now!
More Relevant Stories for Your Company
Greenovation: Adani and Google Cloud Join Forces for a Sustainable and Data-Driven India
As the biggest renewable energy developer, Adani aims to put India on the green map. Working with the world's cleanest cloud inspired the organisation to solve their own efficiency issues, and better manage their facilities across the country to lower their carbon footprint. For more on Adani, visit: https://www.adani.com/ For
CFO Watch: A Handy Guide to Financial Governance in the Cloud
With a growing number of enterprises across industries making the move from on-premise infrastructure to on-demand cloud services, there has been a major shift from CapEx to OpEx spending. As a result, budgeting can no longer be a one-time operational process completed annually. Instead, spending must be monitored and controlled
Google Extends Support for Windows Server Containers on Anthos for Faster App Modernization and Consistent Dev Experience
Today, many applications in organizations’ data centers run on Windows Server. Modernizing these traditional Windows apps onto Kubernetes promises a host of benefits: a consistent platform across environments, better portability, scalability, availability, simplified management and speed of deployment, just to name a few. But how? Rewriting traditional .NET applications to
How does Google Pick its Data Center ?
Google is well known for its sustainable tech and hardware initiatives. Did you know alongside its environmental friendly designs of its data centers, it takes into account various factors such as redundant power supplies, data replication, network connectivity, etc. Watch the video to learn more.
