As part of our ongoing series on cost management for observability data in Google Cloud, we’re going to share four steps for getting the most out of your logs while on a budget. While we’ll focus on optimizing your costs within Google Cloud, we’ve found that this works with customers with infrastructure and logs on prem and in other clouds as well.

Step 1: Analyze your current spending on logging tools

To get started, create an itemized list of what volume of data is going where and what it costs. We’ll start with the billing report and the obvious line items including those under Operations Tools/Cloud Logging:

• Log Volume – the cost to write log data to disk once (see our previous blog post for an explanation)
• Log Storage Volume – the cost to retain logs for more than 30 days 

If you’re using tools outside Cloud Logging, you’ll also need to include any costs related to these solutions. Here’s a list to get you started:

• Log vendor and hardware costs — what are you paying to observability vendors? If you’re running your own logging solution, you’ll want to include the cost of compute and disk.
• If you export logs within Google Cloud, include Cloud Storage and BigQuery costs
• Processing costs — consider the costs for Kafka, Pub/Sub or Dataflow to process logs. Network egress charges may apply if you’re moving logs outside Google Cloud.
• Engineering resources dedicated to managing your logging tools across your enterprise often are significant too!

Step 2: Eliminate waste — don’t pay for logs you don’t need

While not all costs scale directly with volume, optimizing your log volume is often the best way to reduce spend. Even if you are using a vendor with a contract that locks you into a fixed price for a period of time, you may still have costs in your pipeline that can be reduced by avoiding wasteful logs such as Kafka, Pub/Sub or Dataflow costs. 

Finding chatty logs in Google Cloud

The easiest way to understand which sources are generating the highest volume of logs within Google Cloud is to start with our pre-built dashboards in Cloud Monitoring. To access the available dashboards:

1. Go to Monitoring -> Dashboards
2. Select “Sample Library” -> “Logging”

This blog post has some specific recommendations for optimizing logs for GKE and GCE using prebuilt dashboards.

As a second option, you can use Metrics Explorer and system metrics to analyze the volume of logs. For example, type “log bytes ingested” into the filter. This specific metric corresponds to the Cloud Logging “Log Volume” charge. There are many ways to filter this data. To get a big picture, we often start with grouping by both “resource_type” and “project_id”. 

To narrow down the resource type in a particular project, add a “project_id” filter. Select “sum” under the Advanced Options -> Click on Aligner and select “sum”. Sort by volume to see the resources with the highest log volume.

While these rich metrics are great for understanding volumes, you’ll probably want to eventually look at the logs to see whether they’re critical to your observability strategy. In Logs Explorer, the log fields on the left side help you understand volumes and filter logs from a resource type.

Reducing log volume with the Logs Router 

Now that we understand what types of logs are expensive, we can use the Log Router and our sink definitions to reduce these volumes. Your strategy will depend on your observability goals, but here are some general tools we’ve found to work well.

The most obvious way to reduce your log volume is not to send the same logs to multiple storage destinations. One common example of this is when a central security team uses an aggregated log sink to centralize their audit logs but individual projects still ingest these logs. Instead, use exclusion filters on the _Default log sink and any other log sinks in each project to avoid these logs. Exclusion filters also work on log sinks to BigQuery, Pub/Sub, or Cloud Storage.

Similarly, if you’re paying to store logs in an external log management tool, you don’t have to save these same logs to Cloud Logging. We recommend keeping a small set of system logs from GCP services such as GKE in Cloud Logging in case you need assistance from GCP support but what you store is up to you, and you can still export them to the destination of your choice!

Another powerful tool to reduce log volume is to sample a percentage of chatty logs. This can be particularly useful with 2XX log balancer logs, for example. This can be a powerful tool, but we recommend you design a sampling strategy based on your usage, security and compliance requirements and document it clearly.

Step 3: Optimize costs over the lifecycle of your logs

Another option to reduce costs is to avoid storing logs for more time than you need them. Cloud Logging charges based on the monthly log volume retained per month. There’s no need to switch between hot and cold storage in Cloud Logging; doubling the default amount of retention only increases the cost by 2%. You can change your custom log retention at any time.

If you are storing your logs outside of Cloud Logging, it is a good idea to compare the cost to retain logs and make a decision. 

Step 4: Setup alerts to avoid surprise bills

Once you are confident that the volume of logs being routed through log sinks fit in your budget, set up alerts so that you can detect any spikes before you get a large bill. To alert based on the volume of logs ingested into Cloud Logging:

1. Go to the Logs-based metrics page. Scroll down to the bottom of the page and click the three dots on “billing/bytes_ingested” under System-defined metrics. 
2. Click “ Create alert from metric”
3. Add filters (For example: use resource_id or project_id. This is optional). 
4. Select the logs based metric for the alert policy.

You can also set up similar alerts on the volume for log sinks to Pub/Sub, BigQuery or Cloud Storage.

Conclusion

One final way to stretch your observability budget is to use more Cloud Operations. We’re always working to bring our customers the most value possible for their budget such as our latest feature, Log Analytics, which adds querying capabilities but also makes the same data available for analytics, reducing the need for data silos. Many small customers can operate entirely on our free tier. Larger customers have expressed their appreciation for the scalable Log Router functionality available at no extra charge that would otherwise require an expensive event store to process data. So it’s no surprise that a 2022 IDC report showed that more than half of respondents surveyed stated that managing and monitoring tools from public cloud platforms provide more value compared to third-party tools. Get started with Cloud Logging and Monitoring today.

It has been seven years since Marc Andreesen’s famous article “Why Software is Eating the World” was published in the Wall Street Journal, and given the slow pace of change in many companies, some executives still may not be taking the threat of being “eaten” seriously enough. 

Software, and platform business models based on software, have the potential to deliver powerful economic forces into virtually any company or industry. Every company has valuable assets—such as data, expertise or access to certain services or user bases—and most of these assets can be delivered via software. Once an asset is expressed as software in a modern way—that is, as an application programming interface (API)—it can be combined with other software to create new applications and digital experiences.

Benefits of this approach, just to name a few, include near-zero marginal cost to scale up APIs for new users or use cases; global reach for both partners using APIs and end users consuming the digital experiences those APIs power; and network effects triggered as more partners use a given company’s digital assets and spread its services into new markets and use cases.

Disruption by software-powered business models

In the last two decades we’ve seen individual companies and entire industries upended by these kinds of software-powered business models. Examples abound: Amazon and the retail industry, Netflix and movie rentals, Uber and ride hailing, Airbnb and hotels, etc. We’ve reached the point that these companies’ names have become verbs synonymous with being “eaten” by software (e.g., “Amazoned” or “Netflixed”).

The most famous examples of digital disruption involve digital natives, of course, but legacy businesses are leveraging software to evolve too. Brazilian retailer Magazine Luiza—a company I’ve worked with through my employer, Google Cloud’s Apigee team—has enjoyed enormous revenue growth and seen its stock soar, for example, as it has built out its digital platform capabilities and transitioned from a primarily brick-and-mortar model to an omnichannel one. The point is, whether a company has been in business five decades, five years or five months, software remains ravenous and is always looking for new companies and industries to “eat.”

Change in the face of serious threats

Facing this threat, how should corporate leadership respond? There are some excellent examples of CEOs who have galvanized their companies and led them through the massive, gut-wrenching change required to pivot in the face of a serious threat. A few of the biggest examples include: 

• In 1995, it became apparent to Microsoft co-founder and then-CEO Bill Gates that the internet was “the most important single development to come along since the IBM PC,” and, if not embraced in haste, a threat to many of Microsoft’s businesses. In May of that year he published the “The Internet Tidal Wave” memo and focused all of Microsoft on adopting and building for the internet. Almost 20 years later, current Microsoft CEO Satya Nadella similarly made the bold decision to redirect the company for a cloud-first world. 

• Facebook went public at $38 per share in May of 2012 but within months, stocks could be had for a little over half that. The concern? Facebook was a desktop-optimized website without a polished mobile presence, and by 2012, consumer attention had begun to accelerate towards mobile at a much higher rate than many initially predicted.  Facebook CEO Mark Zuckerberg reacted by not only proclaiming Facebook a mobile-first company, but also backing up that proclamation with action.  

• Turning to another company I’ve worked with via Apigee, T-Mobile launched its highly visible “Uncarrier” campaign—which offers streamlined, customer-friendly plans and services—while also investing in and executing a new IT vision dedicated to ongoing digital transformation. T-Mobile execs have credited the technology effort, spearheaded by CEO John Legere, with helping the company to introduce new services and better service customers. T-Mobile’s market cap has more than doubled since Legere took over in 2012.  

Keeping pace with changing customer needs

In the face of an existential threat, strong executive leadership is required to pivot the company to safety, as these examples attest. Digital transformationisn’t about deploying new technologies just to make an existing approach more efficient or to add a few new apps or features to the status quo; digital transformation is about keeping pace with changing customer needs by leveraging software platforms to continuously evolve how the business operates. This can be akin to turning an enormous ship—and a ship can’t turn very well without her captain, first mate, and other leaders showing the way.  

Research supports this. A recent Deloitte survey, for example, found that over “80 percent of respondents from digitally maturing organizations say their leaders have sufficient knowledge and ability to lead the company’s digital strategy,” compared to only “22 percent of early-stage business respondents [who] have the same belief.” Similarly, Gartner research finds that CEOs are seeking a “deeper understanding of digital business” as they shift their focus from growth in general to how technology helps them attain it.  

More recently, the onslaught of software devouring the world has been further accelerated by machine learning making everything smarter, voice interfaces changing how people interact with devices, and more. To keep pace, corporate leaders need to galvanize their companies to build and deploy software faster, make systems and data easily accessible inside and outside their companies, and improve digital experiences through not only machine learning but also constant data-driven iteration. 

Seven years after Andreesen’s editorial, the pace of digital disruption is still increasing, and so is the need for strong leadership to pivot fully into digital. Over half of the Fortune 500 has been acquired, merged or declared bankruptcy since 2000—and the companies that survive in coming years won’t be those whose leaders treat technology as an IT concern rather than a core part of the business.