As part of our ongoing series on cost management for observability data in Google Cloud, we’re going to share four steps for getting the most out of your logs while on a budget. While we’ll focus on optimizing your costs within Google Cloud, we’ve found that this works with customers with infrastructure and logs on prem and in other clouds as well.

Step 1: Analyze your current spending on logging tools

To get started, create an itemized list of what volume of data is going where and what it costs. We’ll start with the billing report and the obvious line items including those under Operations Tools/Cloud Logging:

• Log Volume – the cost to write log data to disk once (see our previous blog post for an explanation)
• Log Storage Volume – the cost to retain logs for more than 30 days 

If you’re using tools outside Cloud Logging, you’ll also need to include any costs related to these solutions. Here’s a list to get you started:

• Log vendor and hardware costs — what are you paying to observability vendors? If you’re running your own logging solution, you’ll want to include the cost of compute and disk.
• If you export logs within Google Cloud, include Cloud Storage and BigQuery costs
• Processing costs — consider the costs for Kafka, Pub/Sub or Dataflow to process logs. Network egress charges may apply if you’re moving logs outside Google Cloud.
• Engineering resources dedicated to managing your logging tools across your enterprise often are significant too!

Step 2: Eliminate waste — don’t pay for logs you don’t need

While not all costs scale directly with volume, optimizing your log volume is often the best way to reduce spend. Even if you are using a vendor with a contract that locks you into a fixed price for a period of time, you may still have costs in your pipeline that can be reduced by avoiding wasteful logs such as Kafka, Pub/Sub or Dataflow costs. 

Finding chatty logs in Google Cloud

The easiest way to understand which sources are generating the highest volume of logs within Google Cloud is to start with our pre-built dashboards in Cloud Monitoring. To access the available dashboards:

1. Go to Monitoring -> Dashboards
2. Select “Sample Library” -> “Logging”

This blog post has some specific recommendations for optimizing logs for GKE and GCE using prebuilt dashboards.

As a second option, you can use Metrics Explorer and system metrics to analyze the volume of logs. For example, type “log bytes ingested” into the filter. This specific metric corresponds to the Cloud Logging “Log Volume” charge. There are many ways to filter this data. To get a big picture, we often start with grouping by both “resource_type” and “project_id”. 

To narrow down the resource type in a particular project, add a “project_id” filter. Select “sum” under the Advanced Options -> Click on Aligner and select “sum”. Sort by volume to see the resources with the highest log volume.

While these rich metrics are great for understanding volumes, you’ll probably want to eventually look at the logs to see whether they’re critical to your observability strategy. In Logs Explorer, the log fields on the left side help you understand volumes and filter logs from a resource type.

Reducing log volume with the Logs Router 

Now that we understand what types of logs are expensive, we can use the Log Router and our sink definitions to reduce these volumes. Your strategy will depend on your observability goals, but here are some general tools we’ve found to work well.

The most obvious way to reduce your log volume is not to send the same logs to multiple storage destinations. One common example of this is when a central security team uses an aggregated log sink to centralize their audit logs but individual projects still ingest these logs. Instead, use exclusion filters on the _Default log sink and any other log sinks in each project to avoid these logs. Exclusion filters also work on log sinks to BigQuery, Pub/Sub, or Cloud Storage.

Similarly, if you’re paying to store logs in an external log management tool, you don’t have to save these same logs to Cloud Logging. We recommend keeping a small set of system logs from GCP services such as GKE in Cloud Logging in case you need assistance from GCP support but what you store is up to you, and you can still export them to the destination of your choice!

Another powerful tool to reduce log volume is to sample a percentage of chatty logs. This can be particularly useful with 2XX log balancer logs, for example. This can be a powerful tool, but we recommend you design a sampling strategy based on your usage, security and compliance requirements and document it clearly.

Step 3: Optimize costs over the lifecycle of your logs

Another option to reduce costs is to avoid storing logs for more time than you need them. Cloud Logging charges based on the monthly log volume retained per month. There’s no need to switch between hot and cold storage in Cloud Logging; doubling the default amount of retention only increases the cost by 2%. You can change your custom log retention at any time.

If you are storing your logs outside of Cloud Logging, it is a good idea to compare the cost to retain logs and make a decision. 

Step 4: Setup alerts to avoid surprise bills

Once you are confident that the volume of logs being routed through log sinks fit in your budget, set up alerts so that you can detect any spikes before you get a large bill. To alert based on the volume of logs ingested into Cloud Logging:

1. Go to the Logs-based metrics page. Scroll down to the bottom of the page and click the three dots on “billing/bytes_ingested” under System-defined metrics. 
2. Click “ Create alert from metric”
3. Add filters (For example: use resource_id or project_id. This is optional). 
4. Select the logs based metric for the alert policy.

You can also set up similar alerts on the volume for log sinks to Pub/Sub, BigQuery or Cloud Storage.

Conclusion

One final way to stretch your observability budget is to use more Cloud Operations. We’re always working to bring our customers the most value possible for their budget such as our latest feature, Log Analytics, which adds querying capabilities but also makes the same data available for analytics, reducing the need for data silos. Many small customers can operate entirely on our free tier. Larger customers have expressed their appreciation for the scalable Log Router functionality available at no extra charge that would otherwise require an expensive event store to process data. So it’s no surprise that a 2022 IDC report showed that more than half of respondents surveyed stated that managing and monitoring tools from public cloud platforms provide more value compared to third-party tools. Get started with Cloud Logging and Monitoring today.

Financial markets were among the first to adopt new technologies, and that has certainly been true of the derivatives markets, which were early adopters of electronic trading. Going forward, new capabilities will transform the way industry participants communicate, analyze, and trade.

I sat down with Google Cloud’s Phil Moyer and former SEC Commissioner, Troy Paredes, for a fireside chat at FIA Boca 2022 to discuss the future of markets and policy, the new technologies that are already paving the way for greater speed and transparency, and how cloud can help promote greater resiliency, performance, and security to enable the long-term vision for the market. The following is a summary of our discussion.

The current state of cloud technology

When it comes to technology adoption, we’re seeing the market and participants adopt cloud technologies, and increasingly, machine learning (ML) on a wider scale. Cloud technology allows for easier, faster, and much more secure experimentation with large datasets and ML.

A recent Google sponsored study by Coalition Greenwich (September, 2021) showed that more than 93% of trading systems, exchanges, and data providers are in some way providing services on the cloud. The same study, revealed that about 72% of the financial industry across the buy side and sell side, intend to consume public cloud-data based market data within the next 12 months.

Data-driven decision-making and risk management have always been, and continue to remain, the cornerstones of the financial markets. Over time, technology innovation has facilitated access to better insights from data, and therefore, better decision-making and the ability to manage risk. That expectation is now mainstream, and will continue to grow in sophistication.

The multi-phased technology trajectory

The movement of exchanges to the cloud will occur in a “crawl-walk-run” fashion, with low-hanging fruits the first to be picked in the near term while bigger, paradigmatic changes will occur over the medium and long term. Some organizations are starting all three stages simultaneously, understanding that each will move at an independent cadence.

The “crawl” phase is one in which foundations are built, starting with organizations moving data to the cloud and experimenting with some degree of analytics. It’s one of the most important phases because it’s where the opportunity to increase transparency and risk management takes shape.

In moving to the cloud, the infrastructure – which in the past relied on a combination of people, processes, and some technology – becomes the code that runs applications. This early phase is key to empowering organizations to shift to a cloud-based, agile-first operating model that makes it easier and more seamless to launch new products in the future, including by freeing up people and resources from IT management to more mission-focused work.

Establishing the cloud operating model simplifies the “walk” and “run” phases where compliance is more automated, latency-sensitive applications are more readily available, and the next generation of exchanges, market participants, and regulators is better prepared to meet future challenges.

The “walk” phase is where much of the innovation happens. Exchanges are making significant progress in leveraging foundational data decisions in the “crawl” phase and innovations in the cloud to improve settlement, clearing, risk management, collateral management, and compliance, and launch new products.

And finally, the “run” phase is where organizations will start to move the latency-sensitive markets to the cloud, as the markets increasingly will demand low-latency and high performance along with transparency and analytics to solve historical obstacles to market access.

Opportunities for both regulators and market participants

Any time significant technological change takes place, regulators explore its implications, particularly with respect to their ability to meet their regulatory objectives.

Increasingly, we are seeing technological change driving more opportunities for regulators and market participants alike. Such changes may also allow better protection of the marketplace, with greater integrity and transparency.

Over time, regulatory regimes – rules, regulations, statutes, interpretations, and guidance – will also adjust to new technologies, both benefiting the marketplace and advancing regulatory goals.

As one example, the cloud is increasing the ability to meet compliance obligations by allowing compliance to be built into transactions. Moreover, predicated on the vision of real-time regulatory reporting, and given the pace of technological change in the marketplace over the last several years, various regulators have been using more advanced analytics. This trend will continue to help them more effectively and efficiently meet their objectives, and monitor and meet the expectations they have for the entire market.

Machine learning’s role in the financial markets

Google Cloud’s head of AI and Industry Solutions, Andrew Moore, said that ML will be doing three key things for us in the next 10 years: giving us meaning, providing concierge services, and serving as a guardian. Extracting information that is critical to investor decision-making can be extremely important. With more data than ever, ML can increase the ability to process it while also becoming more accessible in the cloud and better supporting regulatory objectives.

The technology will likely manifest in trading and anti-money laundering activities as they relate market functions, as well as managing a wide variety of risks – supporting the interests of both investors and regulators in terms of decision-making, surveillance, and protections.

Rather than taking individuals out of the equation, the digitization of markets, assets, and guard rails combined with ML will allow people to focus their expertise in different ways to achieve key objectives.

Building the market foundation for the future

The goals of operational resiliency, security, and privacy will continue to be critical for building the market foundation for both participants and regulators. While technology promises to create advantages in concrete, tangible ways, it will be important to scrutinize potential risks and concerns.

Priority one for technology providers is to build an environment of trustless security, including encryption at motion and encryption at rest, ensuring that markets are operationally resilient while instilling confidence for any exchange that runs on top of that infrastructure. Multicloud architectures and approaches are likely also to be part of the solution for operational resilience.

Throughout time, liquidity has been the outcome of improved access, transparency, and security. Technology providers are responding by sharing both the responsibility for, and fate of, the markets of the future to build an efficient, faster, and more transparent and secure financial industry.

You can learn more about our approach in our newest white paper, Building the financial markets foundation for the future.

Editor’s note: In this blog post we learn how TapClicks migrated to Google Cloud to offer their marketing customers a unified platform for data management, operations, insights, and analysis.

TapClicks is a smart marketing cloud, powered by data, that unifies our customer’s marketing. By choosing to migrate our core applications last year to Google Cloud, we cut costs, solved data-sharing concerns for our customers, and opened our stack up to a new ecosystem of possibilities. 

The core problem that we’re solving for our customers is how to manage their marketing infrastructures data and operations. Life isn’t easy for marketers now. There are 7,000 different vendors servicing this space today – creating much complexity between digital agencies, media, and brands. Marketers face challenges in navigating all of these systems, logging in and out, understanding pacing goals, and managing the flow of marketing data so they can analyze and report internally as well as to their clients at scale.

We unify omnichannel campaign data (250 API connectors and 6000 Smart Connectors ™ ) from a plethora of marketing sources on an automated data warehousing solution, creating simplicity for organizations. Over 4,000 agencies, media companies, and brands use our Marketing Operations and Data Management Platform, which imports data at scale and creates an automatic data warehouse on Google Cloud. Teams can also leverage TapClicks, like our world class Facebook connector, to import data directly into Google Data Studios.  Beyond importing and storing, we also provide data exporting to other Google solutions like Google Data Studio and Google Sheets.  We also create interactive dashboards that let stakeholders and clients analyze their data, as well as automated, multi-channel reports that go out to clients at specified times. So channel comparisons, optimizations, attribution, and calculations are easily performed.  Some of our customers are able to generate hundreds of thousands of individual reports and dashboards for their clients.

Although we may be best known for our reporting and analytics, we also empower teams managing the marketing operations workflow from customers and internal stakeholders, especially at scale. Our user-friendly, configurable system helps manage their orders and campaigns. Through automation of this process, we deliver tremendous amounts of efficiency, time saving, cost savings, and reduction of errors. The combination of these solutions makes up our unified platform, with additional capabilities like marketing intelligence that offers competitive and brand-level analysis. This is a disruptive solution in use by all leading media companies, agencies and many brands.

Partnering for possibilities

We faced a few challenges with our original tech stack, which included a mix of the leader in web services revenue, leaders in high performance data warehousing, as well as vendors on bare metal servers. 

• One challenge was around costs, which were growing. 
• Second, many of our customers work with multiple brands, and are very hesitant to share their data with the leader in web services, who’s often viewed as their competitor. 
• Third, these vendors are more focused on their own revenue rather than a true long term partnership that would enable their customers to enjoy similar success as they have experienced.

When looking at other cloud providers, Google Cloud emerged for us as the front runner. They were competitive on costs, and their native Kubernetes support was superior— a big selling point for our DevOps team. There’s also a movement in the marketing and advertising industry away from AWS toward Google Cloud because of the data-sharing concern. Finally, most of our customers are already using Google Cloud tools, so there’s brand recognition and familiarity there, and easier integrations with their own systems.

Migrating to Google Cloud

Our migration, which took about five months, involved moving a significant chunk of our infrastructure, including our core applications, using Google Kubernetes Engine (GKE). In our legacy architecture, each of our clients was assigned to one of our virtual machines (VMs), and there was a lot of unused capacity because we had to provision for the max usage. We appreciated GKE’s cloud native capabilities, especially autoscaling, a huge benefit for our web application. We have varying usage patterns during the day, and though our application is mostly used during business hours, there are also days in the month of higher usage, and autoscaling saves us time and costs. GKE also makes deployments much easier, and we anticipate a lot of benefits there for our developer environments. We’ve moved some of our microservices into GKE and plan to move more in the future. All in all, we were able to migrate our core products and the bulk of our AWS spend successfully to Google Cloud. 

We also moved from our other vendors Relational Database Service (RDS) to running MySQL on our own VMs on Google Cloud, which gives us more flexibility in terms of settings and fine tuning. We’re still trying to find the best mix as we’re modernizing our infrastructure, and we took this opportunity to migrate from MySQL 5.7 to 8.0.  

Our next stage is exploring more of the capabilities and services of Google Cloud, including BigQuery, which we’re considering for our own data warehouse. The fact that we could also run Snowflake on Google Cloud, if needed, was another selling point for our migration. 

We’re especially interested in BigQuery ML’s machine learning and natural language processing capabilities, which enabled better predictive insights. Our customers want insights from their campaigns— which are working, which are paying off, where should they invest next? Using our platform, they’re looking not only to generate reporting, but also identify opportunities to improve campaign performance. We plan to use AI and ML to improve those capabilities, so that our customers can seamlessly unlock insight and intelligence from their marketing data and campaigns.

Double-clicking on Google Cloud

For us, being able to deeply leverage and partner with Google Cloud to deliver those solutions on a single stack is critical, and we think our customers will love it. We see TapClicks and Google Cloud partnering at a level beyond what you typically see in a cloud provider relationship. Already, fifty percent of our company is working with various Google Cloud solutions, and we envision TapClicks and Google Cloud as extensions of each other, providing a single, powerful platform solution. 

Google Cloud understands the partnership concept, and their team was able to shine a light on their services and what they could bring to the table. Compared to our previous experiences, dealing with the Google Cloud team has been a true pleasure. Now that we’ve migrated, we’re ready to take our next steps into the services available to us in the Google Cloud ecosystem, and the problems we’ll continue to solve for our customers. Learn more about TapClicks and BigQuery ML.

To help improve your security posture, improve the reliability of your applications, and reduce configuration drift in your environment, we’re excited to announce a new Anthos reference architecture.

Written in collaboration across our product, engineering, support, and field teams, this new reference architecture helps you plan, deploy, and configure the required components for Anthos hybrid environments.

Anthos hybrid environments give you the flexibility to deploy on-premises components that run container-based workloads and VMs using Anthos clusters on VMware and Anthos clusters on bare metal. You can continue to utilize existing investments in your on-premises infrastructure, and start to add components like Anthos Config Management. When you’re ready to bring everything together, you can add additional Google Cloud-based services like Artifact Registry, Cloud Monitoring, and Identity and Access Management (IAM).

The following sneak peek covers some of our best practices for architecting an Anthos hybrid environment. For more detailed guidance and planning information, see the full Anthos hybrid environment reference architecture.

When you design and deploy an Anthos hybrid environment, we recommend that you use two or more on-premises computing customer sites and two or more Google Cloud regions. In your sites, run multiple clusters. This approach is recommended for several reasons, such as:

• Disaster recovery. If one cluster or site fails, you can continue to run workloads.
• Multiple environments, like production and staging, to test infrastructure changes.
• Different cluster types in each environment: admin clusters and user clusters. This approach separates administrative resources, which is a security best-practice.

The following diagram shows an example of an Anthos hybrid environment that’s spread across customer sites and regions, with different clusters for admin and user workloads and for production and staging:

In each site, you can use Anthos clusters on VMware or Anthos clusters on bare metal. For both products, we recommend the following:

• Use a highly available (HA) control plane with three members for continued control plane availability concurrently with operating system upgrades, control plane software updates, or single-machine hardware or kernel failures.
• Deploy two admin clusters so that admin cluster configuration changes and updates can be tested in the staging environment first.

The following diagram shows an example of Anthos clusters on bare metal with control plane and worker nodes spread across physical machines. With Anthos clusters on VMware, the control plane and worker nodes are spread across VMware VMs:

Configure your on-premises clusters and applications to send logging and monitoring data back to Google Cloud for analysis and review. Different personas should only be granted access to the environments they need. The following diagram shows how application developers and application or platform operators can then view logging and monitoring data in Google Cloud:

Use Anthos Config Management to manage Kubernetes objects in your clusters. Anthos Config Management is a GitOps-style tool that uses a Git repository or Open Container Initiative (OCI) as its storage mechanism and source of truth. Git provider workflows allow multiple stakeholders to participate in review of changes.

As shown in the following diagram, a common Anthos Config Management deployment uses one folder containing configuration for all clusters. Use separate additional folders to hold configuration data, one for application:

Plan and implement a way to secure the network traffic in your Anthos hybrid environment. The following services help with authentication, connectivity, and communication in a cluster:

• Anthos Identity Service connects clusters to on-site identity providers to authenticate local access.
• Connect gateway and workforce identity federation can provide secure cloud-mediated access to mobile workforce clusters without using a VPN.
• Workload Identity provides on-premises workloads with managed short-lifetime credentials for access to cloud resources.
• Anthos Service Mesh encrypts and controls communication between services in the same cluster.

The following diagram shows how Anthos Service Mesh can control the flow of traffic between services within your clusters:

You don’t have to implement all these cloud-based services as part of your initial on-premises deployments. As you become more comfortable and want to expand your capabilities, you can add in some of these hybrid offerings. But, we hope that this blog post has given you some ideas to think about when you start to plan and design your own Anthos hybrid environments.

For more detailed guidance and planning information, see the full Anthos hybrid environment reference architecture. Let us know what you think!

The software development process requires complex, cross-functional collaboration while continuously improving products and services. Our customers who build software say that they value Google Workspace for its ability to drive innovation and collaboration throughout the entire software development life cycle. Developers can hold standups and scrums in Google Chat, Meet, and Spaces, create and collaborate on requirements documentation in Google Docs and Sheets, build team presentations in Google Slides, and manage their focus time and availability with Google Calendar.

Development teams also use many other tools to get work done, like tracking issues and tasks in Atlassian’s Jira, managing workloads with Asana, and incident management in PagerDuty. One of the benefits of Google Workspace is that it’s an open platform tailored to improve the performance of your tools by seamlessly integrating them together. We’re constantly expanding our ecosystem and improving Google Workspace, giving you the power to push your software development even further.

Make software development more agile

Google Workspace gives you real-time visibility into project progress and decisions to help you ship quality code fast and stay connected with your stakeholders, all without switching tools and tabs. By leveraging applications from our partners, you can pull valuable information out of silos, making collaborating on requirements, code reviews, bug triage, deployment updates, and monitoring operations easy for the whole team. This allows your teams to stay focused on their priorities while keeping everyone aligned, ensuring collaborators are always in the loop.

Plan and execute together

When combined with integrations, Google Workspace makes the software development planning process more collaborative and efficient. For example, many organizations use Asana—a leading work management platform—to coordinate and manage everything from daily tasks to cross-functional strategic initiatives. To make the experience more seamless, Asana built integrations so users can always have access to their tasks and projects with Google Drive, Gmail, and Chat. With these integrations for Google Workspace, you can turn your conversations into action and create new tasks in Asana—all without leaving Google Workspace.

“We’ve seen exceptional, heavy adoption of tasks being created from within the Gmail add-on. Our customers and community have also shown very strong interest in future development work, which is something we’ll continue to prioritize.” Strand Sylvester, Product Manager, Asana

To date, users have installed the Asana for Gmail add-on over 2.5 million times, as well as over 3.8 million installs of the Asana for Google Workspace add-on for Google Drive.

Start coding quickly

Google Workspace makes it easy for product managers, UX designers, and engineers to agree on what they’re building and why. By bringing all stakeholders, decisions, and requirements into one place—whether it’s a Gmail or Google Chat conversation, or a document in Google Docs, Sheets, or Slides—Google Workspace removes friction, helping your teams finalize product specifications and get started right away.

Integrations like GitHub for Google Chat make the entire development process fit easily into a developer’s workflow. With this integration, teams can quickly push new commits, make pull requests, do code reviews, and provide real-time feedback that improves the quality of their code—all from Google Chat.

Speed up testing

Integrations like Jira for Google Chat accelerate the entire QA process in the development workflow. The app acts as a team member in the conversation, sending new issues and contextual updates as they are reported to improve the quality of your code and keep everyone informed on your Jira projects.

Ship code faster

Developers use Jenkins—a popular open-source continuous integration and continuous delivery tool—to build and test products continuously. Along with other cloud-native tools, Jenkins supports strong DevOps practices by letting you continuously integrate changes into the software build.

With Jenkins for Google Chat, development and operations teams can connect into their Jenkins pipeline and stay up to date by receiving software build notifications directly in Google Chat.

Proactively monitor your services

Improving the customer experience requires capturing and monitoring data sources to improve application and infrastructure observability. Google Workspace supports DevOps teams and organizations by helping stakeholders collaborate and troubleshoot more effectively. When you integrate Datadog with Google Chat, monitoring data becomes part of your team’s discussion, and you can efficiently collaborate to resolve issues as soon as they arise.

The integration makes it easy to start a discussion with all the relevant teams by sharing a snapshot of a graph in any of your Chat spaces. When an alert notification is triggered, it allows you to notify each Chat space independently, precisely targeting your communication to the right teams.

Improve service reliability

Orchestrating business-wide responses to interruptions is a cross-functional effort. When revenue and brand reputation depends on customer satisfaction, it’s important to proactively manage service-impacting events. Google Workspace supports response teams by ensuring that urgent alerts reach the right people by providing teams with a central space to discover incidents, find the root cause, and resolve them quickly.

PagerDuty for Google Chat empowers developers, DevOps, IT operations, and business leaders to prevent and resolve business-impacting incidents for an exceptional customer experience—all from Google Chat. See and share details with link previews, and perform actions by creating or updating incidents. By keeping all conversations in a central space, new responders can get up to speed and solve issues faster without interrupting others.

Accelerate developer productivity

Integrating your DevOps tools with Google Workspace allows your development teams to centralize their work, stay focused on what’s important—like managing their work—build code quickly, ship quality products, and communicate better during service impacting incidents. For more apps and solutions that help centralize your work so you and your teams can connect, create, and get things done, check out Google Workspace Marketplace, where you’ll find more than 5,300 public applications that integrate directly into Google Workspace.