Scaling with Breaking News: BBC’s Serverless Infrastructure on Google Cloud
1275
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Editors note: Today’s post is from Neil Craig at the British Broadcasting Corporation (BBC), the national broadcaster of the United Kingdom. Neil is part of the BBC’s Digital Distribution team which is responsible for building the services such as the public-facing www bbc.co.uk and .com websites and ensuring they are able to scale and operate reliably.
The BBC’s public-facing websites inform, educate, and entertain over 498 million adults per week across the world. Because breaking news is so unpredictable, we need a core content delivery platform that can easily scale in response to surges in traffic, which can be quite unpredictable.
To this end, we recently rebuilt our log-processing infrastructure on a Google Cloud serverless platform. We’ve found that the new system, based on Cloud Storage, Eventarc, Cloud Run and BigQuery, enables us to provide a reliable and stable service without us having to worry about scaling up during busy times. We’re also able to save license fee payers money by operating the service more cost effectively than our previous architecture. Not having to manually manage the scale of major components of the stack has freed up our time, allowing us to spend it on using, rather than creating the data.
A log in time
To operate the site and ensure our services run smoothly we continually monitor Traffic Manager and CDN access logs. Our websites generate more than 3B log lines per day, and handle large data bursts during major news events; on a busy day our system supports over 26B log lines in a single day.
As initially designed, we stored log data in a Cloud Storage bucket. But every time we needed to access that data, we had to download terabytes of logs down to a virtual machine (VM) with a large amount of attached storage, and use the ‘grep’ tool to search and analyze them. From beginning to end, this took us several hours. On heavy news days, the time lag made it difficult for the engineering team to do their jobs.
We needed a more efficient way to make this log data available, so we designed and deployed a new system that deals with logs and reacts to spikes more efficiently as they arrive, improving the timeliness of critical information significantly.
In this new system, we still leverage Cloud Storage buckets, but on arrival, each log generates an event using EventArc. That event triggers Cloud Run to validate, transform and enrich various pieces of information about the log file such as filename, prefix, and type, then processes it and outputs the processed data as a stream into BigQuery. This event-driven design allows us to process files quickly and frequently — processing a single log file typically takes less than a second. Most of the files that we feed into the system are small, fewer than 100 Megabytes, but for larger files, we automatically split those into multiple files and Cloud Run automatically creates additional parallel instances very quickly, helping the system scale almost instantly.
The nature of running a global website which provides news coverage means we see frequent, unpredictable large spikes of traffic. We learn from these and optimize our systems where necessary so we’re confident in the system’s ability to handle significant traffic. For example, around the time of the announcement of the Queen’s passing in September, we saw some huge traffic spikes. During the largest, within one minute, we went from running 150 – 200 container instances to over 1000…. and the infrastructure just worked. Because we engineered the log processing system to rely on the elasticity of a serverless architecture, we knew from the get-go that it would be able to handle this type of scaling.
Around the time of the announcement of the Queen’s passing in September, we saw some huge traffic spikes. During the largest, within one minute, we went from running 150 – 200 container instances to over 1000…. and the infrastructure just worked
Our initial concern about choosing serverless was cost. It turns out that using Cloud Run is significantly more cost-effective than running the number of VMs we would need for a system that could survive reasonable traffic spikes with a similar level of confidence.
Switching to Cloud Run also allows us to use our time more efficiently, as we no longer need to spend time managing and monitoring VM scaling or resource usage. We picked Cloud Run intentionally because we wanted a system that could scale well without manual intervention. As the digital distribution team, our job is not to do ops work on the underlying components of this system — we leave that to the specialist ops teams at Google.
Another conscious choice we made whilst rebuilding the system was to use the built-in service-to-service authentication in Google Cloud. Rather than implementing and maintaining the authentication mechanism ourselves, we add some simple configuration which instructs the client side to create and send a OIDC token for a service account we define and the server side to authenticate and authorize the client. Another example is pushing events into Cloud Run, where we can configure Cloud Run authorization to only accept events from specific EventArc triggers, so it is fully private.
Going forward, the new system has allowed us to make better use of our data safely. For example, BigQuery’s per-column permissions allow us to open up access to our logs to other engineering teams around the organization, without having to worry about sharing PII that’s restricted to approved users.
The goal of our team is to empower all teams within the BBC to get the content they want on the web when they want it, make it reliable, secure, and make sure it can scale. Google Cloud serverless products helped us to achieve these goals with relatively little effort and require significantly less management than previous generations of technology.
5338
Of your peers have already watched this video.
31:30 Minutes
The most insightful time you'll spend today!
Enhancing Collaboration with Sheets, Python, and Google Cloud
See how you can enhance collaboration within your organization using Google Sheets. Watch to learn about a new set of tools to create custom functions that tap into the power of Python and to expose functions in a standardized fashion throughout your organization.
The Evolving Landscape of Multicloud: A Journey, Not a Destination
2709
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Editor’s note: This post is part of an ongoing series on IT predictions from Google Cloud experts. Check out the full list of our predictions on how IT will change in the coming years.
Prediction: Over half of all organizations using public cloud will freely switch their primary cloud provider as a result of available multicloud capabilities
In the years ahead, companies will use a multicloud strategy not just as a way to hedge their bets, but as a way to switch from their first cloud to their next one. Research shows that the majority of companies are already multicloud, meaning they use more than one hyperscale cloud provider.
More and more, we’re talking to companies that describe using multicloud technologies as a way to do switch not just workloads — but mindshare — to a different cloud. In other words, for many people, multicloud is a phase, not a permanent state.
You may start using one cloud but still need to be able to incorporate existing investments you’ve already made in other clouds without having to move anything. Here at Google Cloud, we’ve made unique investments to make sure we can meet our customers wherever, and in whatever cloud, they are.
For instance, Anthos, our multicloud management plane, ensures consistency when working with your compute and data on other clouds. You can view workloads, deploy services, and apply common security policies across multiple clouds. BigQuery Omni allows you to query data in other cloud storage accounts in Amazon S3 or Azure Storage without having to move the data itself, helping to bring analytics to data wherever it resides.
Building new skills and getting comfortable in other clouds is not where multicloud stops. Many organizations are taking it a step further — upgrading technology, moving core data, and continuing to grow cloud adoption with a secondary provider.
What starts out as wanting the best capabilities to achieve IT goals will often lead organizations to swap from their first cloud to their next cloud — and by 2025, we believe that most organizations will be doing just that.
Kaluza & Google Cloud: Committed to Powering Up 73 Million EVs by 2040
1227
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Electric vehicles already account for one in seven car sales globally, and with new gas and diesel cars being phased out across the world, global sales are forecast to reach 73 million units in 2040. But with power grids becoming increasingly dependent on variable energy sources such as wind and solar, rising demand from electric vehicles risks overstraining grids at peak times, potentially leading to power outages.
At Kaluza, we believe that our platform has a vital role to play in helping power grids and utility companies to stabilize their networks, while at the same time delivering more affordable, cleaner energy to the consumer. Powered by Google Cloud, the advanced algorithms behind our Kaluza Flex solution automatically charge electric vehicles when the power supply is at its cheapest and greenest, helping to accelerate the global transition towards a zero-carbon future.
Decarbonizing the grid with low-cost smart charging programs
Launched by OVO Energy in 2019, Kaluza has taken its deep understanding of the energy market to partner with some of the world’s major energy suppliers and vehicle manufacturers, including AGL in Australia, Fiat and Nissan in the UK, and Mitsubishi Corporation and Chubu in Japan, to launch smart charging programs that help customers save money while reducing their carbon footprint.
A good example of this is Charge Anytime, which we recently launched with OVO Energy in the UK. With this tariff, customers use Kaluza to smart-charge their electric vehicle, and pay just 10p per kWh — a third of their household electricity rate — to do so. This means that if the customer plugs in their vehicle to charge when they get home from work at, say, 6:00 p.m. — a time when both demand and the carbon intensity on the grid are at their highest — their vehicle will then be smartly charged at the lowest cost and greenest periods throughout the night, ready for when they need it in the morning.
This smart charging reduces the energy company’s costs by enabling them to take advantage of lower wholesale electricity prices. These savings are then passed on to the end customer through tariffs such as Charge Anytime, saving customers hundreds of pounds a year and reducing their carbon footprint. Meanwhile, the National Grid is able to reduce the strain on the network during peak hours, while simultaneously using up the excess renewable energy that might otherwise have gone to waste.
Optimized charging schedules, fueled by Google Cloud
Behind Kaluza’s smart charging solution lies some sophisticated technology, all of which is built on Google Cloud. Our core optimization engine gathers real-time data from a wide range of sources, including battery and charging data from the electric vehicles, and data from the energy suppliers and grid operators, such as the carbon intensity, and price forecasts.
After passing through our real-time data backbone, that data is stored in BigQuery where it’s used to train and validate our smart charging optimization models. These models are then deployed with Google Kubernetes Engine so that whenever a customer plugs in an electric vehicle, data from that vehicle passes in real-time through our optimization engine to calculate the ideal charging schedule for that vehicle, ensuring it uses the cheapest, least carbon-intensive energy available.
Customer interface: easy to use, simple to build
Of course, the customer isn’t aware of any of this complexity. All they need to do is open their charging app and use Kaluza’s intuitive interface to set what time they want their car to be ready and how much charge they want in their battery. Then they simply plug in their car, and our algorithms take care of the rest.
Customers can also use Kaluza to view breakdowns of how much carbon and money they’ve saved, along with insights around things like billing and battery life, all of which is backed by Cloud SQL.
With Google Cloud, we were able to roll out this end-user app very quickly. Instead of having to build a different version of the app for each operating system, we were able to build an OS-agnostic version using Flutter, which then builds the app for each platform, enabling us to get to market faster.
This has been a benefit of our architecture in general. With Google Cloud taking the complexity out of otherwise time-consuming development processes, we’ve been able to experiment with and validate propositions rapidly, and roll out new products and features at speed, to ensure that we remain at the vanguard of a rapidly evolving sector.
Giving energy companies full visibility with BigQuery and Looker
As for the grid operators and energy companies, the Kaluza platform allows them to visualize how many participating electric vehicles are plugged into the network at any one time. BigQuery and Looker Studio dashboards provide granular insights, such as how many vehicles are idle, how many are charging, and how well our optimization engine is working.
The platform also allows companies to view those vehicles on an aggregate level, and identify any issues. Google Cloud machine learning capabilities can even allow grid operators to use this aggregate view to forecast how much energy will be required at any one time, and dial the power generation up or down accordingly.
Ultimately, these insights help network operators and utility companies to optimize energy usage, and balance out the peaks and troughs of supply and demand, ensuring that excess renewable energy is captured, while carbon-intensive fuel use is reduced.
Feeding energy back into the grid with bidirectional charging
Vehicle-to-Grid (V2G), or bidirectional, charging is something that we are very excited about, and have begun rolling out in the UK, as we prepare to launch in other global markets. Built on the same Google Cloud architecture as the rest of Kaluza Flex, V2G not only enables smart charging, but allows electric vehicles to feed stored energy back into the grid.
Imagine an electric vehicle battery that can store 40 kWh of energy, but only uses 5 kWh a day. That leaves 35 kWh a day that can be charged to the battery during times of low demand, when energy is cheapest and greenest, then fed back into the network during peak periods, removing the need for more fossil fuels to be burned to meet demand.
Not only does V2G go further than conventional smart charging to make use of renewable energy when it’s abundant and support the balancing of the energy system, it also results in even lower energy prices for the customer, as they are effectively selling energy back to the grid. For example, as part of a large domestic V2G trial made in partnership with OVO and Nissan, Kaluza saved customers an average of £450 a year, with some customers saving up to £800/year by selling surplus energy back to the grid — transforming their homes into mini power stations.
With Kaluza Flex, we have a platform that offers benefits for all parties, from the grid operators, through to energy retailers and vehicle manufacturers, and all the way to the customer. Now, our aim is to bring this exciting offering to as many markets as possible, a goal which is made easier thanks to the scalable infrastructure and time-saving solutions of Google Cloud.
As more people make the switch to electric vehicles, our goal is to ensure that smart charging becomes standard practice, as we help to deliver on the potential of electric vehicles to contribute to a greener, decarbonized future.
Smart Home Appliances Start-up Chooses Google Cloud to Wow European Customers
3452
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
The internet of things is everywhere now. Almost everyone has at least one connected device at home, likely a virtual assistant and perhaps a smart programmable thermostat. Nearly all new televisions come preinstalled with streaming apps. Even electric vehicles send drivers communications when tire pressure is low or other problems come up.
Smart home appliances aren’t quite as common just yet, but the market is growing rapidly worldwide. And at TecPal, we are helping lead the way.
I’ve had the privilege to work around the world, in Hamburg, Bangkok, and Beijing, and I’ve been in the amazing city of Hong Kong for the past five years. It’s an ideal location for a startup like ours.
I collaborated with a friend from a previous job in mainland China to come up with the idea to start our new venture. He owns a factory that manufactures all kinds of small kitchen appliances and sells them mostly to European retailers. When more and more of his customers started asking about smart appliances, he called me because he knew about my background in IT.
That’s where our idea was born for a more specialized developer of software for smart appliances. Given my friend’s manufacturing expertise and my software experience, we were really more than halfway there before even getting started. That was just four years ago, and today we employ more than 50 people.
We started by adding smart technology to the products his factory produces but realized that what we’re doing could be so much bigger. Why produce smart technology for just a handful of clients when the global market is growing? So now we’re transitioning from creating a customized solution for his products to a software-as-a-service model that can be used in all kinds of smart appliances.
Rapid delivery of custom apps
At TecPal, we started with kitchen appliances, but our software also works in washing machines, ovens, vacuum cleaners, and more. We put everything together in an end-to-end service: hardware integration, cloud services and consumer-facing apps. And the IoT management platform can roll out updates, manage the devices, and ensure their security. Manufacturers and retailers don’t need to know about software development, embedded systems development, cloud development, or mobile development. We can handle that for them so they can stay focused on their core areas of expertise.
Today we have appliances that come preloaded with recipes that are optimized for the appliance so you don’t have to worry about the time, the temperature, and other details. Everything is preset.
We launched multiple IoT products with a major European retailer in 2021 that includes not only preloaded recipes but also user-generated content geared to specific appliances. The Monsieur Cuisine Smart is an advanced multi-cooker utilizing TecPal’s IoT solution. The mobile companion app allows users to control the appliances remotely. They can check the remaining cooking time or stop one of the many cooking programs like kneading, steaming, frying from the living room or from their office across town. The ability to embed content in an appliance is valuable, and so is the ability to control the appliance remotely, but we double down on value by offering both.
Winning with best-in-class IoT technology
Companies in Northeast Asia generally shop around for technology partners based mostly on price, but that’s not us. We chose Google Cloud for the quality of its technology, especially its IoT Core fully managed services. We’re using almost everything in Google Cloud IoT Core, but we’re particularly reliant on the environment for its highly secure two-way device connection and management. We set up some clusters in Google Kubernetes Engine (GKE) and are using that as the underlying virtualization architecture because it allows us to quickly and cost-effectively scale up and down to meet changing customer demands.
We also use Google Cloud Pub/Sub as our message broker to update app statuses automatically and reliably in our back end, for our ETL processes, and for our data pipeline. At the same time, Google Cloud Dataflow is key because it’s a fast, efficient way to funnel all data from different devices and events into BigQuery for data collection, storage, and analytics. BigQuery is a great choice for us because most developers are already familiar with it as a relational database that supports advanced data analytics. We can expand our use of BigQuery without worrying about dimensions, and our new developers can come up to speed quickly on it.
We only end up checking in with a Google Cloud support team once a month, if that. If we don’t find what we’re looking for in documentation, we can just drop them an email. Their response time is excellent.
Beyond the immediate technology advantages we’ve realized by standardizing on Google Cloud, we’re also finding that it helps us win business with big clients in Europe. Google Cloud is an environment that other big businesses know and trust.
We’re still just getting started
We’re already seeing double-digit growth in our market, up to about 25 percent annually. More importantly, there’s a lot of runway in front of us before our growth is expected to plateau.
The market for smart consumer appliances is increasing every day, and there’s a lot of room to reach new regions with new products. Most countertop food processors and slow cookers are still old-fashioned, for instance, but they probably won’t stay that way—especially now that two out of three homeowners say their next appliance purchase will be a smart one.
Together with Google Cloud, we are committed to meeting that demand and exceeding expectations as people embrace new at-home technologies that will make their everyday lives easier.
If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.
Google Cloud is a Leader in Q1 2022’s Public Cloud Container Platform: Forrester
3240
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
We’re thrilled to share the news that leading global research and advisory firm Forrester Research has named Google Cloud a Leader in the recently published report The Forrester WaveTM: Public Cloud Container Platforms, Q1 2022. Forrester evaluated the container and cloud-native offerings of a select group of top public cloud container platform vendors across 29 comprehensive criteria.
We are proud that Forrester evaluated the strength and cohesion of our offerings, including Google Kubernetes Engine (GKE), Cloud Run, Anthos, Cloud Build, Cloud Deploy, Cloud Code and more, writing that, “Google Cloud is the best fit for firms that want extensive cutting-edge cloud-native capabilities for distributed workloads spanning public cloud, private cloud, and multicloud environments.”
Google Cloud received the highest possible scores in the criteria of service and application catalogs, microservice development support, service mesh support, serverless and FaaS support, DevOps automation, container image support, control plane configuration, hybrid cloud support, container networking, product vision, supporting products and services, market approach, revenue, and breadth of offering. We also achieved the highest score in the Strategy category of all the vendors evaluated.
Google Cloud is all-in on containers and cloud native
The cloud-native tools and technologies created by Google Cloud are already powering the most innovative, scalable and secure apps around the world, from the most exciting digital natives to the most important enterprise industry leaders. Why? Cloud native means building and running modern apps that intentionally take advantage of the global scale, pervasive automation, elastic infrastructure, and secure resiliency of the public cloud.
For enterprises, cloud native in practice means using containers, Kubernetes, serverless, and DevOps automation to build amazing customer-facing apps quickly, to modernize existing business-critical apps safely, and to operate them all on cost-efficient, powerful, and secure cloud infrastructure globally.Over the past decade, Google Cloud’s technology innovation has fueled various domains of the cloud native ecosystem, such as Kubernetes and Go languages as the foundation, Istio for service mesh, Kubeflow for machine learning, Knative for serverless, and Tekton for CI/CD. Long-term investment and practices in cloud-native power its superior product vision and excellent supporting products and services.
The Forrester Wave: Public Cloud Container Platforms, Q1 2022
Dedicated to simplicity, speed, and scale for your modern apps
Our mission is to create, integrate, scale, and secure the best open source and commercial cloud-native technologies – backed by a consistent cloud control plane – so you can spend more time dreaming of ways to delight your customers and less time building and operating platforms. We are committed to leading in cloud-native open source communities and making containers and Kubernetes accessible to everyone, from everywhere.
Here are a few recent highlights across our container services and tools, aimed at helping you build and modernize your most important apps with cloud native:
The most scalable fully managed Kubernetes service, Google Kubernetes Engine (GKE), achieved an overall solution score of 92 out of 100 in Gartner’s Solution Scorecard. In 2021, we introduced GKE Autopilot, a fully managed, security-hardened Kubernetes service optimized for production workloads. This unique mode of operation allows you to focus on your workloads while Google manages your cluster infrastructure. There’s nothing else like it. Then, we made GKE apps even faster with GKE image streaming. With proven scalability to 15K nodes in a single cluster and innovations such as four-way autoscaling, node auto-upgrades, integrated logging and metrics, cost optimization insights, native backup and restore, and multi-instance GPUs to accelerate machine learning workloads, GKE remains the best choice in managed Kubernetes services.
With Cloud Run, we expanded the range of users who benefit from containers to those who don’t know much about them. Introduced in 2019, Cloud Run combines the serverless attributes of autoscaling and developer experience with the flexibility of containers. Developers can use any language, runtime or binary, and deploy code using buildpacks to automatically build container images from source without worrying about provisioning machines and clusters. Cloud Run goes beyond FaaS and beyond earlier generations of serverless computing. Cloud Run runs more legacy workloads, integrates with Cloud Build for secure and compliant builds, offers deeper cost controls and billing flexibility, and encourages portability. We added support for social feeds, collaborative editing, and multiplayer games that use bidirectional streaming. Minimum instances reduce cold-start delays so you can run more latency-sensitive applications. And recently, we launched support for network file systems, allowing developers to share and persist data between multiple containers and services.
Anthos is at the heart of the Google Distributed Cloud, a portfolio of software and hardware solutions announced in 2021 that extend Google’s container platform services to the data center and the edge. Anthos is how we extend GKE to wherever you need cloud-native apps. Manage clusters on-premises on bare metal and VMware-virtualized servers, on AWS and Azure, and at the edge – all with a Google Cloud-backed control plane for consistent, automated operations at scale. We added a hosted service for configuration management to keep all your clusters in sync, and trimmed our installation footprint and streamlined cluster management with a new multi-cloud API that enables you to use a single API for full lifecycle management of Anthos Kubernetes clusters in AWS or Azure.
Finally, since no public cloud container platform is complete without powerful DevOps tools, we expanded our CI/CD offerings to make your developers even more productive, wherever they build and deploy cloud native apps. Use Cloud Code and Cloud Shell as your go-to cloud-native IDEs. Cloud Build is a fully managed serverless DevOps automation platform for use cases spanning CI/CD, Infrastructure-as-Code, AI/MLOps, and more, across infrastructure GKE, Cloud Run, Cloud Functions and more. Google Cloud Deploy is a fully managed continuous delivery service that provides one-click release promotion and roll-backs, built-in metrics, and out-of-the-box security. Artifact Registry and Container Analysis provide managed artifact repositories, vulnerability scanning, and help secure the software supply chain.Google Cloud has a solid cloud-native innovation roadmap, targeting simplicity at scale for enterprise clients.
The Forrester Wave: Public Cloud Container Platforms, Q1 2022
We are delighted and humbled to be recognized as a Leader in public cloud container platforms by Forrester. Grab your copy of The Forrester WaveTM: Public Cloud Container Platforms, Q1 2022 today and let us know how we can help you build and modernize your most important apps how you want and where you want.
More Relevant Stories for Your Company
How to Build A Basic Image Search Utility for Natural Language Queries
This post shows how to build an image search utility using natural language queries. Our aim is to use different GCP services to demonstrate this. At the core of our project is OpenAI's CLIP model. It makes use of two encoders - one for images and one for texts. Each encoder
Ease Your Migration and Modernization Journey with Microsoft and Windows on Google Cloud Demo Center
If you’re looking to migrate and modernize your Microsoft and Windows workloads, Google Cloud is your premiere destination. No matter what migration strategy you’ve selected or what value you’re looking to achieve, with Google Cloud you’re able to: simplify your migration and modernization journeyreduce your on-prem footprint and increase agilityoptimize license
Largest Beauty Retailer in the US Powers Digital Transformation with Google Cloud Smart Analytics
Digital technology offers increasing flexibility and choice to consumers. As a result, the retail industry is dramatically shifting toward more tailored and personalized experiences for shoppers, and businesses are rethinking how they deliver value to customers. This couldn’t be more true for the beauty retailing industry where leading companies are
How Toyota’s Google Cloud-powered Voice Assistant Gives a Turboboost to Drivers’ Experience
Over the decades, technology has helped us organize large amounts of physical information in ways that are streamlined, efficient, and easily accessible. Rows upon rows of encyclopedias are no longer needed; simply punch in or speak a query into Google Search and find numerous results at your fingertips. There’s no
