Editor’s note: When Europe’s largest mobile communications company, Vodafone, was asked by the European Commission to help understand population movement across the European Union and the UK to help fight COVID-19, it was able to provide anonymized mobile network-based insights to answer the call. Here’s how Vodafone, with the support of Google Cloud, rapidly mobilized the COVID-19 frontline, while respecting its customers’ privacy.

With the emergence of COVID-19 in early 2020, the European Commission—the executive branch of the European Union (EU)—knew that technology would be instrumental in its fight to control the pandemic. With various lockdowns imposed across its member states, the Commission was keen to predict and prevent the spread of COVID-19 and to manage the related social, political and financial impacts. 

Mobile network data helps track COVID-19 across the EU

Mobile networks produce location data, which can be turned into useful anonymous insights to understand population movement within a geographic area. The European Commission, working with mobile industry association GSMA (Groupe Speciale Mobile Association), asked Europe’s major mobile phone operators for help in producing insights to support the fight against COVID-19. As the largest mobile network operator within the EU, Vodafone saw this as a critical opportunity to participate. 

Vodafone had previous experience of using mobile network data to support pandemic research. For example, in 2019, Vodafone provided mobility pattern analysis to help track the spread of Malaria in Mozambique. And, during the early stages of the COVID-19 pandemic (prior to working with the European Commission), Vodafone assisted the Italian and Spanish governments in understanding their citizens’ mobility patterns. Vodafone had also previously offered anonymized and aggregated population mobility insights to support public transport and tourism authorities and retail organizations in a number of countries. Consequently, Vodafone was perfectly placed to play a greater role in supporting the European Commission’s response to the pandemic. 

When asked to assist the European Commission, Vodafone first considered how it could safely share its data with the governing body without providing details on the individual movements of its customers. It realized it could achieve this through an elaborate set of anonymization and aggregation techniques. Insights are aggregated from a minimum of 50 users and Vodafone only shared these anonymous insights and never the actual raw data with the Commission. As specified by the EU, these insights are then presented onto a large geographical region, typically a city or a county with thousands of people living in that area.

These insights illustrate how people move, helping to determine how lockdowns and self-isolation measures were impacting behaviors.

Using Google Cloud to collate and store population mobility data

In April 2020, Vodafone began migrating its operations, including its mobile data, to Google Cloud on servers in Europe and the UK with elaborate security safeguards, including encryption, building on a previous partnership. 

With the data residing in EU and UK data centers and not the United States, Vodafone could then retrieve anonymous insights from Google Cloud Storage instantaneously. Before supplying any information to the European Commission, however, Vodafone used Dataflow to validate the data and run a series of tests to ensure the database had accurate data, before ingesting and archiving the relevant metrics. For instant access, the data was then made available to the European Commission using a Redis database on Google Kubernetes Engine.

To ensure aggregate Vodafone customer data was always safe, secure, and anonymous, all entry points to the front-end were protected behind Google Cloud Armor, where only specific IP addresses were allowed. Using these tools, seamless data pipelines fed in predefined key performance indicators from each specified European market. While data quality measures ensured the definitions for metrics across markets were consistent and could be accurately compared.

The architecture (pictured below) shows how Vodafone integrated and anonymized its data on Google Cloud.

Live interactive dashboard shows population mobility in real-time

With its data integrated on Google Cloud, Vodafone created a live, interactive dashboard to track mobility patterns and share relevant information with the European Commission in real-time. 

The European Commission Joint Research Center (JRC) was able to gather valuable information from these insights, which enabled them to see where population mobility was aiding the spread of the disease, when cross-referenced with health data. It could also assess the implications of lockdowns on different populations and forecast cross-country spreading.

Mobile data aids disease modeling for multiple stakeholders

The Vodafone data became instrumental in modeling the likely course of the disease too. For example, the University of Southampton in the UK used it to predict the outcome of different coordinated COVID-19 exit strategies across Europe. This research was published in Science Magazine in September 2020. 

The Vodafone data dashboard continues to be used by individual governments, NGOs and organizations to further investigate the impacts of the pandemic and to measure the effectiveness of response strategies alongside the rollout of vaccination programs. The project also helped Vodafone win a DataIQ award for most effective stakeholder engagement. 

Using the learnings from this project, Vodafone has been able to adapt its own B2B solution, called Vodafone Analytics, by adaptIng and migrating the code to work in Google Cloud Platform. This solution has been rolled out across Germany, Greece, Portugal and South Africa, and new countries are being onboarded every day. Vodafone Analytics already has more than 100 customers leveraging it for a variety of use cases—Italian fashion retailer OVS, uses it for its smart retail operation, while global real estate company, JLL, uses it to understand the footfall passing through its properties. 

Working together, Vodafone and Google Cloud continue to help a range of organizations, governments, and NGOs navigate through the ongoing pandemic,  optimize their operations, and help the greater good, without infringing individuals’ fundamental rights to privacy.

To learn more about Google Cloud and Vodafone, watch our full interview here.

App Engine is a fully managed serverless compute option in Google Cloud that you can use to build and deploy low-latency, highly scalable applications. App Engine makes it easy to host and run your applications. It scales them from zero to planet scale without you having to manage infrastructure. App Engine is recommended for a wide variety of applications including web traffic that requires low-latency responses, web frameworks that support routes, HTTP methods, and APIs.

Environments

App Engine offers two environments; here’s how to choose one for your application:

• App Engine Standard – Supports specific runtime environments where applications run in a sandbox. It is ideal for apps with sudden and extreme traffic spikes because it can scale from zero to many requests as needed. Applications deploy in a matter of seconds. If your required runtime is supported and it’s an HTTP application, then App Engine Standard is the way to go.
• App Engine Flex – Is open and flexible and supports custom runtimes because the application instances run within Docker containers on Compute Engine. It is ideal for apps with consistent traffic and regular fluctuations because the instances scale from one to many. Along with HTTP applications it also supports applications requiring WebSockets. The max request timeout is 60 minutes. 

How does it work

No matter which App Engine environment you choose, the app creation and deployment process is the same. First write your code, next specify the app.yaml file with runtime configuration, and finally deploy the app on App Engine using a single command: gcloud app deploy.

Notable features

• Developer friendly – A fully managed environment lets you focus on code while App Engine manages infrastructure. 
• Fast responses – App Engine integrates seamlessly with Memorystore for Redis enabling distributed in-memory data cache for your apps.
• Powerful application diagnostics – Cloud Monitoring and Cloud Logging help monitor the health and performance of your app and Cloud Debugger and Error Reporting help diagnose and fix bugs quickly. 
• Application versioning – Easily host different versions of your app, and easily create development, test, staging, and production environments.
• Traffic splitting – Route incoming requests to different app versions for A/B tests incremental feature rollouts, and similar use cases.
• Application security – Helps safeguard your application by defining access rules with App Engine firewall and leverage managed SSL/TLS certificates by default on your custom domain at no additional cost.

Conclusion

Whether you need to build a modern web application or a scalable mobile backend App Engine has you covered. For a more in-depth look, check out the documentation. Click here for demos on how to use serverless technology and free hands-on training.https://www.youtube.com/embed/Xuf3J6SKVV0?enablejsapi=1&

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.

Google Cloud and the DORA research team are excited to announce our eight sponsors for the 2021 State of DevOps report. We recently launched the 2021 State of DevOps survey, a 25-min survey for the DevOps community to share how they are using DevOps to improve software delivery performance. So if you haven’t taken the survey yet, this is your chance! 

For those unfamiliar with the State Of DevOps report, it is the largest and longest running research of its kind. It provides an independent view into how teams and companies can drive powerful business outcomes, no matter what stage of the DevOps journey

With the State of DevOps reports we aim to capture how teams and companies are driving excellence in technology delivery through the implementation of DevOps practices, no matter what stage of the DevOps journey your team is in. If you’re wondering how your team measures up in your industry take our DevOps Quick Check and discover which capabilities you should focus on to improve your performance.

To capture a diverse array of information on how those in the DevOps community are performing, our eight sponsors of the 2021 State of DevOps survey are focussed on helping organizations of all sizes and industries to develop and deliver software faster across the whole DevOps lifecycle. Captured below, you’ll find more information on our sponsors for this year and why they chose to sponsor the State Of the DevOps 2021 Report.

Armory

Armory enables enterprise companies to ship better software, faster through trusted, reliable, safe, and secure deployments — at its core, Armory is powered by Spinnaker OSS.

“Armory exists to unlock innovation through software. To achieve this, we help enterprises reliably deploy software at scale, naturally aligning with DevOps practices to improve software delivery performance. Given this, we applaud efforts like the State of DevOps Report that provides an independent view into the practices and capabilities that organizations can employ to drive better performance.” 

Carl Timm, Senior Director of Product Marketing at Armory

CircleCI

The world’s best software teams deliver quality code, confidently, with CircleCI. The world’s best software teams use CircleCI to go from next-up to feature shipped, at the speed ambitious businesses require. 

“Though the DevOps space is only over a decade old, it moves incredibly quickly. Google’s State of DevOps report is both a reflection and projection of the industry, capturing how DevOps culture drives business results and where DevOps practitioners can look to improve. At CircleCI, we rely heavily on this survey data to glean valuable insights into our market and our customers overall. We also build upon these insights to guide our own research into how engineering teams move code through pipelines in our annual State of Software Delivery report. Taken together, this research highlights teams’ reported and actual behavior and paints a vibrant picture of how technology-driven organizations drive for success.”

Emma Webb, VP, Corporate Communications, CircleCI 

Continuous Delivery Foundation

CD Foundation is an open-source community improving the world’s ability to deliver software with security and speed.

“Good decision-making is based on good data. Open source is a critical piece of the technology roadmap for DevOps, and the 2021 State of DevOps Report will provide actionable information for high performing teams in organizations of all types and sizes. The report will show where open source and DevOps intersect, and in a space that is changing so quickly, new relevant data is critical,” said Tracy Miranda, Continuous Delivery Foundation Executive Director. “CD Foundation members are deeply involved with producing this annual report, with over 10 years of historic research. Thank you to Google and our other members who have worked so hard on it. This report is a tangible result of working together.”

Tracy Miranda, Continuous Delivery Foundation Executive Director

Deloitte

Deloitte provides audit and assurance, tax, consulting, and risk and financial advisory services to a broad cross-section of the largest corporations and governmental agencies. At Deloitte, they are continuously evolving how they work and how they look at marketplace challenges so they can continually deliver measurable, sustainable results for their clients and communities.

“Software delivery is evolving rapidly and we know our customers need unique and compelling insights to make the right decisions. State of DevOps is a widely used report and considered as an Industry standard to understand the drivers for excellence in Software Development and Delivery. Deloitte is excited to help the team at DORA and Google Cloud in publishing this report through our sponsoring since we believe the insights from this report will help make software delivery better.” 

Manoj Mishra, Consulting Managing Director, Deloitte Consulting LLP

GitLab

GitLab is the open DevOps platform built from the ground up as a single application for all stages of the DevOps lifecycle enabling Product, Development, QA, Security, and Operations teams to work concurrently on the same project.

“We’re happy to sponsor the DORA Report and the vital work behind it. The more we understand the DevOps journey, the better we and our GitLab community can contribute to its future. We’re particularly interested to see this year’s results and the impact COVID-19 and remote work have had on DevOps. We appreciate the chance to be part of this exciting, informative process.”

Brendon O’Leary, Senior Developer Evangelist

Liquibase

Liquibase helps millions of developers easily manage database schema changes by enabling teams to track, version, and deploy database changes by delivering on the promise of CI/CD for the database.

“With their rigorous methodology, the DORA research team delivers actionable information with the simple goal of helping organizations of any size accelerate the development and delivery of software. Liquibase is honored to sponsor the 2021 State of DevOps Report and its role in helping so many organizations build value for their customers and shareholders”

Matt Geise, VP of Marketing at Liquibase

PagerDuty

PagerDuty is a digital operations management platform that empowers the right action, when seconds matter.

“As a leader in digital operations management, PagerDuty is proud to sponsor this year’s report as it aligns with our dedication to helping DevOps professionals make better decisions. This report will inform tech and business leaders about the trends/challenges developers are facing and the opportunities there are to accelerate their own DevOps transformation.” 

Carolyn Guss, VP of Corporate Marketing

SysDig

Sysdig is driving the secure DevOps movement, empowering organizations to confidently secure containers, Kubernetes and cloud.

“There is a learning curve with all new technology, cloud is no exception. Learning from mistakes and sharing best practices is how we will ultimately ship secure applications, faster. For seven years, DORA and Google have partnered to understand the State of DevOps to help vendors and cloud practitioners to learn from each other and implement standards for best practices,”

Loris Degioanni, CTO and founder of Sysdig

Google Cloud, the DORA team, and our sponsors are very excited about this year’s report. We look forward to hearing from you, your colleagues, and networks about how DevOps is integrated into your workflow and ways we can help to further improve your performance. Please share your experience with software delivery by completing our survey that will be used to foster the next generation of DevOps best practices. To provide ample time for the DevOps community to contribute to this industry wide report we have extended the deadline for the survey until July 2nd.

Thank you to everyone who has participated so far, and the Google Cloud DORA team looks forward to hearing from even more of you soon!

The name Kitabisa means “we can” in Bahasa Indonesia, the official language of Indonesia, and captures our aspirational ethos as Indonesia’s most popular fundraising platform. Since 2013, Kitabisa has been collecting donations in times of crisis and natural disasters to help millions in need. Pursuing our mission of “channeling kindness at scale,” we deploy AI algorithms to foster Southeast Asia’s philanthropic spirit with simplicity and transparency.

Unlike e-commerce platforms that can predict spikes in demand, such as during Black Friday, Kitabisa’s mission of raising funds when disasters like earthquakes strike is by definition unpredictable. This is why the ability to scale up and down seamlessly is critical to our social enterprise.

In 2020, Indonesia’s COVID-19 outbreak coincided with Ramadan. Even in normal times, this is a peak period, as the holy month inspires charitable activity. But during the pandemic, the crush of donations pushed our system beyond the breaking point. Our platform went down for a few minutes just as Indonesia’s giving spirit was at its height, creating frustrations for users.

A new cloud beginning

That’s when we realized we needed to embark on a new cloud journey, moving from our monolithic system to one based on microservices. This would enable us to scale up for surges in demand, but also scale down when a wave of giving subsides. We also needed a more flexible database that would allow us to ingest and process the vast amounts of data that flood into our system in times of crisis.

These requirements led us to re-architect our entire platform on Google Cloud. Guided by a proactive Google Cloud team, we migrated to Google Kubernetes Engine (GKE) for our overall containerized computing infrastructure, and from Amazon RDS to Cloud SQL for MySQL and PostgreSQL, for our managed database services.

The result surpassed our expectations. During the following year’s Ramadan season, we gained a 50% boost in computing resources to easily handle escalating crowdfunding demands on our system. This was thanks to both the seamless scaling of GKE and recommendations from the Google Cloud Partnership team on deploying and optimizing Cloud SQL instances with ProxySQL to optimize our managed database instances.

A progressive journey to kindness at scale

While Kitabisa’s mission has never wavered, our journey to optimized performance took us through several stages before we ultimately landed on our current architecture on Google Cloud.

Origins on a monolithic provider
Kitabisa was initially hosted on DigitalOcean, which only allowed us to run monolithic applications based on virtual machines (VMs) and a stateful managed database. This meant manually adding one VM at a time, which led to challenges in scaling up VMs and core memory when a disaster triggered a spike in donations.

Conversely, when a fundraising cycle was complete, we could not scale down automatically from the high specs of manually provisioned VMs, which was a strain on manpower and budgetary resources.

Transition to containers
To improve scalability, Kitabisa migrated from DigitalOcean to Amazon Web Services (AWS), where we hoped deploying load balancers would provide sufficient automated scaling to meet our network needs. However, we still found manual configurations to be too costly and labor-intensive.

We then attempted to improve automation by switching to a microservices-based architecture. But on Amazon Elastic Container Service (Amazon ECS) we hit a new pain point: when launching applications, we needed to ensure that they were compatible with CloudFormation in deployment, which reduced the flexibility of our solution building due to vendor locking.

We decided it was “never too late” to migrate to Kubernetes, which is a more agile containerized solution. Given that we were already using AWS, it seemed natural to move our microservices to Amazon Elastics Kubernetes Service (Amazon EKS). But we soon found that provisioning Kubernetes clusters with EKS was still a manual process that required a lot of configuration work for every deployment.

Unlocking automated scalability
At the height of the COVID-19 crisis, faced with mounting demands on our system, we decided it was time to give Google Kubernetes Engine (GKE) a try. Since Kubernetes is a Google-designed solution, it seemed likeliest that GKE would provide the most flexible microservices deployment, alongside better access to new features.

Through a direct comparison with AWS, we discovered that everything from provisioning Kubernetes clusters to deploying new applications became fully automated, with the latest upgrades and minimal manual setups. By switching to GKE, we can now absorb any unexpected surge in donations, and add new services without expanding the size of our engineering team. The transformative value of GKE became apparent when severe flooding hit Sumatra in November 2021, affecting 25,000 people. Our system easily handled the 30% spike in donations.

Moving to Cloud SQL and ProxySQL

Kitabisa was also held back by its monolithic database system, which was prone to crashing under heavy demand. We started to solve the problem by moving from a stateful DigitalOcean database to a stateless Redis one, which freed us from relying on a single server, giving us better agility and scale.

But the strategy left a major pain point because it still required us to self-manage databases. In addition, we were experiencing high database egress costs due to the need to execute data transfers from a non-Google Cloud database into BigQuery.

In December 2021, we migrated our Amazon RDS to Cloud SQL for MySQL, and immediately saved 10% in egress costs per month. But one of the greatest benefits came when the Google Cloud team recommended using the open source proxy for MySQL to improve the scalability and stability of our data pipelines.

Cloud SQL’s compatibility allowed us to use connection pooling tools such as ProxySQL to better load balance our application. Historically, creating a direct connection to a monolithic database was a single point of failure that could end up in a crash. With Cloud SQL plus ProxySQL, we create layers in front of our database instances. It serves as a load balancer that allows us to connect simultaneously to multiple database instances, by creating a primary and a read replica instance. Now, whenever we have a read query, we redirect the query to our read replica instance instead of the primary instance.

This configuration has transformed the stability of our database environment because we can have multiple database instances running at the same time, with the load distributed across all instances. Since switching to Cloud SQL as our managed database, and using ProxySQL, we have experienced zero downtime on our fundraising platform even when a major crisis hits.

We are also saving costs. Rather than having a separate database for each different Kubernetes cluster, we’ve merged multiple database instances into one instance. We now group databases according to business units instead of per service, yielding database cost reductions of 30%.

Streamlining with Terraform deployment

There’s another key way in which Google Cloud managed services have allowed us to optimize our environment: using Terraform as an infrastructure-as-a-code tool to create new applications and upgrades to our platform.

We also managed to automate the deployment of Terraform code into Google Cloud with the help of Cloud Build, and no human intervention. That means our development team can focus on creative tasks, while Cloud Build deploys a continuous stream of new features to Kitabisa.

The combination of seamless scalability, resilient data pipelines, and creative freedom is enabling us to drive the future of our platform, expanding our mission to inspire people to create a kinder world in other Asian regions.

We believe that having Google Cloud as our infrastructure backbone will be a critical part of our future development, which will include adding exciting new insurtech features. Now firmly established on Google Cloud, we can go further in shaping the future of fundraising to overcome turbulent times.