Climate scientists face a deluge of environmental data to analyze and interpret from real-time sensors and satellites across the globe. The stakes are as high as our planet’s long-term future, but rapidly changing conditions are already impacting communities through extreme events like floods and wildfires as well as management of everyday resources. In this context, detailed environmental maps are key sources for urgent global issues like food security, water quality, and vegetation levels.

Scientists, researchers, and developers rely on state-of-the-art cloud computing tools like Google Earth Engine (EE) to detect changes, map trends, and quantify differences on the Earth’s surface. EE leverages the compute power of Google Cloud to combine a multi-petabyte catalog of satellite imagery and geospatial datasets with planetary-scale analysis capabilities.

To find out more about how climate scientists address these complex research challenges, I spoke to Dr. Richard Fernandes, Research Scientist at Natural Resources Canada (NRCan). Dr. Fernandes is also a 2022-23 member of the Google Research Innovators Program, which provides technical and professional support to a global cohort of leading researchers. I asked him to describe how cloud computing and machine learning (ML) tools can support climate resilience research and drive awareness about climate sustainability.

Dr. Fernandes, can you give me an overview of your research in climate sciences?

My research focuses on mapping the status and trends of vegetation over Canada. Every month we generate maps of vegetation parameters, such as canopy cover at 20m resolution, to support environmental monitoring and assessment. These maps contribute to global datasets that are used to reduce uncertainty in weather and climate forecasts.

Canada has approximately 10 million square kilometers of land and the annual data volume of these maps is equivalent to streaming HD movies for over 750 hours non-stop. And that’s only the tip of the iceberg: the volume of input data we need to generate those maps is typically 100 times more. Unlike movie streaming services, we have to independently process each input pixel to locate it accurately, screen for clouds (and even the shadows of clouds), and then transform it into a vegetation parameter value like canopy cover using ML algorithms.

The volume of high-resolution data and the amount of compute we need are challenging and ever-increasing. Rather than dedicating servers 24/7 for constant monitoring, we rely on cloud computing and ML. Cloud computing allows us to manage all this data in a useful and accessible way. We have also been able to successfully use the Google Artificial Intelligence (AI) Platform to calibrate new ML models for third-party datasets.

How did you start working with Google Cloud?

I started using EE together with open source APIs for integration with Google Drive about four years ago. With the pandemic my research group has transitioned to using EE, Google Cloud, and Google Drive for both our Canada-wide mapping as well as for R&D activities. In January we developed and released the LEAF toolbox, which builds on EE satellite data to create customizable maps of foliage density in near real-time. We do all of the pixel processing in EE and leverage the ability to integrate our own functions using their APIs. We combine EE with Google Cloud to handle and manage output datasets. Fortunately, EE has most of the input data already at hand so we don’t need to deal with that 100x larger volume.

What impact do you expect LEAF will have now–and down the road?

Both the Canadian federal government and provinces already use our data products as inputs to permafrost, crop status, and water resource models. Agriculture Canada had been using a conventional Geographic Information System (GIS) and approached us to ask how we manage our data. They want to use LEAF to assess how crops are progressing, which impacts local economies and the global food supply. We’re in a pilot with them to apply EE to their crop condition assessments.

Also exciting is that the Province of Alberta is integrating the LEAF toolbox within a system for monitoring the reclamation status of oil and gas wells and mines. They want to know how to rehabilitate sites that were developed for mineral and gas deposits. This is another great use case but many others are possible. The technology itself is cool, but that’s not even the point. ML algorithms are constantly evolving and improving. New ML algorithms use active learning that detects mistakes and makes updates on the fly. The technology will be different in five years — or five months! — so our priority is making these tools accessible and useful now.

Why is state-of-the-art technology for climate research so important?

Scientific research must be validated, transparent, and rigorous to drive the best solutions to our complex and changing ecosystem. Climate action needs greater public awareness, which requires more knowledge, which demands the best data. By democratizing information and decision-making, we can create an ecosystem of openness and public-private partnerships. We can lower the barrier to entry for advanced research and help scientists validate and reproduce their results. Also, scientists don’t want to have to become software engineers for these custom highly specific solutions. They want user-friendly tools that let them focus on their analyses and share their insights with the world. That’s one of the major appeals of the Google Research Innovators Program for me: I want to share LEAF with colleagues and collaborate with other researchers who are using new tools in new ways.

Do you have any parting words about your mission?

We’re so fortunate to have Canadian taxpayers and the Government of Canada funding our work. We have worked with many scientists over the past two decades to develop and validate the algorithms we use, especially Fred Baret and Marie Weiss at INRA France, who have championed the idea of free and open access to algorithms and knowledge, and data.

I really believe in democratic access to information. I like the fact that the terms of service of Google products allows us to offer not only maps but the actual processing system in a free and open manner to everyone. I also like that EE provides a simple-to-use user interface that works on mobile devices. We designed LEAF not just for experts, but for individuals. My mom was able to make maps of a nearby park in real time on her tablet. It is my hope that expanding access to critical environmental information will increase our collective awareness of how our actions impact both near and far places — and make us active in the cause of a sustainable future.

That’s an inspiring note to end on. Thank you for your time!

Thanks for having me!

Click here to learn more about Google’s commitment to renewable energy. Or try the LEAF toolbox for yourself!

Editor’s note: PedidosYa is the market leader for online food ordering in Latin America, serving 15 markets and over 400 cities. It’s also one of the largest brands within the German multinational company Delivery Hero SE. With over 20 million app downloads, PedidosYa provides the best online delivery experience through 71,000+ online partners, including restaurants, shops, drugstores, and specialized markets. 

Having constant access to fresh customer data is a key requirement for PedidosYa to improve and innovate our customer’s experience. Our internal stakeholders also require faster insights to drive agile business decisions. Back in early 2020, PedidosYa’s leadership tasked the data team to make the impossible possible. Our team’s mission was to democratize data by providing universal and secure access while creating a comprehensive information ecosystem across PedidosYa. We also had to achieve this goal while keeping costs under control— even during the migration stage and removing operational bottlenecks. 

Challenges with legacy cloud infrastructure

PedidosYa first built its data platform on top of AWS. Our data warehouse ran on Redshift, and our data lake was in S3. We used Presto and Hue as the user interfaces for our data analysts. However, maintaining this infrastructure was a daunting task. Our legacy platform couldn’t keep up with the increasing analytics demands. For example, the data stored on S3 complemented by Presto/Hue required high operational overhead. This was because Presto and our IAM (identity access management) didn’t integrate well in our legacy ecosystem. Managing individual users and mapping IAM roles with groups and Kerberos was operationally time-consuming and costly. Further, sharding access on the S3 files was far too complicated to enable seamless ACLs (access control lists).  

There were also challenges with workload management. Our data warehouse had batch data loaded overnight. If one analyst scheduled a query to run during the overnight ETL (extract, transform, load) workload, it would disrupt the current ETL task. This could stop the entire data pipeline. We’d have to wait until data engineers intervened with a manual fix.

It was also difficult to understand whether a query error was due to performance issues or platform resource exhaustion. This lack of clarity affected our data analysts’ ability to autonomously improve querying efficiency. Data team members needed to manually inspect personal queries looking for performance issues. Also,  the current architecture was prone to a ‘tragedy of the commons’ situation; it was seen as an unlimited and free resource. As a result, it was impossible to disentangle the infrastructure from different stakeholder teams, as all had very different needs. 

The decision to modernize our data warehouse

Given the growing challenges from our legacy platform, our tech team decided to transform our analytics environment with a modern data warehouse. They required the following key criteria from their next data platform: 

• Scalability – The ability to grow with elastic infrastructure.
• Cost control – Cost management and transparency. These factors promote efficiency and ownership—both key aspects of data democratization.
• Metadata management – Intuitive data platform focusing on users’ previous SQL knowledge. Plus, being able to enrich the informational ecosystem with metadata,  to diminish data gatekeepers.
• Ease of management – The team needed to reduce operational costs with a serverless solution. Data engineers wanted to focus on their key roles rather than acting as database administrators and infrastructure engineers. The team also wanted much higher availability, and to reduce the impact of maintenance windows and vacuum/analysis.
• Data governance and access rights – With a growing employee base with varying data access requirements, the team needed a simple yet comprehensive solution to understand and track user access to data.

Migrating to Google Cloud

After exploring other alternatives, we concluded Google Cloud had an answer to each of our decision drivers. Google Cloud’s serverless, managed, and integrated data platform, coupled with its seamless integration across open-source solutions, was the perfect answer for our organization. In particular, the natural integration with Airflow as a job orchestrator and Kubernetes for flexible on-demand infrastructure was key.  

We  used Dataflow together with Pub/Sub and Cloud Functions for our data ingestion requirements, which has made our deployment process with Terraform seamless. Because we set up everything in our environment programmatically, operation time has diminished. Google Cloud reduced the deployment process from about 16 hours in our legacy platform to 4 hours.  This is partly due to the friendliness of automating the deployment (such as schema check, load test, table creation, build.) process with Terraform, Cloud Functions, Pub/Sub, Dataflow, and BigQuery on GCP. Input messages processed with Dataflow allow us to abstract and plan the schema changes according to the needs of the functional team. For example, schema changes raise an alarm, and then we can modify the raw layer table schema. By doing this, we ensure that backend modifications that we don’t control do not affect upper layers.

A key reason why we picked Google Cloud was because of its advanced cost and workload management coupled with its transparent log analytics. This information gives us a complete view into any query performance issues to make improvements on the fly. Further, we achieved a significant amount of cost savings by consolidating multiple tools to BigQuery.With BigQuery, we’ve been able to reduce our total cost per query by 5x.

This was due to a number of reasons:

• Automating pipeline deployment made it much simpler to maintain the data processing processes. 
• Analysts are conscious about what queries they’re running, resulting in running better, more optimized queries. 
• Analysts use a Data Studio dashboard to see their queries and all the associated costs. As a result, there’s a lot more transparency for each persona.

 With these changes, we can easily manage and assign costs associated with each workload with their own cost centers using specific Google Cloud projects.

Change management is always challenging. However, BigQuery is intuitive and doesn’t have a steep learning curve from Hue/Hive on SQL basics. BigQuery also allowed the team to expand its capabilities and enabled them to properly work with nested structures, avoiding unnecessary joins and improving query efficiency. Additionally, we now use Data Catalog as our unique point of truth for metadata management. This allows our team to break the data access barriers and enable federation of data across the organization. By using Airflow to orchestrate everything, we keep track of every data stream. With this information, each end user can see their regularly used data entities’ status via the dashboard. This also adds transparency to our everyday data processes.

Finally, with Google Cloud’s IAM rules applied across the different products, data sharing and access is close to a noOps experience. We have programmatically implemented access according to roles and level access within the company. This allows certain pre-validated roles to view more sensitive information. These solutions help drive a more automated data governance experience. 

Up next: Google Cloud AI/ML

The new stack based on BigQuery has created significant productivity gains. Freed from the burden of operational management, PedidosYa’s data team can now focus on adding value through data tools and products.  

• Our data engineers are better equipped to integrate constantly changing transactional and operational data.
• The dataOps team can automate the infrastructure and provide autonomy to the end user.
• Our data quality team can focus on bringing added value to data stakeholders. 
• Data scientists and data analytics can spend more time analyzing data and less time asking data gatekeepers for data access.

PedidosYa can now democratize data access with a well-governed architecture. We are still at the beginning of our journey, but we are closer to achieving our vision of building a data-driven organization. Up next: expanding our artificial intelligence and machine learning capabilities.

Tune in to Google Cloud’s Applied ML Summit on June 10th, 2021, or listen on-demand later, to learn how to apply groundbreaking machine learning technology in your projects.

In the past year, Google has worked to surface timely and reliable health information, amplify public health campaigns, and help nonprofits get urgent support to Indians in need. Now, we are continuing to focus on helping India’s businesses accelerate their digital transformation, deepening our commitment to India’s digitization and economic recovery. To support customers and the public sector in India and across Asia Pacific, we’re excited to announce that our new Google Cloud region in Delhi National Capital Region (NCR) is now open. 

Designed to help both Indian and global companies alike build highly available applications for their customers, the Delhi NCR region is our second Google Cloud region in India and 10th to open in Asia Pacific. 

What customers and partners are saying

Navigating this past year has been a challenge for companies as they grapple with changing customers demands and economic uncertainty. Technology has played a critical role, and we’ve been fortunate to partner with and serve people, companies, and government institutions around the world to help them adapt. The Google Cloud region in Delhi NCR will help our customers adapt to new requirements, new opportunities and new ways of working, like we’ve helped so many companies do in the region: 

• InMobi scaled a personalized AI platform to support 120+ million active users. “With the arrival of the Google Cloud Delhi NCR, InMobi Group sees the opportunity to continue closing the gap between our users and products,” says Mohit Saxena, Co-founder and Group CTO of Inmobi. “Glance, especially, has been serving AI-powered personalised content to over 120 million active users. We can’t wait to continue giving them truly meaningful experiences that are speedy, scale well, and are relevant to them, by expanding the use of our current tools working on Google Cloud with the opening of a new region.”
• Groww now supports a sizable user base. “Google Cloud provides great technology that enables us to build and scale infrastructure to millions of users, and the new Google Cloud region in Delhi NCR will continue to help more businesses and startups in India access powerful cloud-based infrastructure, products and services,” says Neeraj Singh, Co-founder and Chief Technology Officer, Groww.
• HDFC Bank is positioned for the future. “At HDFC Bank, we are harnessing technology platforms to both run and build the bank. As we progress to be future ready, the objective is to invest in future technologies that give us scale, efficiency and resiliency. Towards this the Google Cloud region in Delhi NCR will enable us to enhance our resiliency and help us in building an active-active design framework for our new generation applications on cloud,” says Ramesh Lakshminarayanan, CIO, HDFC Bank.  
• Dr. Reddy’s Lab built a modern data platform with Google Cloud. “At Dr Reddy’s, we pride ourselves in helping patients regain good health, acting quickly to provide innovative solutions to address patients’ unmet needs and in accelerating access to medicines to people worldwide. Our Google Cloud-powered data platform is helping us realize these objectives and we welcome Google’s investment in the new Delhi NCR region as helping us and other businesses in India make further contributions to our social and economic future,” says Mukesh Rathi, Senior Vice President & CIO, Dr. Reddy’s Laboratories.
• “To survive the disruption caused by the pandemic and to succeed in the long term, organizations need to become digital natives, so they can be more agile, explore new business models and build new capabilities that boost resilience. A cloud-first strategy plays a key role in enabling businesses to do this,” said Piyush N. Singh, Lead – India market unit & lead – Growth and Strategic Client Relationships, Asia Pacific and Latin America, Accenture. “Harnessing the potential of cloud requires the right data infrastructure and this expansion by Google Cloud will undoubtedly help Indian enterprises in their digital transformation journeys.”

A global network of regions

Delhi NCR joins 25 existing Google Cloud regions connected via our high-performance network, helping customers better serve their users and customers throughout the globe. As the second region in India, customers benefit from improved business continuity planning with distributed, secure infrastructure needed to meet IT and business requirements for disaster recovery, while maintaining data sovereignty.

With this new region, Google Cloud customers operating in India also benefit from low latency and high performance of their cloud-based workloads and data. Designed for high availability, the region opens with three availability zones to protect against service disruptions, and offers a portfolio of key products, including Compute Engine, App Engine, Google Kubernetes Engine, Cloud Bigtable, Cloud Spanner, and BigQuery. 

Supporting India’s recovery with training and education

Google and Google Cloud will also continue to support our customers with people and education programs. We’re investing in local talent and the local developer community to help enterprises digitally transform and support economic recovery. 

Through the India Digitization Fund, we expanded our efforts to support India’s recovery from COVID-19—in particular, through programs to support education and small businesses. In addition to expanding internet access, and investments to help start-ups accelerate India’s digital transformation, we’ve grown our Grow with Google efforts. Businesses can access digital tools to maintain business continuity, find resources like quick help videos, and learn digital skills—in both English and in Hindi.

Helping customers build their transformation clouds

Google Cloud is here to support businesses, helping them get smarter with data, deploy faster, connect more easily with people and customers throughout the globe, and protect everything that matters to their businesses. The cloud region in Delhi NCR offers new technology and tools that can be a catalyst for this change. To learn more, visit the Google Cloud locations page, and be sure to watch the region launch event here.

Google Cloud Results

• Enables accurate allocation of marketing spend to underserved areas
• Supports expansion into international markets
• Helps ensure accurate forecasting of inventory levels
• Improves accuracy of forecasted delivery times by at least 60%

Operating in India since 2011, Rebel Foods has grown from a brick-and-mortar business that provided wraps to customers to a cloud kitchen that delivers cuisine to about one million consumers per month. “We started with the Faasos food brand and now we have scaled up to 10 brands,” says Soumyadeep Barman, Chief Technology Officer at Rebel Foods. “We have doubled our revenue every year from 2014 until now, and we operate kitchens in 15 cities across India. Each kitchen offers at least seven of our brands to customers.”

Barman attributes Rebel Foods’ success to the fact that it is a full stack company. “We procure, we have our own inventory, we prepare the food, we deliver the food to customers, and we make sure customers are delighted every time they order,” he says.

The rapid emergence and adoption of mobile technologies and services in India gave the business its opportunity to expand quickly. “The boom in applications and the web really got going in India in about 2014,” Barman says. “The subsequent emergence of smart devices and mobile applications opened up new markets, including older people who had not really used a computer until then.”

The business released the first iteration of its mobile application in 2013 on servers in an on-premises data center. “However, we experienced breakages because our infrastructure was not scalable or dependable enough, and we decided to move to another solution,” Barman says.

Google Maps Platform delivers opportunity

In 2014, Rebel Foods decided to move to the cloud and selected Google Cloud because of its stability, reliability, and scalability.

The business also wanted to take advantage of the opportunities Google Maps Platform presented to improve the efficiency and effectiveness of its delivery service. With 175 kitchens delivering to about 900 locations across India, Rebel Foods needs to provide estimated delivery times and meet delivery guarantees, while accounting for all the factors that might affect how quickly a rider can reach a customer’s doorstep.

The business turned to Google Maps Platform Premier Partner Searce for support in leveraging Google Maps Platform APIs to deliver a compelling customer experience and improve its efficiency. “Searce helped us determine the Google Maps Platform APIs we should use across our mobile applications and websites, and how many licenses we needed to conduct activities like calculating estimated delivery time and reviewing order heat maps,” Barman says. “Thanks to the firm’s support, Google Maps Platform APIs were a game changer for us.”

Mapping customer locations

Customers accurately pinpoint their location in a map through functionality made available through the Places API and Geocoding API, in conjunction with the JavaScript API. Drivers use the Directions API to identify the quickest route to customers.

Customers can also track the progress of delivery and estimated time of arrival using an Android or iOS application, or the brand websites.

Deploying Google Maps Platform APIs enabled Rebel Foods to improve by up to 60 percent the accuracy of forecasted delivery times. “Rather than tell a customer we can reach them in, say, 45 minutes, based on previous experience and gut feeling, we can retrieve an accurate traffic scenario and calculate delivery times based on traffic congestion levels and likely average speeds,” Barman says

Allocating budget effectively

Google Maps Platform also allows the business to combine mapping of customers to individual kitchens and to how often customers place orders – and for what value. This enabled the business to understand where to allocate budget for local marketing to stimulate demand in underserved areas.

Google Maps Platform technologies complement Rebel Foods’ use of Google Cloud Platform services such as the BigQuery analytics data warehouse to process data used to forecast inventory levels and provide recommendations to customers based on previous usage and behaviors. The business also runs its key applications in Kubernetes Engine to achieve cost-effective scalability, so it can expand to international markets.

“We are targeting growth into a range of international markets in January 2019, including Australia, the Middle East, and Southeast Asia,” Barman says. “With the user data and experience provided by Google Maps Platform in particular, we are poised for success.”

State and local government (SLG) agencies are reeling from a combination of unbudgeted COVID-related expenses and reduced tax revenue caused by unemployment and business closures. Any way you look at it, the situation is challenging. To understand how SLG agencies are coping, Google Cloud collaborated with MeriTalk to survey 200 SLG IT and program managers, uncovering some revealing trends in SLG technology innovation. Unsurprisingly, approximately 84% of SLG organizations report making budgetary tradeoffs to bridge funding the gaps the ongoing pandemic has created. 

However, researchers discovered a silver lining: The pandemic has also been a catalyst to modernize the legacy infrastructure in states and cities. The majority of survey respondents (88%) reported that their agency made greater modernization progress this past year than in the prior 10 years.

Walking a tightrope between innovation and budget pressure

According to 89% of state and local leaders, now is the time to invest in technology modernization. But 80% are experiencing a funding gap due to unbudgeted expenses related to the pandemic and declining tax revenue, which makes finding that balance between innovation and budget a serious challenge.

Some agencies are achieving the impossible, though. For example, the City of Pittsburgh Department of Innovation and Performance is working with Google Cloud to migrate  and modernize its legacy IT infrastructure. By decommissioning their data center and moving to Google Cloud, the city can build new data analytics tools to drive smart city initiatives and create entirely new applications to improve digital service delivery for its residents. As a result, the city will save costs, abandon its brittle legacy IT structure, and create a cloud-based technology platform for the future—becoming the region’s leader in cloud-native software development.

Google Cloud is enabling the city’s IT team by curating and delivering our certification training at no cost. The program includes live training sessions as well as on-demand training. 

Bridging funding gaps

In their drive to modernization, many SLG leaders are turning to grants as an important source of funding. Approximately 84% of those surveyed report making tradeoffs to bridge funding gaps, such as moving resources away from operations and maintenance (37%), increasing reliance on pandemic-related funding (31%), and delaying internal modernization efforts to enable remote work for employees (29%). One way that states are dealing with this tension between budget gaps and the need for innovation is to turn to Google Cloud for cost savings and improved capabilities.

For example, Google Cloud is helping the State of West Virginia innovate and enhance IT security despite decreased state funding. The state entered a multi-year agreement to ensure full access to enterprise-level Google Workspace capabilities for 25,000 state employees, keeping the state at the forefront of technology advancements at a projected cost savings of $11.5 million.

Similarly, Google Cloud helped build the Rhode Island Virtual Career Center to help the state’s constituents get back to work. Using familiar productivity tools within Google Workspace, employees can access new career opportunities quickly, while employers can reach more candidates. Skipper, the CareerCompass RI bot, uses data and machine learning to connect Rhode Islanders with potential new career paths and reskilling opportunities.

Enhancing services

Google Cloud is also helping agencies enhance services, including working with the State of Illinois to get unemployment funding to constituents in need.The state is using Contact Center AI to rapidly deploy virtual agents that help more than 1 million out of work citizens file unemployment claims faster. Capable of engaging in human-like conversations, these intelligent agents provide constituents with 24/7 access and enable government employees to focus on more complex, mission-critical tasks—such as combating fraud. In summer 2020, the virtual agents handled more than 140,000 phone and web inquiries per day, including 40,000 after-hours calls every night. The state anticipates an estimated annual savings of $100 million from the solution, which was deployed in just two weeks.

Working with Google, Ohio also uncovered $2 billion in fraudulent unemployment claims. We will continue to partner with the state to find fraudulent claims, and prioritize the processing of legitimate claims.

 Focusing on cybersecurity

Despite expanding security threats topping NASCIO’s list of 2021 State CIO priorities, more than one in three IT managers (35%) say their organization reduces security measures to expedite timelines. Partnering with Google Cloud has enabled many agencies to enhance their security measures while modernizing and staying within budget, investing in support for remote work devices, digital services for residents, and cybersecurity.

NYC Cyber Command works with city agencies to ensure systems are designed, built, and operated in a highly secure manner. NYC3 followed a cloud-first strategy using the Google Cloud Platform. The virtual operations demanded by the pandemic have increased the importance of security and compliance in SLG. Google Cloud is committed to act as a security transformation partner and be the trusted cloud for public sector agencies. 

Finally, to strengthen public and private partnerships, SLG organizations told MeriTalk that they need vendor partners to support modernization efforts for flexibility and collaboration (46%), need innovation-focused leadership groups to help balance technology needs with budget constraints (41%), and they expect significant returns on investments in cloud computing (38%), and data management/analytics (33%).

Google Cloud is helping SLG customers across the country invest in innovation to walk the tech tightrope—balancing innovation and budgets—and helping to build a more resilient future. Visit the State and Local Government solutions page to learn more.