Like countless other industries, farming is going digital and undergoing big changes—driven by access to more actionable information. The agriculture business can now gather and analyze georeferenced data from satellites, combined with data from IoT sensors in fields, crop rotation and yield histories, weather patterns, seed genotypes and soil composition to help increase the quantity and quality of crops.

This is essential for businesses in the agriculture industry, but it’s also critical to address growing food shortages around the world. 

At S4, we create technology to de-risk crop production. We provide customers seeking agricultural risk management solutions with the tools to make better, data-driven decisions for their crop planning, based on machine learning and proprietary algorithms.

We interpret plant evolution on a global scale with predictive modeling and analytics, and offer super-efficient risk-transferring solutions. Our multi-cloud platform includes a petabyte-scale database, an open source stack, and—after 50 proof-of-concept evaluations—BigQuery for our data warehouse and the Cloud SQL database service to handle OLTP queries to our PostgreSQL database.

These PoCs included, among others, Microsoft Azure Data Lake Analytics, IBM Netezza, Postgres/PostGIS running on IBM bare-metal servers with SATA SSDs and on Google’s Compute Engine with NVMe disks, and on-premises memSQL, CitusData and Yandex ClickHouse. 

Weeding out risk in an uncertain market

According to recent research, climate extreme events like drought, heat waves, and heavy precipitation are responsible for 18-43% of global variation in crop yields for maize, spring wheat, rice, and soybeans. This is a clear trend for other crops as well. Such variation poses risks of food shortages as well as large financial risks to farmers, insurers, and regions dependent on successful crop yields. Also, it creates vast humanitarian difficulties.

Our mission at S4 is to help de-risk crop production by matching the right data with analytics tools so farmers and other participants in the agricultural value chain can plan better, resulting in more reliable food supplies.

In a nutshell, we create indices out of biological assets. These indices measure yield losses on crops that are caused by the effects of weather and other factors, which are then used as underlying assets for products, such as swap/derivative contracts and parametric insurance policies, to transfer risk to the financial markets.

We enable insurers and lenders to buy and sell agricultural risks through the futures market. Also, our other products help farmers and seed and fertilizer companies provide customized genotype recommendations and fertilization requirements. This helps to optimize planting by geography, resources, and crop species, monitor phenological, pests and humidity evolution throughout the crop season, and estimate yields.

Local communities benefit from S4’s technology, as the ability to manage weather risks allows farmers to stabilize their cash flows, invest more to produce more with fewer risks, and develop in a more sustainable manner.

Growing data sources, reducing costs, accelerating performance

With the volume of diverse data sources and analytical complexity both growing at a very fast pace, we decided that using a major cloud services provider with a broad roadmap and global partnerships would be beneficial to S4’s future evolution.

At the same time, we wanted to bring our services to users faster and cut costs by consolidating our on-premises technology stack. When we started evaluating providers, our leading criteria included a powerful geospatial database and data analytics tools along with excellent support, all at a competitive price. GCP prevailed in nearly all criteria categories among the 50 companies we measured. 

Our previous platform architecture included a hybrid relational database that used Compute Engine for virtual machines and Cloud Storage for database backup. The RDBMS was slow. Maintaining our own data warehouse was complex and expensive.

We wanted to use machine learning and neural networks, but couldn’t do so easily and affordably. The complexity of that system meant that products or services requiring small changes or additions to the data model translated to expensive expansions of infrastructure or project time.

Also, agronomical or product teams couldn’t test these changes by themselves, always requiring the intervention on no small part of the IT team, which led to further delays.

We added GCP services like BigQuery as S4’s cloud data warehouse and use BigQuery GIS for geospatial analysis, Cloud Dataflow for simplified stream and batch data processing, and Cloud SQL for queries to the S4 database platform, which have all made a huge impact on our services and bottom line.

Database and analytics costs have decreased by 40% and customers are receiving our analytical results 25% faster. In addition, we’ve eliminated the time-consuming downloading of images, reducing storage and processing costs by 80%, because we no longer need expensive tools licenses, and have greatly reduced classification processing times.

Our customers working in the agriculture industry are also benefiting from this infrastructure change. They are now able to speed up their data analytics using our GCP-based platform.

“S4 products and technologies unlock the full potential of satellite imagery for crop prescriptions, monitoring and yield estimates,” says Nicolás Loria, Manager of Marketing Services, Southern Cone, Corteva Agriscience.

“We’ve worked with S4 for the last three (and starting year number four) crop seasons as its team capabilities, data integration capacities, and analytics insights have allowed Corteva to perform an entire new solution. Thanks to S4’s customized 360° approach, fast response and delivery times, we have safely outsourced our remote crop analytic technical needs.”

Also, this new architecture has allowed us to scale our models and databases with almost no limits, at a fraction of the cost vs. the previous models.

We’ve saved a lot of time on executing processes and reduced work needed by our internal teams to do certain tasks, like preparing images, converting them, validating results, and more. Using Google Earth Engine has decreased the execution time of daily tasks anywhere from 50% to 90% of the previous time, going from an average time of 30 minutes to between four and 15 minutes, depending on the task.

In addition to saving money and time, we are able to focus on innovation with the GCP performance and features we’re using. We’re able to seamlessly add satellite data to analytics using both public datasets and our own private data, and deliver GIS data management, analytics, crop classification and monitoring in real time.

We can do semi-automatic crop classification and classification using spectral signatures with Google Earth Engine. Later this year, we’ll be using neural networks for pattern recognition and machine learning in new applications to improve crop yields and fine-tune risk models. And using GCP and Google Earth Engine infrastructure means we can run models for customers in South America and around the world, since Google Earth Engine has global satellite imagery available. 

We’ve heard from our customer Indigo Argentina that they’re able to bring customers data insights faster.

“We are working with S4 in the development of two different applications for satellite crop monitoring and yield assessment,” says Carlos Becco, CEO, Indigo Argentina. “S4’s technology allowed us to manage and analyze multiple sources and layers of information in real time, letting us uncover valuable insights in Indigo’s own microbiome technologies, and at a very competitive cost.” 

Analytical products and app development thrive with GCP

With GCP, we are updating and improving algorithms that we built manually with machine learning processes to develop drought indices for upcoming crop seasons. Algorithms can recognize specific phases of crop phenology (e.g., bud burst, flowering, fruiting, leaf fall) and correlate them with photosynthetic activity, light, water, temperature, radiation, and plant genetics factors. Other analytical products like crop monitoring, pre-planting recommendations, financial scoring, and yield estimation can now do a lot more for users by offering multiple layers and datasets, faster image processing, and real-time access via APIs.

We also replaced our bare-metal S4 app deployment with the App Engine serverless application platform. It provides tighter integration between the S4 platform and our BigQuery data warehouse for integration with marketplaces and third-party solutions.

We get all of these Google Cloud features with all the benefits of managed cloud services, from multiversioning and security to automatic backups and high availability.

At S4, we trust technology to decode plant growth and help protect farmers and their communities from climate change. With growing food shortages due to increasing populations and intensifying weather, data and analytics can have a huge impact in lowering financial risks and improving agricultural yields. It’s one sector where cloud, database, analytics, and other technologies are combining to improve business outcomes and affect the lives of billions of people. Learn more about S4’s work and learn more about data analytics on Google Cloud.

Without AI, you’re not getting the most out of your data.
Without data, you risk stale, out-of-date, suboptimal models.

But most companies are still struggling with how to keep these highly interdependent technologies in sync and operationalize AI to take meaningful action from data.

We’ve learned from Google’s years of experience in AI development how to make data-to-AI workflows as cohesive as possible and as a result our data cloud is the most complete and unified data and AI solution provider in the market. By bridging data and AI, data analysts can take advantage of user-friendly, accessible ML tools, and data scientists can get the most out of their organization’s data. All of this comes together with built-in MLOps to ensure all AI work — across teams — is ready for production use.

In this blog we’ll show you how all of this works, including exciting announcements from the Data Cloud Summit:

Vertex AI Workbench is now GA bringing together Google Cloud’s data and ML systems into a single interface so that teams have a common toolset across data analytics, data science, and machine learning. With native integrations across BigQuery, Spark, Dataproc, and Dataplex data scientists can build, train and deploy ML models 5X faster than traditional notebooks.

Introducing Vertex AI Model Registry, a central repository to manage and govern the lifecycle of your ML models. Designed to work with any type of model and deployment target, including BigQuery ML, Vertex AI Model Registry makes it easy to manage and deploy models.

Use ML to get the most out of your data, no matter the format
Analyzing structured data in a data warehouse, like using SQL in BigQuery, is the bread and butter for many data analysts. Once you have data in a database, you can see trends, generate reports, and get a better sense of your business. Unfortunately, a lot of useful business data isn’t in the tidy tabular format of rows and columns. It’s often spread out over multiple locations and in different formats, frequently as so-called “unstructured data” — images, videos, audio transcripts, PDFs — can be cumbersome and difficult to work with.

Here, AI can help. ML models can be used to transcribe audio and videos, analyze language, and extract text from images—that is, to translate elements of unstructured data into a form that can be stored and queried in a database like BigQuery. Google Cloud’s Document AI platform, for example, uses ML to understand documents like forms and contracts. Below, you can see how this platform is able to intelligently extract structured text data from an unstructured document like a resume. Once this data is extracted, it can be stored in a data warehouse like BigQuery.

Bring machine learning to data analysts via familiar tools

Today, one of the biggest barriers to ML is that the tools and frameworks needed to do ML are new and unfamiliar. But this doesn’t have to be the case. BigQuery ML, for example, allows you to train sophisticated ML models at scale using SQL code, directly from within BigQuery. Bringing ML to your data warehouse alleviates the complexities of setting up additional infrastructure and writing model code. Anyone who can write SQL code can train a ML model quickly and easily.

Easily access data with a unified notebook interface

One of the most popular ML interfaces today are notebooks: interactive environments that allow you to write code, visualize and pre-process data, train models, and a whole lot more. Data scientists often spend most of their day building models within notebook environments. It’s crucial, then, that notebook environments have access to all of the data that makes your organization run, including tools that make that data easy to work with.

Vertex AI Workbench, now generally available, is the single development environment for the entire data science workflow. Integrations across Google Cloud’s data portfolio allow you to natively analyze your data without switching between services:

Cloud Storage: access unstructured data

BigQuery: access data with SQL, take advantage of models trained with BigQuery ML

Dataproc: execute your notebook using your Dataproc cluster for control

Spark: transform and prepare data with autoscaling serverless Spark

Below, you’ll see how you can easily run a SQL query on BigQuery data with Vertex AI Workbench.

But what happens after you’ve trained the model? How can both data analysts and data scientists make sure their models can be utilized by application developers and maintained over time?

Go from prototyping to production with MLOps

While training accurate models is important, getting those models to be scalable, resilient, and accurate in production is its own art, known as MLOps. MLOps allow you to:

• Know what data your models are trained on
• Monitor models in production
• Make training process repeatable
• Serve and scale model predictions
• A whole lot more! (See the “Practitioners Guide to MLOps” whitepaper for a full and detailed overview of MLOps)

Built-in MLOps tools within Vertex AI’s unified platform remove the complexity of model maintenance. Practical tools can help with everything from training and hosting ML models, managing model metadata, governance, model monitoring, and running pipelines – all critical aspects of running ML in production and at scale.

And now, we’re extending our capabilities to make MLOps accessible to anyone working with ML in your organization.

Easy handoff to MLOps with Vertex AI Model Registry

Today, we’re announcing Vertex AI Model Registry, a central repository that allows you to register, organize, track, and version trained ML models and is designed to work with any type of model and deployment target, whether that’s through BigQuery, Vertex AI, AutoML, custom deployments on GCP or even out of the cloud.

Vertex AI Model Registry is particularly beneficial for BigQuery ML. While BigQuery ML brings the powerful scalability of BigQuery for batch predictions, using a data warehouse engine for real-time predictions just isn’t practical. Furthermore, you might start to wonder how to orchestrate your ML workflows based in BigQuery. You can now discover and manage BigQuery ML models and easily deploy those models to Vertex AI for real-time predictions and MLOps tools.

End-to-End MLOps with pipelines

One of the most popular approaches to MLOps is the concept of ML pipelines: where each distinct step in your ML workflow from data preparation to model training and deployment are automated for sharing and reliably reproducing.

Vertex AI Pipelines is a serverless tool for orchestrating ML tasks using pre-built components or your own custom code. Now, you can easily process data and train models with BigQuery, BigQuery ML, and Dataproc directly within a pipeline. With this capability, you can combine familiar ML development within BigQuery and Dataproc into reproducible, resilient pipelines and orchestrate your ML workflows faster than ever.

See an example of how this works with the new BigQuery and BigQuery ML components.

Learn more and get started

We’re excited to share more about our unified data and AI offering today at the Data Cloud Summit. Please join us for the spotlight session on our “AI/ML strategy and product roadmap” or the “AI/ML notebooks ‘how to’ session.”

And if you’re ready to get hands on with Vertex AI, check out these resources:

Codelab: Training an AutoML model in Vertex AI

Codelab: Intro to Vertex AI Workbench

Video Series: AI Simplified: Vertex AI

GitHub: Example Notebooks

Training: Vertex AI: Qwik Start

The Home Depot (THD) is the world’s largest home-improvement chain, growing to more than 2,200 stores and 700,000 products in four decades. Much of that success was driven through the analysis of data. This included developing sales forecasts, replenishing inventory through the supply chain network, and providing timely performance scorecards.

However, to compete in today’s business world, THD has taken this data-driven approach to an entirely new level of success on Google Cloud, providing capabilities not practical on legacy technologies.

The pressures of contemporary growth that drove much of the work are familiar to many businesses. In addition to everything it was doing, THD needed to better integrate the complexities in its related businesses, like tool rental and home services. It needed to better empower teams, including a fast-growing data analysis staff and store associates with mobile computing devices. It wanted to better use online commerce and artificial intelligence to meet customer needs, while maintaining better security.

Even before addressing these new challenges, THD’s existing on-premises data warehouse was under stress as more data was required for analytics and data analysts were utilizing the data with increasingly complex use cases. This drove rapid growth of the data warehouse, but also created constant challenges for the team in managing priorities, performance, and cost.

In order to add capacity to the environment, it was a major planning, architecture, and testing effort. In one case, adding on-premises capacity took six months of planning and a three-day service outage. Within a year, capacity was again scarce, impacting performance and ability to execute all the reporting and analytics workloads required. The capacity refresh cycles were shrinking, and the expecations for data were growing. There had to be a better way.

Still, THD did not take its move to the cloud lightly. A large-scale enterprise data warehouse migration involves tremendous effort among people, process, and technology. After careful consideration, THD chose Google Cloud’s BigQuery for its cloud enterprise data warehouse.

BigQuery, a scalable serverless data warehouse, was better on cost, infrastructure agility, and analytics capability, driving better insights with improved performance. There are no service interruptions when capacity is added, and that capacity can be added within a week (and soon same day). It doesn’t require complex system administration, and its standard SQL support means people can easily ramp up quickly. Valuable BigQuery products like Identity and Access Management meant THD could create many separate Google Cloud projects, while ensuring that different teams weren’t interfering with each other or accessing protected data.

THD also utilizes BigQuery’s flat-rate monthly pricing model that allows teams to budget their capacity based on need and provides billing predictability. The capacity not being used by a given project is available for enterprise use. This ensures no surprises when the monthly bill arrives and provides all analytical users access to significant computing power.

While THD’s legacy data warehouse contained 450 terabytes of data, the BigQuery enterprise data warehouse has over 15 petabytes. That means better decision-making by utilizing new datasets like website clickstream data and by analyzing additional years of data.

As for performance, look at this chart:

With the cloud EDW migration complete, and the legacy on-premises data warehouse retired, analysts now execute more complex and demanding workloads that they would not have been able to complete before, such as utilizing Datalab for orchestrating analytics through Python Notebooks, utilizing BigQuery ML for machine learning directly against the BigQuery data (no movement of large datasets), and AutoML to help determine the best model for predictions.

Additionally, engineers at THD have adapted BigQuery to monitor, analyze, and act on application performance data across all its stores and warehouses in real time, something that was not practical in the on-premises system.

With over 600 projects that THD now has on Google Cloud, the BigQuery story is just one of the many ways that Google Cloud is working with THD to deliver meaningful business results, every day.

When I was growing up in Thailand, I witnessed the challenges facing farmers including rising food demand, shortage of labor, and uneven crop yields caused by climate change. As a result, many smallholder farmers found themselves trapped in a vicious circle, unable to reduce food insecurity due to low yields, but lacking the resources to invest for a more profitable future.

I was determined to make a difference. After earning my master’s degree in Information Management I joined a research project with the University of Tokyo where we used sensors to monitor spinach fields in Thailand. The results of this early experiment in precision farming were impressive. We proved that it was possible to grow organic crops with minimal use of fertilizers, while consumers benefited from higher quality spinach grown in Thailand.  

This inspired me to found ListenField in 2017. Our mission is to transform farm management by collecting data from multiple sources, including field sensors, soil scanning, weather data, seasonal forecasts, and satellite imagery. By modeling this data, we provide farmers with insights that enable them to optimize production ‘from soil to harvest’.

A bumper crop of farming data

Artificial intelligence and machine learning play a central role in our prediction platform, combining crop health monitoring, growth prediction, and soil nutrition analysis. Farmers can apply real-time insights to their schedule from our FarmAI Mobile App, while our FarmAI Dashboard enables agri-food businesses to collaborate with agronomists and farmers so that all parties benefit from higher profit margins and more sustainable growing strategies.

Another important feature of our business model is AgroAPI, which makes our analytics available to third parties. Clients can embed deep analytics in their applications, including crop growth prediction and remote sensing analysis, without needing to develop complicated algorithms and data pipelines by themselves.

We are also excited about our research into genomic prediction in collaboration with the Japanese government and several research companies. Using our Data-Driven Breeding Platform, breeders and seed companies can upload their genomic data and gain practical insights that help accelerate the reproduction of high-quality seeds and plants.

Today, more than 30,000 farmers use our technology especially in Vietnam and Thailand where it is used to improve rice, cassava, and sugar cane yields. The technology is also being rolled out for orange and mango farmers enabling them to monitor individual trees and adjust irrigation to improve the sweetness of the fruit at harvest time.

Responding fast to changing conditions

We were using another cloud provider to run our business, but one of the ListenField team members drew our attention to Google Cloud. As well as the technology, we were also attracted by the Google for Startups Cloud Program which provides us with Google Cloud credits that cover our first and second years of Google Cloud usage. We also met our Account Representative, who provides us with training, business and tech support, and Google-wide discounts. It’s great to have a point of contact that can help us on our startup journey and make the most of Google’s resources.

By reducing the pressure on our finances and human resources, we were able to experiment and adapt in response to early experiments. This flexibility also enabled us to demonstrate a compelling business case to new and existing investors.

Our Google Cloud Platform Partner, Navagis, gave us additional momentum thanks to their expertise in mapping and geospatial data. They also played a crucial role in the integration of Google Earth Engine, which we use to map agricultural areas.

We also use Firebase for application development and Google Workspace for team collaboration. Colab and Vertex AI enable us to build, deploy, and scale our machine learning models quickly, ensuring that we remain competitive and attractive to new customers.

Giving female entrepreneurs the opportunity to flourish

Both the Google Cloud team and our colleagues at Navagis helped us to navigate the challenges many early-stage startups face. My background is in science and academia, so I appreciated the business mindset offered by both organizations to help us continue to grow and scale.

Being a female entrepreneur leading a startup can also be tough, but Google Cloud and Navagis helped me to build a strong network, access funding, and make my voice heard. Today, ListenField has several female executives, while 50% of our researchers and many of the farmers on our platform are women.  

Above all, Google Cloud helps us to power an agriculture revolution in south-east Asia. Smallholder farmers can transition from analog to digital farming, improving their yields and reducing waste. The benefits to the economy are also significant including greater food security and reducing the use of industrial fertilizers that generate potent greenhouse gasses.

And that’s just the beginning of what we can do. Our next milestone is to reach 50,000 farmers and cut one million tonnes of greenhouse gas emissions. It sounds ambitious, but with the Google Cloud and Navagis teams behind us, I’m confident that we will reach these targets.

ListenField team members

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.