The value of streaming analytics comes from the insights a business draws from instantaneous data processing, and the timely responses it can implement to adapt its product or service for a better customer experience. “Instantaneous data insights,” however, is a concept that varies with each use case. Some businesses optimize their data analysis for speed, while others optimize for execution cost. 

In this post, we’ll offer some tips on estimating the cost of a job in Dataflow, Google Cloud’s fully managed streaming and batch analytics service. Dataflow provides the ability to optimize a streaming analytics job through its serverless approach to resource provisioning and management. It automatically partitions your data and distributes your worker code to Compute Engine instances for parallel processing, optimizes potentially costly operations such as data aggregations, and provides on-the-fly adjustments with features like autoscaling and dynamic work rebalancing. 

The flexibility that Dataflow’s adaptive resource allocation offers is powerful; it takes away the overhead of estimating workloads to avoid paying for unutilized resources or causing failures due to the lack of processing capacity. Adaptive resource allocation can give the impression that cost estimation is unpredictable too. But it doesn’t have to be. To help you add predictability, our Dataflow team ran some simulations that provide useful mechanisms you can use when estimating the cost of any of your Dataflow jobs. 

The main insight we found from the simulations is that the cost of a Dataflow job increases linearly when sufficient resource optimization is achieved. Under this premise, running small load experiments to find your job’s optimal performance provides you with a throughput factor that you can then use to extrapolate your job’s total cost. At a high level, we recommend following these steps to estimate the cost of your Dataflow jobs: 

1. Design small load tests that help you reach 80% to 90% of resource utilization
2. Use the throughput of this pipeline as your throughput factor
3. Extrapolate your throughput factor to your production data size and calculate the number of workers you’ll need to process it all
4. Use the Google Cloud Pricing Calculator to estimate your job cost

This mechanism works well for simple jobs, such as a streaming job that moves data from Pub/Sub to BigQuery or a batch job that moves text from Cloud Storage to BigQuery. In this post, we will walk you through the process we followed to prove that throughput factors can be linearly applied to estimate total job costs for Dataflow.  

Finding the throughput factor for a streaming Dataflow job

To calculate the throughput factor of a streaming Dataflow job, we selected one of the most common use cases: ingesting data from Google’s Pub/Sub, transforming it using Dataflow’s streaming engine, then pushing the new data to BigQuery tables. We created a simulated Dataflow job that mirrored a recent client’s use case, which was a job that read 10 subscriptions from Pub/Sub as a JSON payload. Then, the 10 pipelines were flattened and pushed to 10 different BigQuery tables using dynamic destinations and BigQueryIO, as shown in the image below.

The team ran 11 small load tests for this job. The first few tests were focused on finding the job’s optimal throughput and resource allocation to calculate the job’s throughput factor. 

For the tests, we generated messages in Pub/Sub that were 500 KB on average, and we adjusted the number of messages per topic to obtain the total loads to feed each test. We tested a range of loads from 3MB/s to 250MB/s. The table below shows five of the most representative jobs with their adjusted parameters:

In order to ensure maximum resource utilization, we monitored the backlog of each test using the backlog graph in the Dataflow interface. We recommend targeting an 80% to 90% utilization so that your pipeline has enough capacity to handle small load increases. We considered 86% to 91% of CPU utilization to be our optimal utilization. In this case, it meant a 2.5MB/s per virtual CPU (vCPU) load. This is job #4 on the table above. In all tests, we used n1-standard-2 machines, which are the recommended type for streaming jobs and have two vCPUs. The rest of the tests were focused on proving that resources scale linearly using the optimal throughput, and we confirmed it.

Using the throughput factor to estimate the approximate total cost of a streaming job

Let’s assume that our full-scale job runs with a throughput of 1GB/s and runs five hours per month. Our throughput factor estimates that 2.5MB/s is the ideal throughput per worker using the n1-standard-2 machines. To support a 1GB/s throughput, we’ll need approximately 400 workers, so 200 n1-standard-2 machines.

We entered this data in the Google Cloud Pricing Calculator and found that the total cost of our full-scale job is estimated at $166.30/month. In addition to worker costs, there is also the cost of streaming data processed when you use the streaming engine. This data is priced by volume measured in gigabytes, and is typically between 30% to 50% of the worker costs. For our use case, we took a conservative approach and estimated 50%, totaling $83.15 per month. The total cost of our use case is $249.45 per month.

Finding the throughput factor for a simple batch Dataflow job

The most common use case in batch analysis using Dataflow is transferring text from Cloud Storage to BigQuery. Our small load experiments read a CSV file from Cloud Storage and transformed it into a TableRow, which was then pushed into BigQuery in batch mode. The source was split into 1 GB files. We ran tests with file sizes from 10GB to 1TB to demonstrate that optimal resource allocation scales linearly. Here are the results of these tests:

These tests demonstrated that batch analysis applies autoscaling efficiently. Once your job finds an optimized resource utilization, it scales to allocate the resources needed to complete the job with a consistent price per unit of processed data in a similar processing time.

Let’s assume that our real scale job here processes 10TB of data, given that our estimated cost using resources in us-central1 is about $0.0017/GB of processed data. The total cost of our real scale job would be about $18.06. This estimation follows this equation: cost(y) = cost(x) * Y/X, where cost(x) is the cost of your optimized small load test, X is the amount of data processed in your small load test, and Y is the amount of data processed in your real scale job. 

The key in this and the previous examples is to design small-load experiments to find your optimized pipeline setup. This setup will give you the parameters for a throughput factor that you can scale to estimate the resources needed to run your real scale job. You can then input these resource estimations in the Pricing Calculator to calculate your total job cost. 

Like countless other industries, farming is going digital and undergoing big changes—driven by access to more actionable information. The agriculture business can now gather and analyze georeferenced data from satellites, combined with data from IoT sensors in fields, crop rotation and yield histories, weather patterns, seed genotypes and soil composition to help increase the quantity and quality of crops.

This is essential for businesses in the agriculture industry, but it’s also critical to address growing food shortages around the world. 

At S4, we create technology to de-risk crop production. We provide customers seeking agricultural risk management solutions with the tools to make better, data-driven decisions for their crop planning, based on machine learning and proprietary algorithms.

We interpret plant evolution on a global scale with predictive modeling and analytics, and offer super-efficient risk-transferring solutions. Our multi-cloud platform includes a petabyte-scale database, an open source stack, and—after 50 proof-of-concept evaluations—BigQuery for our data warehouse and the Cloud SQL database service to handle OLTP queries to our PostgreSQL database.

These PoCs included, among others, Microsoft Azure Data Lake Analytics, IBM Netezza, Postgres/PostGIS running on IBM bare-metal servers with SATA SSDs and on Google’s Compute Engine with NVMe disks, and on-premises memSQL, CitusData and Yandex ClickHouse. 

Weeding out risk in an uncertain market

According to recent research, climate extreme events like drought, heat waves, and heavy precipitation are responsible for 18-43% of global variation in crop yields for maize, spring wheat, rice, and soybeans. This is a clear trend for other crops as well. Such variation poses risks of food shortages as well as large financial risks to farmers, insurers, and regions dependent on successful crop yields. Also, it creates vast humanitarian difficulties.

Our mission at S4 is to help de-risk crop production by matching the right data with analytics tools so farmers and other participants in the agricultural value chain can plan better, resulting in more reliable food supplies.

In a nutshell, we create indices out of biological assets. These indices measure yield losses on crops that are caused by the effects of weather and other factors, which are then used as underlying assets for products, such as swap/derivative contracts and parametric insurance policies, to transfer risk to the financial markets.

We enable insurers and lenders to buy and sell agricultural risks through the futures market. Also, our other products help farmers and seed and fertilizer companies provide customized genotype recommendations and fertilization requirements. This helps to optimize planting by geography, resources, and crop species, monitor phenological, pests and humidity evolution throughout the crop season, and estimate yields.

Local communities benefit from S4’s technology, as the ability to manage weather risks allows farmers to stabilize their cash flows, invest more to produce more with fewer risks, and develop in a more sustainable manner.

Growing data sources, reducing costs, accelerating performance

With the volume of diverse data sources and analytical complexity both growing at a very fast pace, we decided that using a major cloud services provider with a broad roadmap and global partnerships would be beneficial to S4’s future evolution.

At the same time, we wanted to bring our services to users faster and cut costs by consolidating our on-premises technology stack. When we started evaluating providers, our leading criteria included a powerful geospatial database and data analytics tools along with excellent support, all at a competitive price. GCP prevailed in nearly all criteria categories among the 50 companies we measured. 

Our previous platform architecture included a hybrid relational database that used Compute Engine for virtual machines and Cloud Storage for database backup. The RDBMS was slow. Maintaining our own data warehouse was complex and expensive.

We wanted to use machine learning and neural networks, but couldn’t do so easily and affordably. The complexity of that system meant that products or services requiring small changes or additions to the data model translated to expensive expansions of infrastructure or project time.

Also, agronomical or product teams couldn’t test these changes by themselves, always requiring the intervention on no small part of the IT team, which led to further delays.

We added GCP services like BigQuery as S4’s cloud data warehouse and use BigQuery GIS for geospatial analysis, Cloud Dataflow for simplified stream and batch data processing, and Cloud SQL for queries to the S4 database platform, which have all made a huge impact on our services and bottom line.

Database and analytics costs have decreased by 40% and customers are receiving our analytical results 25% faster. In addition, we’ve eliminated the time-consuming downloading of images, reducing storage and processing costs by 80%, because we no longer need expensive tools licenses, and have greatly reduced classification processing times.

Our customers working in the agriculture industry are also benefiting from this infrastructure change. They are now able to speed up their data analytics using our GCP-based platform.

“S4 products and technologies unlock the full potential of satellite imagery for crop prescriptions, monitoring and yield estimates,” says Nicolás Loria, Manager of Marketing Services, Southern Cone, Corteva Agriscience.

“We’ve worked with S4 for the last three (and starting year number four) crop seasons as its team capabilities, data integration capacities, and analytics insights have allowed Corteva to perform an entire new solution. Thanks to S4’s customized 360° approach, fast response and delivery times, we have safely outsourced our remote crop analytic technical needs.”

Also, this new architecture has allowed us to scale our models and databases with almost no limits, at a fraction of the cost vs. the previous models.

We’ve saved a lot of time on executing processes and reduced work needed by our internal teams to do certain tasks, like preparing images, converting them, validating results, and more. Using Google Earth Engine has decreased the execution time of daily tasks anywhere from 50% to 90% of the previous time, going from an average time of 30 minutes to between four and 15 minutes, depending on the task.

In addition to saving money and time, we are able to focus on innovation with the GCP performance and features we’re using. We’re able to seamlessly add satellite data to analytics using both public datasets and our own private data, and deliver GIS data management, analytics, crop classification and monitoring in real time.

We can do semi-automatic crop classification and classification using spectral signatures with Google Earth Engine. Later this year, we’ll be using neural networks for pattern recognition and machine learning in new applications to improve crop yields and fine-tune risk models. And using GCP and Google Earth Engine infrastructure means we can run models for customers in South America and around the world, since Google Earth Engine has global satellite imagery available. 

We’ve heard from our customer Indigo Argentina that they’re able to bring customers data insights faster.

“We are working with S4 in the development of two different applications for satellite crop monitoring and yield assessment,” says Carlos Becco, CEO, Indigo Argentina. “S4’s technology allowed us to manage and analyze multiple sources and layers of information in real time, letting us uncover valuable insights in Indigo’s own microbiome technologies, and at a very competitive cost.” 

Analytical products and app development thrive with GCP

With GCP, we are updating and improving algorithms that we built manually with machine learning processes to develop drought indices for upcoming crop seasons. Algorithms can recognize specific phases of crop phenology (e.g., bud burst, flowering, fruiting, leaf fall) and correlate them with photosynthetic activity, light, water, temperature, radiation, and plant genetics factors. Other analytical products like crop monitoring, pre-planting recommendations, financial scoring, and yield estimation can now do a lot more for users by offering multiple layers and datasets, faster image processing, and real-time access via APIs.

We also replaced our bare-metal S4 app deployment with the App Engine serverless application platform. It provides tighter integration between the S4 platform and our BigQuery data warehouse for integration with marketplaces and third-party solutions.

We get all of these Google Cloud features with all the benefits of managed cloud services, from multiversioning and security to automatic backups and high availability.

At S4, we trust technology to decode plant growth and help protect farmers and their communities from climate change. With growing food shortages due to increasing populations and intensifying weather, data and analytics can have a huge impact in lowering financial risks and improving agricultural yields. It’s one sector where cloud, database, analytics, and other technologies are combining to improve business outcomes and affect the lives of billions of people. Learn more about S4’s work and learn more about data analytics on Google Cloud.

In the past few years, a series of new technologies and regulatory changes has been transforming Brazil’s financial industry. The concept of blockchain added security and agility to financial transactions. The open banking system being deployed by the country’s Central Bank (BC) allows for platform integration and data sharing -with the user’s consent- between financial institutions. Recently, the launch of Pix, also by the BC, has shaken the market by creating a new payment method that is instant, free for individuals and 24/7.

Since 2018, the startup Bit Capital has been working on the development of 100%-digital financial solutions in the cloud to help its clients adapt to this new scenario in a convenient way, without the need to build an infrastructure for that or hiring different providers. Pix was not the exception.

In less than 3 months, the team was able to build and go live with a solution that can be used by both direct participants (i.e. those with banking licenses granted by BC) and indirect participants (i.e. companies depending on direct participants to offer payments on Pix) through Bit Capital’s platform.

Besides this differential, Pix and the startup’s other solutions are based on blockchain and Google Cloud’s cloud, ensuring greater security, scalability and availability for customers and allowing for an easy integration between Bit Capital’s platform’s solutions and those from other companies.

“Now we have a platform with an API and various microsservices, allowing clients to connect and develop their own financial product without having to start from scratch.”

—Francesco Miolo, CFO, Bit Capital

A robust structure to handle Pix’s high demand

Bit Capital was established with a certain urgency to develop its platform. The idea was well-developed and customers were interested. The fast deployment of Google Cloud’s tools was one of the main factors that attracted the company to use it as a basis for its infrastructure.

“We managed to deliver everything we have today in Google Cloud with a small team. That was another challenge: being able to grow with a few people,” says Juliano Souza, the startup’s head of IT infrastructure. “We sought other cloud partners, but they had a steep curve. We chose Google Cloud because we needed quick, quality scaling.”

The same thing happened when they built the solution for Pix, despite the specific challenges involved. The BC had performance requirements that led the company to spend some time experimenting until they reached the best suite of tools to meet those requirements. The startup wanted to create a unique architecture that could help both large and small customers and be integrated to the platform’s other microservices.

With the support from Google Cloud’s team to answer doubts and make the best decisions, the solution was built in a few months, at a cost about two-thirds lower than other providers.

The solution’s architecture is based on apps running in Docker in Google Kubernetes Engine (GKE), with interconnected microservices. The blockchain system runs on Compute Engine, its rows are managed in Pub/Sub and Dataflow, and persistent data, in Cloud SQL. Cloud Interconnect is used for the connection with BC. Cloud KMS and Secrets Management store the company’s pre-credentials. All of this is supported by Cloud Load Balancing for load balancing and autoscaling.

According to the team, GKE was essential for the solution’s success. Using infrastructure as code in the tool made uploading of a group of microservices significantly easier. Developers are able to help set up this environment, which has helped spread the DevOps culture in the team. Besides orchestrating and integrating apps, GKE also provides an elastic structure to handle demand peaks and visibility to monitor internal components.

“Google Kubernetes Engine was the only way to ensure availability, observability and elasticity for all clients, whether small, middle-sized or large.”

—Juliano Souza, Head of IT Infrastructure, Bit Capital

Nowadays, the solution’s environment has 20 clusters with over 1,500 pods – 250GB in data per month, providing a robust structure to support a service involving periods of intense demand such as Black Friday.

Ease to monitor, fix and improve

Just 10 days after Pix’s official launch date, the company had its first test: Black Friday. This allowed a major e-commerce customer to test the scale and see the success of the architecture that had been built. “It worked great regarding what the cloud could deliver. And we found what we needed to fix very quickly. Operations [formerly Stackdriver], in particular Cloud Trace, showed us clearly what needed to be done to improve performance,” Souza explains.

Using Operations added reliability by putting deliveries into production. Checking Cloud Trace to see if there were any performance issues and the exact point where they were happening became routine for Bit Capital’s developers. Google Cloud’s security tools and Google Safety Center provided the resources needed to monitor and secure the environment, with automatic data encryption at rest and in transit.

“Google Cloud has security as a premise. When we upload any kind of component, like a database or a virtual machine, the drive is encrypted by default. With other providers, you must specify that you want it encrypted.

”—Juliano Souza, Head of IT Infrastructure, Bit Capital

The easy service monitoring and management accelerates product and technology development because the team no longer needs to worry about infrastructure. Automated management allows professionals to spend more time on new projects and the company’s business. Also, the deployment of services in a Google Cloud multi-region impacts on customer experience, providing high availability for their solutions.

In the coming months, Bit Capital aims to ramp up service usage and the creation of new projects in Google Cloud by adding more customers to Pix’s solution and Banking as a Service (BaaS) solutions. Anthos will be incorporated to the tool suite to make it easier to connect apps with the customers’ on-prem environments. And the deployment of open banking has the potential to be a business driver for the company.

“We joke that we’re already doing open banking, because we have various connections with different providers, which allows us to offer an integrated solution,” says Francesco Miolo, the startup’s CFO. “With the arrival of the new regulations, something we are waiting for since we started out, we will be able, through our platform, to take our customers to the open finance ecosystem, enabling the development of disruptive business models.”