In our conversations with technology leaders about data-driven transformation using Google Data Cloud –  industry’s leading unified data and AI solution – , one important topic is incorporating continuous intelligence to move from answering questions such as “What has happened? to questions like “What is happening?” and “What might happen?”. The core to this evolution is the need for an underlying data processing that not only provides powerful real-time capabilities for events happening close to origination, but also brings together existing data sources under one unified data platform to enable organizations to draw insights and take actions holistically. Dataflow, Google’s cloud-native data processing and streaming analytics platform, is a key component of any modern data and AI architecture and data transformation journey, along with BigQuery, Google’s internet-scale warehouse with built-in streaming, BI engine and ML; Pub/Sub, a global no-ops event delivery service; and Looker, a modern BI and embedded analytics platform. One of the key evaluation factors is potential economic value of Dataflow to their organization, particularly in the context of engaging other stakeholders is key for many of the leaders that we engage with. So we commissioned Forrester Consulting to conduct a comprehensive study on the impact that Dataflow had on their organization by interviewing actual customers . 

Today we’re excited to share our commissioned study conducted by Forrester Consulting, the Total Economic Impact™ of Google Cloud Dataflow, which allows data leaders to understand and quantify the benefits of Dataflow, and use cases it enables. Forrester conducted interviews with Dataflow customers to evaluate the benefits, costs, and risks of investing in Dataflow across an organization. Based on their interviews, Forrester identified major financial benefits across four different areas: business growth, infrastructure cost savings, data engineer productivity, and administration efficiency. In fact, Forrester found that customers adopting Dataflow can achieve a 55% boost in developer productivity and a 50% reduction in infrastructure costs. In fact, Forrester projects that customers adopting Dataflow can achieve a range of up to 171% Return on Investment (ROI) and a less than six months payback period. Customers can now use figures in the report to compute their own Return on Investment (ROI) and payback period.

“Dataflow is integral to accelerating time-to-market, decreasing time-to-production, reducing time to figure out how to use data for use cases, focusing time on value-add tasks, streamlining ingestion, and reducing total cost of ownership.” – Lead technical architect, CPG

Let’s take a deeper look at the ways that Forrester found that Dataflow can help you achieve your goals and unlock your business potential. 

Benefit #1: Increase data engineer productivity by 55%

Developers can choose among a variety of programming languages to define and execute data workflows. Dataflow also seamlessly integrates with other Google Cloud Platform and open source technologies to maximize value and applicability to a wide variety of use cases. Dataflow streamlined workflows with code reusability,dynamic templates, and the simplicity of a managed service. Engineers trusted pipelines to run correctly and adhere to governance. Data engineers avoided laborious issue-monitoring and remediation tasks that were common in the legacy environments such as poor performance, lack of availability, and failed jobs. Teams valued the language flexibility and open source base.

“Dataflow provided us with ETL replacement that opened limitless potential use cases and enabled us to do smarter data enhancement while data remains in motion.” — Director of data projects, financial services

Benefit #2: Reduce infrastructure costs by up-to 50% for batch and streaming workloads 

Dataflow’s serverless autoscaling and discrete control of job needs, scheduling, and regions eliminated overhead and optimized technology spending. Consolidating global data processing solutions to Dataflow further eliminated excess costs while ensuring performance, resilience, and governance across environments. Dataflow’s unified streaming and batch data platform gives organizations the flexibility to define either workload in the same programming model, run it on the same infrastructure, and manage it from a single operational management tool. 

“Our costs with our cloud data platform using Dataflow are just a fraction of the costs we faced before. Now we only pay for cloud infrastructure consumption because the open source base helps us avoid licensing costs. We spend about $120,000 per year with Dataflow, but we’d be spending millions with our old technologies.” – Lead technical architect, CPG

Benefit #3: Increase top-line revenue by improving customer experience and retention with payback time of < 6 months

Streaming analytics is an essential capability in today’s digital world to gain real-time actionable insights. Likewise, organizations must also have flexible, high- performance batch environments to analyze historical data for building machine learning models, business intelligence, and advanced analytics. Dataflow enabled real-time streaming use cases, improved data enrichment, encouraged data exploration,improved performance and resiliency, reduced errors, increased trust, and eliminated barriers to scale. As a result, organizations provided customers with more accurate, relevant, and in-the-moment data-backed services and insights — boosting customer experience, creating new revenue streams, and improving acquisition, retention, and enrichment.

“It’s already been proven that we are getting more business [with Dataflow] because we can turn around results faster for customers.” – VP of technology, financial services technology

“When we provide data to our customers and partners with Dataflow, we are much more confident in those numbers and can provide accurate data within a minute. Our customers and partners have taken note and commented on this. It’s reduced complaints and prevented churn.” – Senior software engineer, media

Other benefits 

Eliminated administrative overhead and toil

As a cloud-native managed service, all administration tasks such as provisioning, scaling, and updates are automatically handled by Google Cloud. Teams no longer need to manage servers and related software for legacy data processing solutions. Admins also streamlined processes for setting up data sources, adding pipelines, and enforcing governance.

Saved business operations costs for support teams and data end users

Dataflow improved the speed, quality, reliability, and ease of access to data for insights for general business users, saving time and empowering users to drive better data-backed outcomes. It also reduced support inquiry volume while automating manual job creation.

What’s next?

Download the Forrester Total Economic Impact study today to dive deep into the economic impact Dataflow can deliver your organization. We would love to partner with you to explore the potential Dataflow can unlock in your teams. Please reach out to our sales team to start a conversation about your data transformation with Google Cloud.

Google Cloud and Synopsys Inc are partnering to help semiconductor companies drive Electronic Design Automation (EDA) innovation in the cloud to accelerate time to market, and lower costs across the entire semiconductor product life cycle. Synopsys is the industry’s largest provider of EDA technology used in the design and verification of integrated circuits, or semiconductor chips. Bringing Synopsys technology onto Google Cloud enables customers to benefit from scalable cloud bursting and complementary licensing models to help them quickly deploy and scale Synopsys’ EDA tools. Semiconductor companies can significantly accelerate chip design and production, increase designer productivity, and empower innovation in power-performance-area optimization.

Google Cloud Platform has enabled high performance computing workloads, such as those used by today’s semiconductor chip designers, to deliver exceptionally fast results and reduce prototyping from years to months. Google Cloud’s HPC services have delivered highly scalable solutions across multiple industries – from scientific research to autonomous vehicle testing and simulation.

EDA software is a large consumer of high performance computing capacity in the cloud. With the release of Synopsys Cloud bring-your-own-cloud (BYOC) solution on Google Cloud, chip designers can now scale their Google Cloud infrastructure with Synopsys’s leading EDA tools under the flexible FlexEDA pay-per-use model and access unlimited EDA software license availability on-demand by the hour or minute.

The Synopsys Cloud BYOC deployment architecture on Google Cloud is enabled through their unique cloud metering service which uses Google Cloud regional MIGs (Managed Instance Groups) for autoscaling and multi-zone deployment. This enables the service to scale up to meet customer workload demands and scale down to optimize costs. Secrets used by Synopsys Cloud are securely stored in Google Secret Manager and usage data is encrypted using Google Cloud key management, providing customers with a highly secure design environment.

Synopsys Cloud also leverages Google Cloud’s Ops Agent and Operations Suite Dashboards are used to show metric data, alerting policies, and log entries, providing customers with detailed analytics visibility to make better chip design project lifecycle management decisions. The Synopsys Cloud BYOC solution has been validated with EDA workloads scaling out to thousands of cores on Google Cloud, using Google Filestore network file storage during validation to provide the highest throughput performance.

Vikram Bhatia, Head, Synopsys Cloud Product Management, Synopsys said, “With the release of Synopsys Cloud BYOC solution on Google Cloud, we are transforming the way our mutual semiconductor customers can design the chips of the future. Google Cloud has been leading the innovation wave for semiconductors in the cloud and we are excited about being an early adopter in leveraging those innovations for our unique EDA offerings on the cloud.” Customers can evaluate a full featured Synopsys Cloud BYOC environment on Google Cloud for free by signing up at: synopsys.com/cloud.

“Combined with GCP’s unique platform services for AI, security, and shared storage, the Synopsys Cloud BYOC solution creates a compelling package for semiconductor designers who will create the next generation of chips for the world’s insatiable needs” says Simon Floyd, Industry Director, Manufacturing & Transportation, Google Cloud.

Google Cloud provides everything you need, including free Google Cloud credits to get you up and running. Click here to learn more about Semiconductors on Google Cloud.

Editor’s note: Today we are hearing from Jono MacDougall , Principal Software Engineer at Ravelin. Ravelin delivers market-leading online fraud detection and payment acceptance solutions for online retailers. To help us meet the scaling, throughput, and latency demands of our growing roster of large-scale clients, we migrated to Google Cloud and its suite of managed services, including Cloud Bigtable, the scalable NoSQL database for large workloads.

As a fraud detection company for online retailers, each new client brings new data that must be kept in a secure manner and new financial transactions to analyze. This means our data infrastructure must be highly scalable and constantly maintain low latency. Our goal is to bring these new organizations on quickly without interrupting their business. We help our clients with checkout flows, so we need latencies that won’t interrupt that process—a critical concern in the booming online retail sector. 

We like Cloud Bigtable because it can quickly and securely ingest and process a high volume of data. Our software accesses data in Bigtable every time it makes a fraud decision. When a client’s customer places an order, we need to process their full history and as much data as possible about that customer in order to detect fraud, all while keeping their data secure. Bigtable excels at accessing and processing that data in a short time window. With a customer key, we can quickly access data, bring it into our feature extraction process, and generate features for our models and rules. The data stays encrypted at rest in Bigtable, which keeps us and our customers safe.

Bigtable also lets us present customer profiles in our dashboard to our client, so that if we make a fraud decision, our clients can confirm the fraud using the same data source we use.

We have configured our bigtable clusters to only be accessible within our private network and have restricted our pods access to it using targeted service accounts. This way the majority of our code does not have access to bigtable and only the bits that do the reading and writing have those privileges.

We also use Bigtable for debugging, logging, and tracing, because we have spare capacity and it’s a fast, convenient location. 

We conduct load testings against Bigtable.  We started at a low rate of ~10 Bigtable requests per second and we peaked at ~167000 mixed read and write requests per second  at absolute peak. The only intervention that was done to achieve this was pressing a single button to increase the number of nodes in the database. No other changes were made.

In terms of real traffic to our production system, we have seen ~22,000 req/s (combined read/write) on Bigtable in our live environment as a peak within the last 6 weeks.

Migrating seamlessly to Google Cloud 

Like many startups, we started with Postgres, since it was easy and it was what we knew, but we quickly realized that scaling would be a challenge, and we didn’t want to manage enormous Postgres instances. We looked for a kind of key value store, because we weren’t doing crazy JOINS or complex WHERE clauses. We wanted to provide a customer ID and get everything we knew about it, and that’s where key value really shines.  

I used Cassandra at a previous company, but we had to hire several people just for that chore. At Ravelin we wanted to move to managed services and save ourselves that headache. We were already heavy users and fans of BigQuery, Google Cloud’s serverless, scalable data warehouse, and we also wanted to start using Kubernetes. This was five years ago, and though quite a few providers offer Kubernetes services now, we still see Google Cloud at the top of that stack with Google Kubernetes Engine (GKE). We also like Bigtable’s versioning capability that helped with a use case involving upserts. All of these features helped us choose Bigtable.

Migrations can be intimidating, especially in retail where downtime isn’t an option. We were migrating not just from Postgres to Bigtable, but also from AWS to Google Cloud. To prepare, we ran in AWS like always, but at the same time we set up a queue at our API level to mirror every request over to Google Cloud. We looked at those requests to see if any were failing, and confirmed if the results and response times were the same as in AWS. We did that for a month, fine tuning along the way. 

Then we took the big step and flipped a config flag and it was 100% over to Google Cloud. At the exact same time, we flipped the queue over to AWS so that we could still send traffic into our legacy environment. That way, if anything went wrong, we could fail back without missing data. We ran like that for about a month, and we never had to fail back. In the end, we pulled off a seamless, issue-free online migration to Google Cloud.

Flexing Bigtable’s features

For our database structure, we originally had everything spread across rows, and we’d use a hash of a customer ID as a prefix. Then we could scan each record of history, such as orders or transactions. But eventually we got customers that were too big, where the scanning wasn’t fast enough. So we switched and put all of the customer data into one row and the history into columns. Then each cell was a different record, order, payment method, or transaction. Now, we can quickly look up the one row and get all the necessary details of that customer. Some of our clients send us test customers who place an order, say, every minute, and that quickly becomes problematic if you want to pull out enormous amounts of data without any limits on your row size. The garbage collection feature makes it easy to clean up big customers.  

We also use Bigtable replication to increase reliability, atomicity, and consistency. We need strong consistency guarantees within the context of a single request to our API since we make multiple bigtable requests within that scope. So within a request we always hit the same replica of Bigtable and if we have a failure, we retry the whole request. That allows us to make use of the replica and some of the consistency guarantees, a nice little trade-off where we can choose where we want our consistency to live.https://www.youtube.com/embed/0-eH5u7rrQQ?enablejsapi=1&

We also use BigQuery with Bigtable for training on customer records or queries with complicated WHERE clauses. We put the data in Bigtable, and also asynchronously in BigQuery using streaming inserts, which allows our data scientists to query it in every way you can imagine, build models, and investigate patterns and not worry about query engine limitations. Since our Bigtable production cluster is completely separate, doing a query on BigQuery has no impact on our response times. When we were on Postgres many years ago, it was used for both analysis and real time traffic and it was not the optimal solution for us. We also use Elasticsearch for powering text searches for our dashboard.

If you’re using Bigtable, we recommend three features:

• Key visualizer. If we get latency or errors coming back from Bigtable, we look at the key visualizer first. We may have a hotkey or a wide row, and the visualizer will alert us and provide the exact key range where the key lives, or the row in question. Then we can go in and fix it at that level. It’s useful to know how your data is hitting Bigtable and if you’re using any anti-patterns or if your clients have changed their traffic pattern that exacerbated some issue.
• Garbage collection. We can prevent big row issues by putting size limits in place with the garbage collection policies.  
• Cell versioning. Bigtable has a 3d array, with rows, columns, and cells, which are all the different versions. You can make use of the versioning to get history of a particular value or to build a time series within one row. Getting a single row is very fast in Bigtable so as long as you can keep the data volume in check for that row, making use of cell versions is a very powerful and fast option. There are patterns in the docs that are quite useful and not immediately obvious. For example, one trick is to reverse your timestamps (MAXINT64 – now) so instead of the latest version, you can get the oldest version effectively reversing the cell version sorting if you need it.

Google Cloud and Bigtable help us meet the low-latency demands of the growing online retail sector, with speed and easy integration with other Google Cloud services like BigQuery. With their managed services, we freed up time to focus on innovations and meet the needs of bigger and bigger customers. 

Learn more about Ravelin and Bigtable, and check out our recent blog, How BIG is Cloud Bigtable?

B2B marketplace IndiaMART aims to help businesses escape the restrictions of traditional supply chains. By providing access to a digital platform optimized for access from desktops and mobile devices, businesses can improve their operations and generate more revenue. IndiaMART’s suite of services includes web storefront, enquiry support, priority listings, premium number services, a lead management system, and payment facilitation.

IndiaMART also provides “behavior-based matchmaking” that identifies the supplier best equipped to meet buyer needs by product or service, category and location. IndiaMART then matches the designated suppliers with the buyers. Finally, IndiaMART operates as a “horizontal marketplace”—enabling suppliers to market to a large number of potential buyers—presenting a compelling offering for both groups.

Amarinder S. Dhaliwal, Chief Product Officer of IndiaMART, says once IndiaMART grew to a certain size, it benefited from a network effect—as more buyers used the marketplace, more suppliers came on board, prompting yet more buyers to access the service and so on. Suppliers becoming buyers and using IndiaMART to purchase products is another growth driver. “There is not just a network effect, but a community effect as well, as a supplier becomes a buyer,” says Dhaliwal. “This increases affinity, and supplier and buyer ‘lock-in’ to the marketplace increases multifold.”

Positioned to address challenges

IndiaMART’s proactive approach positioned the business well to address the challenges presented by new trends and market conditions. Traffic from mobile devices to its business-to-business marketplace has grown from about 30% to 75% over the last four years.

“Mobile traffic has grown at a compound annual growth rate of almost 100% over the same period,” says Dhaliwal. “The proliferation of smartphones and other mobile devices has brought a considerable number of new users onto the internet and these users look for value—the right price from the right supplier,” he adds. “Furthermore, they can connect at any time and from any location they can access a network.”

The mobility revolution also challenged IndiaMART to provide a user interface and experience optimized for devices of various types and sizes—and that incorporated screens much smaller than the screens incorporated in desktops. The organization also had to help users overcome issues such as inconsistent network coverage and quality—particularly in remote areas.

Becoming a mobile-first organization

IndiaMART is responding by becoming, for buyers, a “mobile-first” organization that meets the group’s technical and user experience requirements.

IndiaMART is also adapting its marketplace to support two key trends:
• Buyers using long, conversational sentences to conduct online searches rather than simply typing in keywords
• Non English-users—the vast majority of people in India—stepping up their use of the marketplace

“We expect that, within a few years, we will have more non-English users than English users on IndiaMART,” says Dhaliwal.

IndiaMART is also benefiting from Indian government measures to reform taxation and stimulate the digital economy. “There has been a huge focus on areas such as digital payments and the digitization of identity,” says Dhaliwal. “We have embraced elements of this agenda by implementing a digital payment platform and are continuing to look at ways of providing new digital services to suppliers.

“Meanwhile, the Indian government’s recent implementation of GST allows us to validate suppliers’ businesses and bring more qualified, more verified suppliers on our platform—improving the experience for buyers and suppliers.”

Speed and reliability an issue

IndiaMART had started operations with servers, storage, networking, and associated systems co-located in a data center in the United States. However, as buyers and suppliers increasingly used mobile devices—over occasionally unreliable networks—to access the marketplace, access speeds and reliability became an issue. With most requests traveling between India and the United States, network latency was unacceptably high. Furthermore, business growth meant IndiaMART needed an environment that could scale to meet demand for the next five to 10 years.

IndiaMART opted for a multi-cloud architecture and established criteria for cloud providers to win its business. “We required an infrastructure that could scale and meet our demand for fast response time without putting our business at risk,” says Dhaliwal. “This meant taking a phased rather than one-shot approach to the migration. We also needed to minimize any wasteful duplication of infrastructure and reduce latency. In addition, as we scaled, we needed to protect our systems, transactions and information, including the details of buyers and suppliers.”

Google Cloud team’s high-quality support

The organization performed proof of concept with the three largest multinational cloud services providers and found Google Cloud was best positioned to act as the cornerstone of its multi-cloud architecture. “The Google Cloud team gave us considerable support in helping us run a proof of concept of its services,” says Dhaliwal.

“The proof of concept also illustrated that Google Cloud was superior to the other cloud services we looked at.

“We could run our marketplace across multiple geo-locations under a single IP address, avoiding duplication, and users in India could connect to Google Cloud via the closest access point, with their traffic passing quickly across the Google network.

“In addition, Google Cloud’s load balancing service would enable encryption between load balancing layers and back ends to ensure security, while all communication would move across Google’s own protected network.”

Being one of India’s early users of G Suite, the organization was also familiar with Google Cloud applications and services.

IndiaMART then opted to work with the Google Cloud team and a certified partner on a step-by-step implementation that minimized any risk of disruption.

Deep engagement from Google

“We engaged very deeply with both Google and the partner to complete this migration,” says Dhaliwal. “The Google team worked very hard to understand our requirements and provide a solution that catered to our needs and could be deployed in a phased manner.” The team ran workshops and technical sessions with IndiaMART and, at a Google Summit, connected the marketplace provider to Google experts in databases and infrastructure.

“These discussions really helped us formulate a strategy moving forward,” says Dhaliwal.

Based on input from Google and its own evaluation, IndiaMART developed an architecture comprising virtual machine instances delivered through Compute Engine, Google Cloud’s infrastructure-as-a-service offering; Cloud Load Balancing to support cloud resources distributed across multiple locations; Cloud Pub/Sub to provide enterprise messaging; and Cloud Dataflow to transform and enrich data.

Cloud Armor works with Cloud Load Balancing to defend against distributed denial of service (DDoS) attacks; and Geocoding API helps the organization convert geographic coordinates into readable addresses and vice versa. Cloud AutoML allows IndiaMART to train machine learning models to meet its requirements. With Geocoding API, IndiaMART can matchmake buyers and suppliers based on location—providing a high quality experience for both parties. Finally, AutoML Translation allows the organization to create a custom machine learning model that converts product names from English into Hindi and other languages, effectively opening up new markets for buyers and suppliers.

Phase one complete

IndiaMART has completed phase one of the migration that involved moving its web properties across to Google Cloud. The organization is now experimenting with moving its APIs and databases to the service and anticipates completing the exercise over the coming year.

Average page load time down

The initial phase of the project has already delivered considerable benefits to IndiaMART. The organization has cut average page loading time from five seconds to three seconds, and Dhaliwal attributes close to one second of that reduction to the move to Google Cloud. “With Google Cloud, buyers and suppliers can access our marketplace much faster than previously,” says Dhaliwal. “This impacts positively on engagement, time spent on our marketplace, and the user’s entire journey with us.”

DDoS attack repelled

Google Cloud’s security features have already passed their first test. As IndiaMART undertook stage one of the migration, the business experienced a DDoS attack that generated request loads more than 400 times greater than normal. “Because we were on Google Cloud infrastructure, we could develop a solution to combat this severe DDoS attack,” says Sunil Parolia, Sr. VP at IndiaMART. “From a security perspective, this really justified our decision to go with Google Cloud.”

Google Cloud is also helping deliver the availability required by IndiaMART and the scalability to support growing demand. “As the number of people in India who access the internet grows from about 500 million to 700-800 million over the next couple of years, we will continue to build our traffic and be the dominant business-to-business platform,” says Dhaliwal. “On the supplier side, we expect to see more and more businesses come onto our marketplace—ranging from small-to-medium businesses up to larger brands. Google Cloud will enable us to accommodate this traffic without compromising the experience we provide.”