In the last year, there’s been an upward trend in cloud adoption in India. In fact, NASSCOM finds that cloud spending in India is estimated to grow at 30% per annum to cross the US$7 billion mark by 2022.

At Google, in our conversations with customers, discussions have evolved beyond cost savings and efficiencies. While those are still very relevant reasons for adopting cloud technologies, Indian enterprises are looking to Google Cloud to help them drive digital transformation, identify new revenue generating business models, reach previously untapped consumer markets, and build customer loyalty through greater insight and personalization.

Here are some companies and their stories.

Tata Steel: Mining data and maximizing its power

Tata Steel is a great example of an established enterprise from a traditional industry that is modernizing and embracing cloud computing. With an ambition to be a leader in manufacturing in India and a digital-first organization by 2022, Tata Steel believes smart analytics is key to enhancing operational efficiency and gaining business advantage. 

To organize data from siloed systems across the organization and make it easily accessible to all employees, Tata Steel is using Cloud Search and plans to scale it to more than one million documents and 28 disparate enterprise content sources including enterprise resource planning (ERP) and SharePoint. In fact, Tata Steel is one of the first Indian enterprises to harness the power of Cloud Search to meet some of the most aggressive ingestion demands, with indexing durations reduced from weeks to seconds.

They are also leveraging Google Cloud Platform (GCP) services like Google Cloud Storage and BigQuery to build their data lake and enterprise data warehouse so they can take advantage of advanced analytics and machine learning. Managed services such as AI Platform further enable Tata Steel to manage end-to-end AI/ML workflows within the GCP console. This complements their existing on-premise reporting and analytics tools, and brings data management to the forefront of everything they do—from forecasting market demand to predictive equipment maintenance.

“Digital is not just a goal, it’s become a way of life. We are digitizing everything from the deployment of factory vehicles to improving material throughput to marketing and sales. As a result, we have petabytes of structured and unstructured data that is not only waiting to be mined, but that we can generate intelligence from to create opportunities across our multiple lines of business using GCP,” said Sarajit Jha, Chief Business Transformation & Digital Solutions at Tata Steel.

Helping L&T Financial Services reach customers in rural communities

In rural communities, quick access to financial services can make a tremendous difference to livelihoods. L&T Financial Services provides farm-equipment finance, micro loans and two-wheeler finance to consumers across rural India backed by a strong digital and analytics platform. Their digital-loan approval app, which runs on GCP, makes it significantly faster and easier for people to apply for financial assistance to purchase important things such as farming equipment and two-wheelers. It also helps rural women entrepreneurs get quicker access to funds for their businesses through micro loans.

L&T Financial found G Suite to be a far better collaborative tool to help staff work together efficiently. Employees can interact with each other in real time using Hangouts Meet, and the task of information sharing is more seamless and secure through Drive. BigQuery also helps L&T Financial Services generate behavior scorecards to track credit quality of its micro-loan customers.

“Cloud is the technology that enables us to achieve scale and reach. Today there are countless data points available about rural consumers which enable us to personalize our products to serve them better. With access to faster compute power, we can also on-board consumers more efficiently. Our rural businesses have clocked a disbursement CAGR of 60% over the past three years.” said Sunil Prabhune, Chief Executive-Rural Finance, and Group Head-Digital, IT and Analytics, L&T Financial Services.

Creating conversational connections for Digitate’s customers

Digitate, a venture of TCS (Tata Consultancy Services), has integrated Dialogflow into its flagship brand ignio, an award-winning artificial intelligence platform for driving IT operations, workload operations and ERP operations for diverse enterprises. This integration is the next step in ignio’s product development journey, and will enable users to chat or talk with ignio to detect issues, triage problems, resolve them and even predict system behavior.

“ignio combines its unique self-healing AIOps capabilities for enterprise IT and business operations with Dialogflow’s AI/ML-based, easy to use, natural and rich conversational capabilities to create an unparalleled, intuitive and feature-rich experience for our customers,” says Akhilesh Tripathi, Head of Digitate.

Indian enterprises going G Suite

The base of Indian enterprises that are making the switch to G Suite to streamline their productivity and collaboration also continues to grow. Sharechat, BookMyShow, Hero MotorCorp, DB Corp and Royal Enfield are now able to move faster within their organizations, using intelligent, cloud-based apps to transform the way they work.

A hybrid and multi-cloud future in India

IDC predicts that by 2023, 55% of India 500 organizations will have a multi-cloud management strategy that includes integrated tools across public and private clouds. (IDC FutureScape: Worldwide Cloud 2019 Predictions  — India Implications (# AP43922319). We look forward to sharing more success stories of Indian enterprises that have taken the next step in their digital transformation journey.

May was a very busy month for data analytics product innovation. If you didn’t have the chance to attend our inaugural Data Cloud Summit, video replays of all our sessions are now available so feel free to watch them at your own pace. 

In this blog, I’d like to share some background behind the innovations we released in May, why we built them the way we did, and the type of value they can bring your company and your team.

But first, a huge thank you!

This week, we had the honor to announce that Google has been named a Leader in The Forrester Wave™: Streaming Analytics, Q2 2021 report. Forrester gave Dataflow a score of 5 out of 5 across 12 different criteria, stating: “Google Cloud Dataflow has strengths in data sequencing, advanced analytics, performance, and high-availability”. 

Google has more than a decade of experience in building real-time and internet-scale systems for its own needs, and we are excited to see that our ability to provide customers with a reliable, scalable, and performant platform is bearing fruit. 

This announcement comes on the back of the release of The Forrester Wave™: Cloud Data Warehouse, Q1 2021 report, which also named Google Cloud as a Leader.

We couldn’t be more excited about the recognition and appreciate all your feedback and trust in the work that we do to support your goal in accelerating data-powered innovation.

Innovation galore

Your feedback and your passion is the fuel that drives our ambition to deliver more and better services to you. That’s why, this year, we didn’t want to wait until Google Cloud Next to share some great products we have been working on. On May 26, our team announced a slew of new products, services and programs. Watch a quick summary below:

https://youtube.com/watch?v=DG1mOPMXJvw%3Fenablejsapi%3D1%26

Meeting you where you are

An important design principle behind all of our services is “meeting you where you are”. This means we aim to provide you with the tools and software you need to innovate on your own terms. Here are three new services that will help you do just that:

Datastream

Datastream, our new serverless change data capture (CDC) and replication service, allows your company to synchronize data across heterogeneous databases, storage systems, and applications reliably and with minimal latency to support real-time analytics, database replication, and event-driven architectures. Datastream delivers change streams from Oracle and MySQL databases into Google Cloud services such as BigQuery, Cloud SQL, Cloud Storage, and Cloud Spanner, saving time and resources while ensuring your data is accurate and up-to-date. 

• Under the hood, Datastream reads CDC events (inserts, updates, and deletes) from source databases, and writes those events with minimal latency to a data destination. It leverages the fact that each database source has its own CDC log—binlog for MySQL and LogMiner for Oracle—which it uses for its own internal replication and consistency purposes. 
• Datastream integrates with purpose-built and extensible Dataflow templates to pull the change streams written to Cloud Storage, and create up-to-date replicated tables in BigQuery for analytics. It also leverages Dataflow templates to replicate and synchronize databases into Cloud SQL or Cloud Spanner for database migrations and hybrid cloud configurations. 
• Datastream also powers a Google-native Oracle connector in Cloud Data Fusion’s new replication feature for easy ETL/ELT pipelining. By delivering change streams directly into Cloud Storage, customers can leverage Datastream to implement modern, event-driven architectures.

Looker and BigQuery Omni on Microsoft Azure

Research on multi cloud adoption is unequivocal — 92% of businesses in 2021 report having a multi cloud strategy. We want to continue supporting your choice by providing the flexibility you need to see your strategy through. 

• This past month, we introduced Looker, hosted on Microsoft Azure. For the first time, you can now choose Azure, Google Cloud, or AWS for your Looker instance. You can also self-host your Looker instance on-premises.
• We also introduced BigQuery Omni for Azure, which along with last year’s introduction of BigQuery Omni for AWS, will help you access and securely analyze data across Google Cloud, AWS, and Azure. 

The cost of moving data between cloud providers isn’t sustainable for many, and it’s still difficult to seamlessly work across clouds. BigQuery Omni represents a new way of analyzing data stored in multiple public clouds, which is made possible by BigQuery’s separation of compute and storage. By decoupling these two, BigQuery provides scalable storage that can reside in Google Cloud or other public clouds, and stateless resilient compute that executes standard SQL queries. 

• Unlike competitors, BigQuery Omni doesn’t require you to move or copy your data from one public cloud to another, where you might incur egress costs. You also benefit from the same BigQuery interface on Google Cloud, enabling you to query data stored in Google Cloud, AWS, and Azure without any cross-cloud movement or copies of data. 
• BigQuery Omni’s query engine runs the necessary compute on clusters in the same region where your data resides. For example, you can query Google Analytics 360 Ads data stored in Google Cloud and query logs data from your ecommerce platform and applications that are stored in AWS S3 and/or Microsoft Azure. 

Then, using Looker, you can build a dashboard that allows you to visualize your audience behavior and purchases alongside your advertising spend. 

Dataplex

We understand that most organizations still struggle to make high-quality data easily discoverable and accessible for analytics, across multiple silos, to a growing number of people and tools within their organization. 

They are often forced to make tradeoffs. For instance, moving and duplicating data across silos to enable diverse analytics use cases or leaving their data distributed but limiting the agility of decisions. 

• Dataplex provides an intelligent data fabric that enables you to centrally manage, monitor, and govern your data across data lakes, data warehouses, and data marts, while also ensuring data is securely accessible to a variety of analytics and data science tools. 
• One of the core tenets of Dataplex is letting you organize and manage your data in a way that makes sense for your business, without data movement or duplication. For that, we provide logical constructs like lakes, data zones, and assets. These constructs enable you to abstract away the underlying storage systems and become the foundation for setting policies around data access, security, lifecycle management, and so on. 
• For example, you can create a lake per department within your organization (e.g. Retail, Sales, Finance, etc.) and create data zones that map to data readiness and usage (e.g. landing, raw, curated_data_analytics, curated_data_science, etc.). 

Once you have your lakes and zones setup, you can attach data to these zones as assets. You can add data from different types of storage (e.g. GCS Bucket and BigQuery dataset) under the same zone. You can also attach data across multiple projects under the same zone. You can ingest data into your lakes and zones using the tools of your choice, including services such as Dataflow, Data Fusion, Dataproc, Pub/Sub, or choose from one of our partner products. Dataplex comes with built-in 1-click templates for common data management tasks. 
To find out more about Dataplex, head to cloud.google.com/dataplex or watch the video below:

https://youtube.com/watch?v=bbFeAt7cw1g%3Fenablejsapi%3D1%26

Helping you innovate everyday

Sharing data is hard. Traditional data sharing techniques use batch data pipelines that are expensive to run, create late arriving data, and can break with any changes to the source data. These techniques also create multiple copies of data, which brings unnecessary costs and can bypass data governance processes. They also fail to offer features for data monetization, such as managing subscriptions and entitlements. Altogether, these challenges mean that organizations are unable to realize the full potential of transforming their business with shared data.

Analytics Hub

To address these limitations, we are introducing Analytics Hub, a new fully managed service that helps organizations unlock the value of data sharing, leading to new insights and increased business value. 

This new service is built on the tremendous experience and feedback we have received over the years. For example, BigQuery has had cross-organizational, in-place data sharing capabilities since its inception in 2010—and the functionality is very popular. Over a 7-day period in April, we had over 3,000 different organizations sharing over 200 petabytes of data. These numbers don’t include data sharing between departments within the same organization.

Analytics Hub takes sharing to the next level, making it easy for you to publish, discover, and subscribe to valuable datasets that you can combine with your own data to derive unique insights. 

This includes: 

• Shared datasets: As a data publisher, you create shared datasets that contain the views of data that you want to deliver to your subscribers. Data subscribers can search through the datasets that are available across all exchanges for which they have access and subscribe to relevant datasets. In addition, the publisher can track subscribers, disable subscriptions, and see aggregated usage information for the shared data.
• Curated, self-service data exchanges: Exchanges are collections used to organize and secure shared datasets. By default, exchanges are completely private, but granular roles and permissions make it easy to deliver data to the right audience—whether internal or public. 

This is just the beginning for Analytics Hub. Please sign up for the preview, which is scheduled to be available in the third quarter of 2021.

Dataflow Prime

At Google Cloud, we have the great privilege of working with some of the most innovative organizations in the world. And this work provides us with a unique perspective into the future of big data processing. Dataflow Prime is a new platform based on a serverless, no-ops, and auto-tuning architecture that brings unparalleled resource utilization and radical operational simplicity to big data processing. This new service introduces a large number of exciting capabilities but I’d like to highlight three key aspects of the product:

• Vertical Autoscaling: Dataflow Prime dynamically adjusts the compute capacity allocated to each worker based on utilization, detecting when jobs are limited by worker resources and automatically adding more resources. Vertical Autoscaling works hand in hand with Horizontal Autoscaling to seamlessly scale workers to best fit the needs of the pipeline. As a result, it no longer takes hours or days to determine the perfect worker configuration to maximize utilization. 
• Right Fitting: Each stage of a pipeline typically has a different resource requirement than the others. Until now, either all workers in the pipeline would have had the higher memory and GPU, or none of them would. Pipelines either had to waste resources or suffer slower workloads. Right Fitting solves this problem by creating stage-specific pools of resources, optimized for each stage. 
• Smart Recommendations: Smart Recommendations automatically detects problems in your pipeline and shows potential fixes. For example, if your pipeline is running into permissions issues, a Smart Recommendation will detect which IAM permissions you need to enable to unblock your job. If you are using an inefficient coder in your job, Smart Recommendations will surface more performant coder implementations that can help you save on costs.

What’s next

We’re excited to hear your thoughts and feedback about all these exciting new services. I would also highly recommend that you connect with members of the community to learn more about their story and journey. A good example to start with is the Data To Value customer panel we produced at our inaugural Data Cloud Summit with the Chief Data Officers of Keybank and Rackspace. You can watch it for free below:

https://youtube.com/watch?v=ITI2Q3MkxuA%3Fenablejsapi%3D1%26

Editor’s note: In February 2021, Google Cloud and TELUS announced a 10-year strategic alliance to drive innovation of new services and solutions across data analytics, machine learning, and go-to-market strategies that support digital transformation within key industries, including communications technology, healthcare, agriculture, and connected home. By December 2021, TELUS had completed a pilot for a use case that leveraged Google Cloud AI and Machine Learning solutions and Telco Edge Anthos to increase safety in the workplace and save lives in manufacturing facilities. The use case leverages Multi-Access Edge Computing (MEC) to move the processing and management of traffic from a centralized cloud to the edge of TELUS’ 5G network, making it possible to deploy applications and process content closer to its customers, and thus yielding several benefits including better performance, security, and customization. Today, we invite Samer Geissah, Head of Technology Strategy and Architecture at TELUS, to share how the company is delivering on its promise to use this technology to drive meaningful change, starting with workers’ well-being.

Whenever a new technology buzzword comes along I think: what problems does this solve, and for whom is this going to make a real difference? That’s because at TELUS, we see innovation as a means to act on our social purpose to drive meaningful change, from modernizing healthcare and making our food supply more sustainable, to reducing our environmental footprint and connecting Canadians in need. Multi-Access Edge Computing (MEC) is a buzzword that offers an opportunity to do just this. That’s why we want to leverage cloud capabilities and optimize our network’s edge computing potential, tapping into our award-winning high-speed 5G connectivity to help solve some of industry’s most complex challenges.

The reason why this presents such a great opportunity is that companies across industries still rely on maintenance-heavy on-premises systems to manage core computing tasks. But, with cloud capabilities delivered at the edge of our 5G network, we open a new world of possibilities for them. For example, manufacturers who currently rely on IoT-enabled equipment in their facilities can deliver new experiences by running advanced AI-based visual inspections directly from 5G-enabled devices–all without the need for local processing power or extra on-site space. In fact, it’s this example that inspired our new use case, where our Connected Worker Safety solution can be applied across a range of business verticals to help improve safety, prevent injury, and save lives, demonstrating how the perfect combination of skilled people and digital technology can make the world a safer place.

Empowering intelligent decision making at the edge

Be it a farm, manufacturing facility, hospital, or a factory floor, workers should be able to work in environments where their health and safety are held as the highest priority. But how can employers ensure that their remote, frontline, and in-office employees are safe and healthy at all times? We’ve found the answer by combining Google Cloud AI/ML capabilities and Anthos as a platform for delivering workloads, with our network’s infrastructure.

Together with Google Cloud, we have been leveraging solutions with the power of MEC and 5G to develop a workers’ safety application in our Edmonton Data Center that enables on-premise video analytics cameras to screen manufacturing facilities and ensure compliance with safety requirements to operate heavy-duty machinery. The CCTV (closed-circuit television) cameras we used are cost-effective and easier to deploy than RTLS (real time location services) solutions that detect worker proximity and avoid collisions. This is a positive, proactive step to steadily improve workplace safety. For example, if a worker’s hand is close to a drill, that drill press will not bore holes in any surface until the video analytics camera detects that the worker’s hand has been removed from the safety zone area.

A few milliseconds could make all the difference when you are operating heavy equipment without guards in place. So, to power the solution’s predetermined actions with immediate response times, we worked with Accenture and hosted the application on an Anthos bare metal Google Cloud environment running on our TELUS multi-edge access computing.

Because all the conditions in our model are programmable, this solution can be replicated at scale across a variety of practical scenarios other than factory floors. The actions in response to the analysis are also programmable, which means companies can use this technology to look at workers’ conditions and decide the best course of action to educate, assist, and protect them. All this is done through a single pane of glass ecosystem, making it easy to customize this solution to meet various business needs.

Meanwhile, leveraging our existing global networks to process data and compute cycles at the edge eliminates the need to transport data to a central location for real-time computation. This means that we can offer this solution to partners while optimizing latency and lowering costs.

Powering blink-of-an-eye communication with Anthos

To put the importance of lowering speed into perspective, consider that the average latency of blinking your eye is about 300 milliseconds. From a safety point of view, preventative processes need to be much faster than that. For this use case, our machine learning models running on edge are currently processing data at a tenth of the time it takes for you to blink your eyes, and we’re aiming to lower that latency further to help build even safer systems.

Our plan is to deploy Anthos clusters on bare metal to our customers across Canada to take advantage of our existing enterprise infrastructure, making it possible for us to run our solution closer to partners and eventually enable just one millisecond of latency.

At that point, we’ll be able to power new use cases that require near real-time feedback, leaving absolutely no room for error. This could include remote surgery, platooning of fleets on autonomous vehicles, and many other cellular vehicle-to-everything (V2X) solutions that require high-speed communication for platform operators to manage remote edge fleets in far-away places.

Improving workers’ safety while enabling new sources of revenue

Although edge computing and 5G have been around for a while, we believe that use cases like this are only just starting to demonstrate the incredible speed of change and high potential that these models provide. The next step for us is to develop our workers’ safety solution and get it to market, making TELUS an early adopter of new 5G solutions at the edge that can help our business and industry partners make workplaces safer.

It’s a great win to be able to combine efforts with Google Cloud and reduce latency in a context where timing can impact and save lives, and I’m confident that workers’ safety is just the beginning of a series of industry challenges that we’ll address together.

Google Cloud is built on world-class technical infrastructure that supports services that are loved and relied on by billions of people across the globe: Google Search, YouTube, Gmail, Google Maps and more. A core tenet at Google Cloud is to leverage Google’s experience building and operating highly available and highly reliable planetary-scale compute, storage and networking systems and data centers. 

Google takes a workload-optimized approach to building its infrastructure, employing a combination of dedicated hardware and software components to meet its workloads’ ever-growing demands. Underpinning this infrastructure is Titanium, a system of purpose-built, custom silicon and multiple tiers of scale-out offloads that together power improvements in the performance, reliability, and security of our customers’ workloads (for example, 25% faster block storage IOPS/instance compared to the other two leading hyperscalers). Unveiled today at Google Cloud Next, you’ll find Titanium technology in many of Google Cloud’s recent infrastructure offerings.

10x demands of tomorrow 

Meeting the growing performance, reliability, and security demands of both legacy and emerging workloads is a constant challenge for cloud infrastructure providers. And now, these demands are multiplying with the heightened adoption of generative AI across almost every industry. Meanwhile, the benefits of Moore’s law have been declining in recent years. We can’t rely on silicon advances alone to meet tomorrow’s needs.

As just one example, this chart shows the exponential computing demands of large language models.

It was clear to us a long time ago that we needed to rethink our infrastructure designs to meet these demands. This is why, for several years, we’ve adopted workload-optimization and intentional design as central principles for our infrastructure platform. We engineer golden paths from silicon to the customer workload, using a combination of purpose-built infrastructure, prescriptive architectures, and an open ecosystem to deliver workload-optimized infrastructure. 

Offloads play a pivotal role

Central to this strategy are offload technologies. Traditionally, the CPU wears many hats: It runs the hypervisor, the virtualization stack to enable your workloads, and manages storage and networking I/O; it’s responsible for security isolation for virtual interfaces and physical hardware, etc. In this model, customer workloads running on the CPU contend for resources with these platform tasks.

Offloads on dedicated hardware perform behind-the-scenes security, networking, and storage functions that were previously performed by the host CPU, allowing the CPU to focus on maximizing performance for customer workloads.

A recent example of an on-host offload or accelerator is the Infrastructure Processing Unit (IPU), a system-on-chip that we co-designed with Intel to enable better security isolation and performance on our 3rd gen compute instances. The IPU enables:

• Predictable and efficient compute
• Programmable packet processing for low latency, 200 Gbps networking with 3x the packets per second compared to our previous-generation compute instances
• In-transit encryption with the PSP protocol

Another important example of Google’s on-host hardware is Titan, a secure, low-power microcontroller that helps ensure that every machine in Google Cloud boots from a trusted state.

But we did not stop there. To meet tomorrow’s demands, we knew we needed to go past the performance that could be achieved using the host’s dedicated offload hardware.

A tiered system of offloads

A key component of Titanium is its modern offload architecture, which combines capabilities whose scale and performance are well-established within Google, as well as new capabilities tailored for cloud use cases. 

Just as modern workloads scale out horizontally in the cloud, with Titanium, we’ve extended the architecture to augment on-host offloads with an additional tier of scale-out offloads that run outside the host. This system of offloads is deployed fleet-wide and dynamically adjusts to changing workload needs to continually deliver the best performance.

Example 1: Block storage
Titanium scale-out offload enables Hyperdisk block storage to deliver stellar I/O performance. Hyperdisk’s Titanium offload on the host IPU works in tandem with the Titanium scale-out offload tier that distributes I/O across Google’s massive cluster-level filesystem, Colossus.

With traditional offload architectures, higher block storage IOPS requires purchasing larger compute instances. For example, you may need to deploy a data-intensive workload on a compute instance with many more vCPUs than the workload needs just to get sufficient storage performance. This tight coupling results in wasted resources and higher costs for customers. Further, even with large instances, storage performance in the cloud may be inadequate relative to what customers are used to with on-prem storage systems.     

With our new block storage, Hyperdisk powered by Titanium, we have decoupled compute-instance size from storage performance. Hyperdisk uses a tier of offloads in our cloud fabric to offload storage I/O from the customer hosts to achieve higher storage performance even with a general-purpose VM.   

In fact, today we are announcing that Titanium-powered C3 VMs with Hyperdisk Extreme now support 500K IOPS per compute instance in preview to meet the needs of the most demanding workloads. This is 25% faster IOPS/instance compared to the other two leading hyperscalers, courtesy of the Titanium system. 

Example 2: Network routing
Virtual network routing is another example of using a second tier of scale-out offloads (“hoverboards”). With Titanium, Google’s Andromeda virtual networking stack on the IPU offload device sends all packets for which it does not have a route to Hoverboard gateways, which have forwarding information for all virtual networks. Hoverboards are standalone software switches that act as default routers for some flows.

Unlike the traditional gateway model, the control plane dynamically detects flows that exceed a specified usage threshold and programs them to be direct host-to-host flows, bypassing the hoverboards allowing hoverboards to focus on the long tail of less frequent flows. Typically, only a small subset of possible VM pairs in a network communicate with one another, so the VMs only have to store and process a small fraction of the usual network configuration on an individual VM host, improving per-server memory utilization and control-plane CPU scalability.

Titanium already powers your workloads

The Titanium journey began years ago with the component technologies described above. Many of our products already benefit from this architecture, and the newest elements of this architecture are now available with our 3rd gen Compute Engine instances such as C3 and the new Hyperdisk block storage. 

Going forward, look for the Titanium architecture to underpin all future generations of our infrastructure offerings, in the process enabling new classes of infrastructure capabilities that move well beyond the confines of a single server.