Google Cloud Helps LiveRamp Capture, Manage, Process and Visualize Data at Scale
4918
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Editor’s note: Today we’re hearing from Sagar Batchu, Director of Engineering at LiveRamp. He shares how Google Cloud helped LiveRamp modernize its data analytics infrastructure to simplify its operations, lower support and infrastructure costs and enable its customers to connect, control, and activate customer data safely and securely.
LiveRamp is a data connectivity platform that provides best in class identity resolution, activation and measurement for customer data so businesses can create a true customer 360 degree view. We run data engineering workloads at scale, often processing petabytes of customer data every day via LiveRamp Connect platform APIs.
As we integrated more internal and external APIs and the sophistication of our product offering grew, the complexity of our data pipelines increased. The status quo for building data pipelines very quickly became painful and cumbersome as these processes take time and knowledge of an increasingly complex data engineering stack. Pipelines became harder to maintain as the dependencies grew and the codebase became increasingly unruly.
Beginning last year, we set out to improve these processes and re-envision how we reduce time to value for data teams by thinking of our canonical ETL/LT analytics pipelines as a set of reusable components. We wanted teams to spend their time adding new features which encapsulate business value rather than spending time figuring out how to run workloads at scale on cloud infrastructure. This was even more pertinent with data science, data analyst and services teams whose daily wheelhouse was not the nitty gritty of deploying pipelines.
With all this in mind, we decided to start a data operations initiative, a concept popularised in the last few years, which aims to accelerate the time to value for data-oriented teams by allowing different personas in the data engineering lifecycle to focus on the “what” rather than the “how.”
We chose Google Cloud to execute on this initiative to speed up our transformation. Our architectural optimizations, coupled with Google Cloud’s platform capabilities simplified our operational model, reduced time to value, and greatly improved the portability of our data ecosystem for easy collaboration. Today, we have ten teams across LiveRamp running hundreds of workloads a day, and in the next quarter, we plan to scale to thousands.
Why LiveRamp Chose Google Cloud
Google Cloud provides all the necessary services in a serverless fashion to build complex data applications and run massive infrastructure. Google Cloud offers data analytics capabilities that help organizations like LiveRamp to easily capture, manage, process and visualize data at scale. Many of the Google Cloud data processing platforms also have open source roots making them extremely collaborative. One such platform is CDAP (Cask Data Application Platform), which Cloud Data Fusion is built on. We were drawn to this for the following reasons:
- CDAP is inherently multicloud. Pipeline building blocks known as Plugins define individual units of work. They can be run through different provisioners which implement managed cloud runtimes.
- The control plane is a set of microservices hosted on Kubernetes, whereas the data plane leverages the best of breed big data cloud products such as Dataproc.
- It is built as a framework and is inherently extensible, and decoupled from the underlying architecture. We can extend it both at the system and user-level through “extensions” and “plugins” respectively. For example, we were able to add a system extension for LiveRamp specific authorisation and build a plugin that encompasses common LiveRamp identity operations.
- It is open sourced, and there is a dedicated team at Google Cloud building and maintaining the core codebase as well as a growing suite of source, transform and sink connectors.
- It aligns with our remote execution and non-data movement strategy. CDAP executes pipelines remotely and manages through a stream of metadata via public cloud APIs.
- CDAP supports an SRE mindset by providing out of the box monitoring and observability tooling.
- It has a rich set of APIs backed by scalable microservices to provide ETL as a Service to other teams.
- Cloud Data Fusion, Google Cloud’s fully managed, native data integration platform is based on CDAP. We benefit from the managed security features of Data Fusion like IAM integration, customer manager encryption keys, role based access controls and data residency to ensure stricter governance requirements around data isolation.
How are teams using the Data Operations Platform?
Through this initiative, we have encouraged data science and engineering teams to focus on business logic and leave data integrations and infrastructure as separate concerns. A centralised team runs CDAP as a service, and custom plugins are hosted in a democratized plugin marketplace where any team can contribute their canonical operations.
Adoption of the platform was driven by one of our most common patterns of data pipelining: The need to resolve customer data using our Identity APIs. LiveRamp Identity APIs connect fragmented and inaccurate customer identity by providing a way to resolve PII to pseudonymous identifiers. This enables client brands to connect, control, and activate customer data safely and securely.
The reality of customer data is that it lives in a variety of formats, storage locations, and often needs bespoke cleanup. Before, technical services teams at LiveRamp had to develop expensive processes to manage these hygiene and validation processes even before the data was resolved to an identity. Over time, a combination of bash and python scripts and custom ETL pipelines became untenable.
By implementing our most used Identity APIs, a series of CDAP plugins, our customers were able to operationalise their processes by logging into a Low Code user interface, select a source of data, run standard validation and hygiene steps, visually inspect using CDAP’s Wrangler interface for especially noisy cases, and channel data into our Identity API. As these workflows became validated, they have been established as standard CDAP pipelines that can now be parameterized and distributed on the internal marketplace. These technical services teams have not only reduced their time to value but have also enabled future teams to leverage their customer pipelines without worrying about the portability to other team’s infrastructures.
What’s Next ?
With critical customer use cases now powered by CDAP, we plan on scaling out usage of the platform to the next batch of teams. We plan on taking on more complex pipelines, cross-team workloads, and adding support for the ever growing LiveRamp platform API suite.
In addition to the Google Cloud community and the external community, we have a growing base of LiveRamp developers building out plugins on CDAP to support routine transforms and APIs. These are used by other teams who push the limits and provide feedback — spinning a flywheel of collaboration between those who build and those who operate. Furthermore, teams internally can continue to use their other favorite data tools like BigQuery and Airflow as we continue to deeply integrate CDAP into our internal data engineering ecosystem.
Our data operations platform powered by CDAP is quickly becoming a center point for data teams – a place to ingest, hygiene, transform, and sink their data consistently.
We are excited by Google Cloud’s roadmap for CDAP and Data Fusion. Support for new execution engines, data sources and sinks, and new features like Datastream and Replication will mean LiveRamp teams can continue to trust that their applications will be able to interoperate with the ever evolving cloud data engineering ecosystem.
Introducing a strong alternative to CentOS: Rocky Linux Optimized for Google Cloud
3343
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
As CentOS 7 reaches end of life, many enterprises are considering their options for an enterprise-grade, downstream Linux distribution on which to run their production applications. Rocky Linux has emerged as a strong alternative that, like CentOS, is 100% compatible with Red Hat Enterprise Linux.
In April 2022, we announced a customer support partnership with CIQ, the official support and services partner and sponsor of Rocky Linux, as the first step in providing a best-in-class enterprise-grade supported experience for Rocky Linux on Google Cloud. Today we’re excited to announce the general availability of Rocky Linux Optimized for Google Cloud. We developed this collection of Compute Engine virtual machine images in close collaboration with CIQ so that you get optimal performance when using Rocky Linux on Compute Engine to run your CentOS workloads.
These new images contain customized variants of the Rocky Linux kernel and modules that optimize networking performance on Compute Engine infrastructure, while retaining bug-for-bug compatibility with Community Rocky Linux and Red Hat Enterprise Linux. The high bandwidth networking enabled by these customizations will be beneficial to virtually any workload, and are especially valuable for clustered workloads such as HPC (see this page for more details on configuring a VM with high bandwidth).
Going forward, we’ll collaborate with CIQ to publish both the community and Optimized for Google Cloud editions of Rocky Linux for every major release, and both sets of images will receive the latest kernel and security updates provided by CIQ and the Rocky Linux community. And of course, we’ll offer support with CIQ for both these images, per our partnership.
Rocky Linux Optimized for Google Cloud lets you take advantage of everything Compute Engine has to offer, including day-one support for our latest VM families, GPUs, and high-bandwidth networking. And for customers building for a multi-cloud deployment environment, the community Rocky images have you covered.
Starting today, Rocky Linux 8 Optimized for Google Cloud is available for all x86-based Compute Engine VM families (and soon for the new Arm-based Tau T2A), with version 9 soon to follow. Give it a try and let us know what you think.
Custom Voice Feature Can Help Brands Tweak IVR for Better Customer Experiences
5126
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
With the rise of digital assistants and conversational interfaces, people have grown accustomed to hearing and speaking to synthetic voices. But what do those voices sound like? Often, pretty repetitive. We’re all familiar with the Google Assistant voice, for example.
That’s why we are excited to announce the general availability of Custom Voice in our Cloud Text-to-Speech (TTS) API, a new feature that lets you train custom voice models with your own audio recordings to create unique experiences.
For businesses looking to build a strong brand identity, establishing a unique voice can help turn mobile app interactions or customer service based on interactive voice responses (IVR) into differentiated customer experiences. Our TTS API has included a speech synthesis service with a static list of voices for some time, but now, with Custom Voice, moving beyond these predefined options is easier than ever.
Custom Voice lets you simply submit your audio recordings to get access to the new voice directly in the TTS API. Custom Voice TTS includes guidance on the audio requirements to help make sure you generate a high quality custom TTS voice model. Once this new model is trained, all you have to do to start using the newly trained voice is reference the model ID in your calls to the Cloud TTS API.
At Google, we are committed to building safe and accountable AI products, not only because it’s the right thing to do, but because it is a critical step in ensuring successful use in production. As part of Google Cloud’s Responsible AI governance process, we conducted a deep ethical evaluation of Custom Voice TTS, and its relation to synthetic media, in order to surface and mitigate potential harms that it may create. If you are interested in Custom Voice TTS, there is a review process to help ensure each use case is aligned with our AI Principles and adequate voice actor consent is given.
Additionally, to verify that voice actors are actually the ones producing the audio, you will need to submit an audio file producing a sentence that Google Cloud chooses (for example: “I agree that my voice will be used to create a synthetic custom Text-to-Speech voice).
We’re looking forward to seeing this API help businesses solve problems in an easy, fast, and scalable way. TTS Custom Voice is now GA in these languages:
English (US)
English (AU)
English (UK)
Spanish (US)
Spanish (Spain)
French (France)
French (Canada)
Italian (Italy)
German (Germany)
Portugues (Brazil)
Japanese (Japan)
We plan to continue expanding this lineup in order to meet your needs. Ready to try for yourself? Contact your seller to get started on your use case evaluation today!
Announcing Apigee’s Pay-as-you-go pricing to provide flexibility and scale seamlessly
3164
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Apigee is Google Cloud’s API management platform that enables organizations to build, operate, manage and monetize their APIs. Customers from industries around the world trust Apigee to build and scale their API programs.
While some organizations operate with mature API-first strategies, others might still be working on a modernization strategy. Even within an organization, different teams often end up with diverse use cases and choices for API management. From our conversations with customers, we are increasingly hearing the need to align our capabilities and pricing with such varied workloads.
We’re excited to introduce a Pay-as-you-go pricing model to enable customers to unlock Apigee’s API management capabilities whilst retaining the flexibility to manage their own costs. Starting today, customers will have the option to use Apigee by paying only for what they are using. This new pricing model is offered as a complement to the existing Subscription plans (or) the ability to evaluate it for free.
Start small, but powerful with Pay-as-you-go pricing
The new Pay-as-you-go pricing model offers flexibility for organizations to:
- Unlock the value of Apigee with no upfront commitment: Get up and running quickly without any upfront purchasing or commitment
- Maintain flexibility and control in costs: Adapt to ever-changing needs whilst maintaining low costs. You can continue to automatically scale with Pay-as-you-go or switch to Subscription tiers based on your usage
- Provide freedom to experiment: Every API management use case is different and with Pay-as-you-go you can experiment with new use cases by unlocking value provided by Apigee without a long term commitment
Pay-as-you-go pricing works just like the rest of your Google Cloud bills, allowing you to get started without any license commitment or upfront purchasing. As part of the Pay-as-you-go pricing model, you will only be charged based on your consumption of
- Apigee gateway nodes: You will be charged on your API traffic based on the number of Apigee gateway nodes (a unit of environment that processes API traffic) used per minute. Any nodes that you provision would be charged every minute and billed for a minimum of one minute.
- API analytics: You will be charged for the total number of API requests analyzed per month. API requests, whether they are successful or not, are processed by Apigee analytics. Analytics data is preserved for three months.
- Networking usage: You will be charged on the networking (such as IP address, network egress, forwarding rules etc.,) based on usage
When is Pay-as-you-go pricing right for me?
Apigee offers three different pricing models
- Evaluation plan to access Apigee’s capabilities at no cost for 60 days
- Subscription plans across Standard, Enterprise or Enterprise plus based on your predictable but high volume API needs
- Pay-as-you-go without any startup costs
Subscription plans are ideal for use cases with predictable workloads for a given time period, whereas Pay-as-you-go pricing is ideal if you are starting small with a high value workload. Here are a few use cases where organizations would choose Pay-as-you-go if they want to:
- Establish usage patterns before choosing a Subscription model
- Evolve their API program by starting with high value and low volume API use cases
- Manage and protect your applications build on Google cloud infrastructure
- Migrate or modernize your services gradually without disruption
Next steps
Every organization is increasingly relying on APIs to build new applications, adopt modern architectures or create new experiences. In such transformation journeys, Apigee’s Pay-as-you-go pricing will provide flexibility for organizations to start small and scale seamlessly with their API management needs.
- To get started with Apigee’s Pay-as-you-go pricing go to console or try it for free here
Check out our documentation and pricing calculator for further details on Apigee’s Pay-as-you-go pricing for API management. For comparison and other information, take a look at our pricing page.
6332
Of your peers have already watched this video.
2:00 Minutes
The most insightful time you'll spend today!
Google Cloud’s ML-based Image Classification App: A Key to Global Wildlife Conservation
Wildlife provides critical benefits to support nature and people. Unfortunately, wildlife is slowly but surely disappearing from our planet and we lack reliable and up-to-date information to understand and prevent this loss. By harnessing the power of technology and science, we can unite millions of photos from [motion sensored cameras] around the world and reveal how wildlife is faring, in near real-time…and make better decisions
wildlifeinsights.org/about
3594
Of your peers have already downloaded this article
1:10 Minutes
The most insightful time you'll spend today!
According to the new Accelerate: State of DevOps 2018: Strategies for a New Economy report, elite DevOps performers have more frequent code deployments, faster lead times, lower change failure rates, and far quicker incident recovery times. And that’s just the beginning.
This year’s report reflects a fundamental industry change in how software and technology are viewed: What we referred to as IT performance in earlier research is now referred to as software delivery performance to differentiate this work from IT helpdesk and other support functions.
The ability to leverage this high-value performance is a key differentiator for organizations. Those that develop and deliver quickly are better able to experiment with ways to increase customer adoption and satisfaction, pivot when necessary, and keep up with compliance and regulatory demands.
This report highlights that any team in any industry, whether subject to a high degree of regulatory compliance or not—across all industry verticals—has the ability to achieve a high degree of software delivery performance. We classify teams into high, medium, and low performers and find that they exist in all organization types and industry verticals. You’ll find these classifications referenced throughout the report.
Download the report to find out what nearly 1,900 professionals said about the role of DevOps in their business, what sets top-performing DevOps teams apart, and how this gives them a competitive edge.
More Relevant Stories for Your Company
Private Services Connect in Google Cloud Regions Enables Customers to Consume Services Faster
At Google Cloud, we believe in making it simple and secure to consume services whether they're from Google, a third party or customer-owned. With Private Service Connect, we have adopted a service-centric approach to our network that abstracts the underlying networking infrastructure. And today, we are announcing Private Service Connect is
Google Cloud Partnership Helps Lowe’s SRE Team Achieve 20X More Releases Per Month
Editor’s note: Today we hear from the Lowe’s SRE team. They share about how they have been able to increase the number of releases they can support by adopting Google’s Site Reliability Engineering (SRE) framework and leveraging their partnership with Google Cloud. At Lowe’s, we’ve made significant progress in our multiyear technology transformation.
Maximizing API Potential: A Look at 7 Prominent API Management Use Cases
Paper currency — which started gaining prominence in the 1600s — changed the face of global economics and ushered in a new era of international monetary regulation. The primary reason currency created such disruption was its ability to standardize the “medium of exchange”. APIs created a similar effect in the
Microservices in the Cloud with Kubernetes and Istio
Are you building or interested in building microservices? They are a powerful method to build a scalable and agile backend, but managing these services can feel daunting: building, deploying, service discovery, load balancing, routing, tracing, auth, graceful failures, rate limits, and more. The most suited solution for you is Istio.
