5197
Of your peers have already watched this video.
5:00 Minutes
The most insightful time you'll spend today!
How to Build and Maintain Spark and Hadoop Clusters Quickly, Easily and Inexpensively
How often have you struggled to create Spark and Hadoop clusters only to have them become unstable?
Building and maintaining clusters that are the building blocks of big data analytics should not be a time-consuming, repetitive, and expensive process. But with the current set of tools, it often is.
Google Cloud Dataproc solves this. It is a fast, easy-to-use, fully-managed cloud service for running Spark and Hadoop clusters in a simple, more cost-efficient way.
Operations that used to take hours or days take now seconds or minutes instead, and you pay only for the resources you use (with per-second billing).
Watch this five-minute demo to find out how—with just a few clicks—you can set up clusters that build and heal themselves. You’ll find out how to creating a large Dataproc cluster with preemptible VMs, which help run large clusters at a low total cost.
How Google Cloud’s Scalable Data Storage and High Compute Resources Fuel Investment Research
6286
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Investment management is a heavily data-driven industry—portfolio managers and investment researchers require a large number of data sources to guide them in shaping their investment strategies.
New cloud capabilities and technologies enable investment managers to process data faster than ever before and iterate on ideas quickly to fuel innovation in the signal generation process and gain a competitive edge.
Using the cloud for investment research workflows makes it easier to onboard data from data providers, spin up large compute workloads in the midst of market volatility or during heavy research cycles, and manage complex machine learning or natural language workflows to gain market insights.
We hear from industry leaders that they’re exploring new ways to run investment research. “Differentiated investment strategies require new types of information sources, and new ways to process that information,” David Easthope, senior analyst, Market Structure and Technology, Greenwich Associates. “And that, of course, relies heavily on having access to reliable and scalable storage, computational, and AI / ML resources. More specifically, quantitative strategies can benefit from the computational platforms and embedded AI/ML capabilities the cloud can offer.”
Google Cloud gives investment managers essential components to work and operate faster as they bring their investment research workflows to the cloud. Here are the key highlights:
1. Simplify, speed up your data acquisition, discovery, and analytics
The foundation of any investment strategy starts with data—acquiring it, detecting patterns, and analyzing it for insights. Enabling data providers to easily share large datasets such as tick history within a high-performance analytics engine can greatly reduce the data engineering overhead when possible.
Once data is onboarded, you can tag business and technical metadata related to your datasets and provide portfolio managers the ability to discover these datasets via a search interface.
We further review analytics options for various scenarios, including aggregating massive datasets, creating dashboards, and incorporating streaming analytics workloads.
2. Take advantage of burst compute workloads
Data engineers and researchers require ready access to burst compute capabilities to perform backtesting, portfolio simulations and run risk calculations. Cloud works well for these workloads due to its elasticity, consumption-based models, and hardware evolution.
Many investment managers are shifting to a container-based strategy along with a Kubernetes-based scheduler for greater consistency, scaling and efficiency in environments with a large number of researchers. Cloud managed services and a rich suite of CI/CD tools can make this vision a reality while improving security and developer productivity.
3. Tackle machine learning (ML) and model deployment with the help from cloud
Quantitative researchers scour vast amounts of market and alternative data sources searching for signals and correlations, while ML engineers have the challenge of taking these signals and moving them to production.
Google Cloud empowers users to create and operationalize their models without wasting valuable time with a comprehensive set of MLOps tools.
In this paper, we explore multiple solutions for ML and model deployment. Those capabilities reduce the amount of time operationalizing ML models, so quants and data scientists have more time to devote to differentiating activities.
4. Get the data you need in less time with Natural Language and Document AI
Thousands of financial filings, news articles, and sell-side research reports are generated every day, and it’s difficult for humans alone to process this volume of information. These documents are often generated in many languages and the ability to do entity recognition, sentiment or syntactical analysis in those languages, or perhaps translate them into the language of the portfolio manager is of critical importance. Google Cloud provides these capabilities through pre-trained models, or allows you to train high-quality models with your own datasets.
Getting started
There are plenty of emerging technologies, tools, and approaches available to help investment managers today. At Google Cloud, we can help you access, organize, and utilize these essential components to make your research faster, reliable, and more valuable.
To learn more about these four keys to better investment research, check out our whitepaper for more.
Serverless and BigQuery Together on Google Cloud: Behind the L’Oreal Beauty Tech Data Platform
5342
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Editor’s note: In Today’s guest post we hear from beauty leader L’Oréal about their approach to building a modern data platform on fully managed services: managing the ingest of diverse datasets into BigQuery with Cloud Run, and orchestrating transformations into relevant business domain representations for stakeholders across the organization. Learn more about how businesses have benefited from Cloud Run in Forrester’s report on Total Economic Impact.
L’Oréal was born out of science. For over 100 years, we have always shaped the future of beauty, and taken its eternal quest to new horizons. This has earned us our current position as the world’s uncontested beauty leader (~€ 32 B annual sales in 2021), present in 150 countries with over 85,000 employees.
Today, with the power of our game-changing science, multiplied by cutting-edge technologies, we continue our lifelong journey of shaping the future of beauty.
As a Beauty Tech company, we leverage our decades-long heritage of rich data assets to empower our decision-making with instant, sophisticated analysis.
Because we oversee global brands, which must adapt to local requirements, we need to maintain a deep understanding of what a brands’ data represents, while managing disparate legal and regulatory requirements for different countries. Our end goal is to run a safe, compliant and sustainable data warehouse as efficiently and effectively as possible.
We sync and aggregate internal and external data from a wide variety of sources across organizations and retail stores. This made the management of our data warehouse infrastructure used to be very complex and hard to manage before Google Cloud. L’Oréal’s footprint was so large that we once found it impossible to have a standardized method to handle data. Every process was vendor-specific, and the infrastructure was brittle. We went looking for a solution to our complex data infrastructure needs, and defined the following non-negotiable principles:
- No Ops: The job of a developer at L’Oréal is not to manage servers. We need an elastic infrastructure that scales on demand, so that our developers can focus on delivering customized and inclusive beauty experiences to all consumers, rather than focusing on managing servers.
- Secure: We have strict security and compliance requirements which vary by country, and we employ a zero-trust security strategy. We must keep both our own internal data and customer data safe and encrypted.
- Sustainable : Our data lives in multiple environments, including on-prem data centers and public cloud services. We must be able to securely access and analyze this data while minimizing the complexity and environmental impact of moving and duplicating data.
- End-to-end supervision: Because developers shouldn’t be managing servers, we need a “single pane of glass” dashboard to monitor and triage the system if something goes wrong.
- Easy-to-deploy: Deploying code safely should not compromise velocity. We are constantly developing innovations that push the boundaries of science and reinvent beauty rituals. We need integrated tools to make our code deployment process seamless and safe.
- Event-driven architecture: Our data is used globally by research, product, business and engineering teams with high expectations on data quality and timeliness. Many of our internal processes and analysis are based on near real-time data.
- Data products delivered “as a service”: We want to empower our employees to drive business value at record speed. To that end, we need solutions that enable us to remove the developers from the critical path of solution delivery as much as possible.
- Extract-load-transform (ELT): Our goal is to implement the pattern to load data as soon as possible into the data warehouse to take advantage of SQL transformations.
After considering multiple vendors on the market, with these principles in mind, we landed on end-to-end Google Cloud serverless and data tooling. We were already using Google Cloud for a few processes, including BigQuery, and loved the experience.
We’ve now expanded our use of Google Cloud to fully support the L’Oréal Beauty Tech Data Platform.
L’Oréal’s Beauty Tech Data Platform incorporates data from two types of sources: directly via API, which is data that adapts easily to our schema and is inserted directly into BigQuery, and bulk data from integrations, which require event-driven transformations using Eventarc mechanisms. These transformations are performed in Cloud Run and Cloud Functions (2nd gen), or directly in SQL. With Google Cloud, we can adapt very quickly.
Today, we currently have 8500 flows for ~5000 users using the native zero-trust capabilities offered by Google Cloud. Indeed, the flows come from Google Cloud and other third-party services.
BigQuery enabled us to adopt standard SQL as our universal language in our data warehouse and meet all expectations for queries and reporting. We were also able to load original data using features like federated queries, and efficiently transitioned from ETL to ELT data ingestion by handling semi-structured data with SQL. This approach of loading original data from sources into BigQuery with non-destructive transformations allows us to reprocess data for new use-cases easily, directly within BigQuery.
Our applications are hosted on multiple environments – on-premises, in Google Cloud, and in other public clouds. This made it difficult for our data engineers and analysts to natively analyze data across clouds until we started using BigQuery Omni. This capability of BigQuery allowed us to globally access and analyze data across clouds through a single pane of glass using the native BigQuery user interface itself. Without BigQuery Omni, it would’ve been impossible for our teams to natively do cross-cloud analytics. Moreover, it eliminated the need for us to move sensitive data, which is not only expensive because of local tax and subsea transport, but also incredibly risky – sometimes even forbidden – because of local regulations.
Today Google Cloud powers our Beauty Tech Data Platform, which stores 100TB of production data in BigQuery and processes 20TB of data each month. We have more than 8000 governed datasets, and 2 millions of BigQuery tables coming from multiple data sources such as Salesforce, SAP, Microsoft, and Google Ads.
For more complex transformations where custom and specific libraries are required, Cloud Workflows help us to manage the complexity very efficiently by orchestrating steps in containers through Cloud Run, Cloud Functions and even BigQuery jobs — the most used way to transform and add value to the L’Oréal data.
Additionally, by using BigQuery and Google Cloud’s serverless compute for API ingestion, bulk data loading, and post-loading transformations, we can keep the entire system in a single boundary of trust at a fraction of the cost. With ingest, queries, and transformations all being fully elastic and on-demand, we no longer have to perform capacity planning for either the compute or analytics components of the system. And of course these services’ pay-as-you-go model perfectly aligns with L’Oréal’s strategy of only paying for something when you use it.
Google Cloud fulfilled the requirements of our Beauty Tech Data Platform. And as if offering us a no-ops, secure, easy-to-deploy, custom-development free, event-based platform with end-to-end supervision wasn’t enough, Google Cloud also helped us with our sustainability efforts.
Being able to measure and understand the environmental footprint of our public cloud usage is also a key part of our sustainable tech roadmap. With Google Cloud Carbon Footprint, we can easily see the impact of our sustainable infrastructure approach and architecture principles. Our Beauty Tech platform is a strategic ambition for L’Oréal: inventing the beauty products of the future while becoming the company of the future.
Sustainable tech is an imperative and a very important step towards this ambition of creating responsible beauty for our consumers, and sustainable-by-design tech services for our employees. We all have a role to play, and by joining forces, we can have a positive impact.
Google Cloud’s data ecosystem and serverless tools are highly complementary, and made it possible to build a next-generation data analytics platform that met all our needs.
Get started using serverless and BigQuery together on Google Cloud today.
Ahead of the Curve: 5 Data and AI Trends Set to Shape 2023
2244
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
How will your organization manage this year’s data growth and business requirements? Your actions and strategies involving data and AI will improve or undermine your organization’s competitiveness in the months and years to come. Our teams at Google Cloud have an eye on the future as we evolve our strategies to protect technology choice, simplify data integration, increase AI adoption, deliver needed information on demand, and meet security requirements.
Google Cloud worked with* IDC on multiple studies involving global organizations across industries in order to explore how data leaders are successfully addressing key data and AI challenges. We compiled the results in our 2023 Data and AI Trends report. In it, you’ll find the metrics-rich research behind the top five data and AI trends, along with tips and customer examples for incorporating them into your plans.
1: Show data silos the door
Given the increasing volumes of data we’re all managing, it’s no surprise that siloed transactional databases and warehousing strategies can’t meet modern demands. Organizations want to improve how they store, manage, analyze, and govern all their data, while reducing costs. They also want to eliminate conflicting insights from replicated data and empower everyone with fresh data.
A unified data cloud enables the integration of data and insights into transformative digital experiences and better decision making.
Andi Gutmans, GM and VP of Engineering for Databases, Google Cloud
In the report, you can learn how to adopt a unified data cloud that supports every stage of the data lifecycle so that you can improve data usage, accessibility, and governance. Inform your strategy by drawing on organizations’ examples such as a data fabric that improves customer experiences by connecting more than 80 data silos, as well as other unified data clouds that save money and simplify growth.
2: Usher in the age of the open data ecosystem
Data is the key to unlocking AI, speeding up development cycles, and increasing ROI. To protect against data and technology lock-in, more organizations are adopting open source software and open APIs.
Tweet this quote
Understand how you can simplify data integration, facilitate multicloud analytics, and use the technologies you want with an open data ecosystem, as described in the report. Learn from metrics about global open source adoption and public dataset usage. And explore how global companies adopted open data ecosystems to improve patient outcomes, increase website traffic by 25%, and cut operating costs by 90%.
3: Embrace the AI tipping point
Pulling useful information out of data is easier with AI and ML. Not only can you identify patterns and answer questions faster but the technologies also make it easier to solve problems at scale.
We’ve reached the AI tipping point. Whether people realize it or not, we’re already using applications powered by AI—every day. Social media platforms, voice assistants, and driving services are easy examples.
June Yang, VP, Cloud AI and Industry Solutions, Google Cloud
Organizations share how they’re reaching their goals using AI and ML by empowering “citizen data scientists” and having them focus on small wins first. Gain tips from Yang and other experts for developing your AI strategy. And read how organizations achieve outcomes such as a reduction of 7,400 tons per year in carbon emissions and a more than 200% increase in ROI from ad spend by using pattern recognition and other AI capabilities.
4: Infuse insights everywhere
Yesterday’s BI solutions have led to outdated insights and user fatigue with the status quo, based on generic metrics and old information. Research shows that as new tools come online, expectations for BI are changing, with companies revising their strategies to improve decision making, speed up the development of new revenue streams, and increase customer acquisition and retention by providing individuals with needed information on demand.
Organizations are equipping business decision-makers with the tools they need to incorporate required insights into their everyday workflows.
Kate Wright, Senior Director, Product Management, Google Cloud
In the report, you’ll discover why and how data leaders are rethinking their BI analytics strategies and applications to improve users’ trust and use of data in automated workflows, customizable dashboards, and on-demand reports. Global companies also share how they improve decision making with self-service BI, customer experiences with IoT analysis, and threat mitigation with embedded analytics.
5: Get to know your unknown data
Increasing data volumes can make it harder to know where and what data they store, which may create risk. Case in point: If a customer unexpectedly shares personally identifiable information during a recorded customer support call or chat session, that data might require specialized governance, which the standardized storage process may not provide.
If you don’t know what data you have, you cannot know that it’s accurately secured. You also don’t know what security risks you are incurring, or what security measures you need to take.
Anton Chuvakin, Senior Staff Security Consultant, Google Cloud
Check out the report to learn about data security risks that are often overlooked and how to develop proactive governance strategies for your sensitive data. You can also read how global organizations have increased customer trust and productivity by improving how they discover, classify, and manage their structured and unstructured data.
Be ready for what’s next
What’s exciting about these trends is that they’re enabling organizations across industries to realize very different goals using their choice of technologies. And although all the trends depend on each other, research shows you can realize measurable benefits whether you adopt one or all five.
Review the report yourself and learn how you can refine your organization’s data and AI strategies by drawing on the collective insights, experiences, and successes of more than 800 global organizations.
7522
Of your peers have already downloaded this article
15:52 Minutes
The most insightful time you'll spend today!
Breakthroughs in artificial intelligence (AI) have captured the imaginations of business and technical leaders alike. The AI techniques underlying these breakthroughs are finding diverse application across every industry. Early adopters are seeing results, particularly encouraging is that AI is starting to transform processes in established industries, from retail to financial services to manufacturing.
However, an organization’s effectiveness in applying these breakthroughs is anchored in the basics: A disciplined foundation in capturing, preparing and analyzing data.
Data scientists spend up to 80% of their time on the “data wrangling,” “data munging” and “data janitor” work required well before the predictive capabilities promised by AI can be realized.
Capturing, preparing and analyzing data creates the foundation for successful AI initiatives. To help business and IT leaders create this virtuous cycle, Google Cloud has prepared a CIO’s guide to data analytics and machine learning that outlines key enabling technologies at each step. Crucially, the guide illustrates how managed cloud services greatly simplify the journey—regardless of an organization’s maturity in handling big data.
This is important because, for many companies, the more fundamental levels of data management present a larger challenge than new capabilities like AI. “Management teams often assume they can leapfrog best practices for basic data analytics by going directly to adopting artificial intelligence and other advanced technologies,” noted Oliver Wyman consultants Nick Harrison and Deborah O’Neill in a recent Harvard Business Review article (aptly titled If Your Company Isn’t Good at Analytics, It’s Not Ready for AI). “Like it or not, you can’t afford to skip the basics.
Building on new research and Google Cloud’s own contributions to big data since the beginning, this guide walks readers through each step in the data management cycle, illustrating what’s possible alongside examples.
Specifically, the CIO’s guide to data analytics and machine learning is designed to help business and IT leaders address some of the essential questions companies face in modernizing data strategy:
- For my most important business processes, how can I capture raw data to ensure a proper foundation for future business questions? How can I do this cost-effectively?
- What about unstructured data outside of my operational/transactional databases: raw files, documents, images, system logs, chat and support transcripts, social media?
- How can I tap the same base of raw data I’ve collected to quickly get answers as new business questions arise?
- Rather than processing historical data in batch, what about processes where I need a real-time view of the business? How can I easily handle data streaming in real time?
- How can I unify the scattered silos of data across my organization to provide a current, end-to-end view? What about data stored off-premises in the multiple cloud and SaaS providers I work with?
- How can I disseminate this capability across my organization—especially to business users, not just developers and data scientists?
Because managed cloud services deal with an organization’s sensitive data, security is a top consideration at each step of the data management cycle. From data ingestion into the cloud, followed by storage, preparation and ongoing analysis as additional data flows in, techniques like data encryption and the ability to connect your network directly to the Google Cloud must reflect data security best practices that keep data assets safe as they yield insights.
Wherever your company is on its path to data maturity, Google Cloud is here to help. We welcome the opportunity to learn more about your challenges and how we can help you unlock the transformational potential of data.
Google Cloud named a leader in the Forrester Wave: Streaming Analytics
DOWNLOAD WHITEPAPER4179
Of your peers have already downloaded this article
15:30 Minutes
The most insightful time you'll spend today!
There’s a lot of pressure on companies today to become data-driven, but they need high-performing technology to make that a reality.
As part of Google’s larger cloud data analytics platform, Cloud Pub/Sub and Cloud Dataflow are designed for ease of use, scalability, and performance.
Google Cloud unifies streaming analytics and batch processing the way it should be. No compromises.
Forrester
Forrester has named Google Cloud as a leader in The Forrester Wave™: Streaming Analytics, Q3 2019. This findings reflect Google Cloud’s market momentum, and what Google hears from enterprise customers who are using Cloud Pub/Sub and Cloud Dataflow in production for streaming analytics in conjunction with our broader platform.
According to Forrester, many leading enterprises realize that real-time analytics—the analytics of what’s happening with data in the present—is an incredible competitive advantage. With it, they can act right away to serve customers, fix operational problems, power internet of things (IoT) apps, and respond decisively to competitors.
The report evaluates the top 11 vendors against 26 rigorous criteria for streaming analytics to help enterprise IT teams understand their options and make informed choices for their organizations. Google scored 5 out of 5 in Forrester’s report evaluation criteria of scalability, availability, aggregates, management, security, extensibility, ability to execute, solution roadmap, partners, community, and customer adoption.
We are very excited about the productivity benefits offered by Cloud Dataflow and Cloud Pub/Sub. It took half a day to rewrite something that had previously taken over six months to build using Apache Spark.
Paul Clarke, Director of Technology, Ocado
While stream analytics is a leading business priority, real-life use cases frequently require batch data as an input as well. Google Cloud Platform (GCP) customers can simplify their pipeline development by reusing code across both batch and stream processing using Apache Beam, which also provides pipeline portability to OSS projects. Further, Google Cloud’s data analytics portfolio autoscales across ingestion, processing, and analysis, which eliminates the need for provisioning and makes handling varying volumes of streaming data automatic. We hear from users that they’re able to ingest and process data much more quickly than in the past, helping to get new business insights faster and allowing more users to do self-serve analytics.
More Relevant Stories for Your Company
A Breakdown of Cloud-based Data Ingestion Practices
Businesses around the globe are realizing the benefits of replacing legacy data silos with cloud-based enterprise data warehouses, including easier collaboration across business units and access to insights within their data that were previously unseen. However, bringing data from numerous disparate data sources into a single data warehouse requires you
How the City of Memphis Uses Technology to Identify 75 Percent More Potholes
At 340 square miles, the City of Memphis is among the largest in the United States in terms of land area. Memphis has over 6,800 lane-miles of city streets, enough to drive back and forth to Los Angeles four times. Keeping these streets well maintained and safe for citizens and visitors is
Chronicle Security Analytics: Key to Address Security Data Overload
Google Cloud's Chronicle is a security analytics platform built for modern use cases to combat modern threats. In today's world, enterprises have undergone significant changes in all aspects, and must adapt to their security needs to counter threats and attacks. Watch the video to learn Google Cloud's initiative to help
Google Cloud Celebrates Journey of 3 Inspiring Founders for the Asian Pacific American Heritage Month
May is Asian Pacific American Heritage Month —a time for us to come together to celebrate and remember the important people and history of Asian and Pacific Island heritage. This feature highlights three AAPI founders from the Google For Startups community. Read on to learn how these three founders built
