Tyson Foods’ Story of Unlocking Opportunities by Integrating Real-time Analytics with AI and BI
6475
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
As data environments become more complex, companies are turning to streaming analytics solutions that analyze data as it’s ingested and deliver immediate, high-value insights into what is happening now. These insights enable decision makers to act in real time to take advantage of opportunities or respond to issues as they occur.
While understanding what is happening now has great business value, forward-thinking companies are taking things a step further, using real-time analytics integrated with artificial intelligence (AI) and business intelligence (BI) to answer the question, “what might happen in the future?” Arkansas-based Tyson Foods has embraced AI/BI analytics to enable predictive insights that unlock new opportunities and drive future growth.
Creating a digital twin for connected intelligence company wide
Before using AI/BI, Tyson’s analytics capabilities consisted of traditional BI solutions focused on KPIs and simplifying data so that humans could understand it. Tyson wanted to leverage its data to uncover ways to improve current processes and grow its business. But with BI alone, Tyson struggled to use data to run the simulations and scenarios essential to make educated decisions. To keep growing, it had to embrace the complexity of its data, building ways to analyze it and use it to inform decision making.
Tyson’s on-premises analytics solutions limited its ability to be aggressive and make intelligent, timely, prescriptive decisions. The solution was to create a digital twin to scale optimizations within business processes, moving from local optimizations to system-wide connected optimizations. Doing so meant shifting entirely to cloud computing, with an initial focus on building the ingestion component of the digital twin platform.
Investing in a digital twin enabled Tyson to accelerate new capabilities like supply chain simulation “what-if” scenarios, prescriptive price elasticity recommendations, and improvement of customer intimacy.
Solving the ingestion problem for faster time to insights
Before its migration to Google Cloud, analytics projects that Tyson suffered from uncertainty over how to obtain the data. This problem was prolific and caused project times to be extended for weeks or even months due to the need to write and support one-off data ingestion processes at the front end. This problem also prevented the IT team from delivering analytics solutions fast enough for the business to take full advantage of them.
To solve this analytics problem, the team created Data Ingestion Compute Engine (DICE). DICE is a Google Cloud-hosted, open-source, cloud-native ingestion platform developed to provide configuration-based, no-ops, code-free ingestion from disparate enterprise data systems, both internal and external. It is centered on three high-level goals:
- Accelerate the speed of delivery of IT analytics solutions
- Enable growth of IT capabilities to produce meaningful insight
- Reduce long-term total cost of ownership for ingestion solutions
Creating DICE ingestion platform with Google Cloud services
Teams use DICE to set up secure data ingestion jobs in minutes without having to manage complex connections or write, deploy, and support their own code. DICE enables unbound scale, highly parallel processing, DevSecOps, open source, and the implementation of Lambda Data Architecture.
A DICE job is the logical unit of work in the DICE platform, consisting of immutable and mutable configurations persisted as JSON documents stored in Firestore. The job exists as an instruction set for the DICE data engine, which is Apache Beam running Dataflow to instruct which data to pull, how to pull it, how often to pull it, how to process it, when it changes, and where to direct it.
Two of DICE’s primary layers include the metadata engine and the data engine. The metadata engine is responsible for the creation and management of DICE job configuration and orchestration. It is made up of many microservices that interact with multiple Google Cloud services, including the job configuration creation API, job build configuration helper API, and job execution scheduler API.
The data engine is responsible for the physical ingestion of data, the change detection processing of that data, and the delivery of that data to specified targets. The data engine is Java code that uses the Apache Beam unified programming model and runs in Dataflow. It is comprised of streaming, jobs, and Dataflow flex template batch jobs. Logically, the data engine is segmented across three layers: the inbound processing layer, the DICE file system layer, and the target processing layer, which takes the data from the DICE file system and moves it to targets.
Rolling DICE for thousands of ingestion jobs each day
DICE was first deployed to a production environment in November 2019, and just two years later, it has more than 3,000 data ingestion jobs from more than a hundred disparate data systems, both internal and external to Tyson Foods. Most of these jobs run multiple times a day. On a daily basis the DICE environment sees more than 25,000 Dataflow jobs running and an average of 3.25 terabytes of new data being ingested.
DICE supports ingestion from many different types of technologies, including BigQuery, SQL Server, SAP HANA, Postgres, Oracle, MySQL, Db2, various types of file systems, and FTP servers. Additionally, DICE supports target platform technologies for ingestion jobs that include multiple JDBC targets, multiple file system targets, and BigQuery and queue-based store and forward technologies.
The platform continues to see linear growth of DICE jobs, all while keeping platform costs relatively flat. With increasing demand for the platform, Tyson’s IT team is constantly enhancing DICE to support new sources and targets.
This intelligent platform keeps adding new value and makes it simple for Tyson to take advantage of its data. This innovation is a necessity in this fast-changing world of digital business in which companies must transform a high volume of complex data into actionable insight.
7990
Of your peers have already downloaded this article
20:30 Minutes
The most insightful time you'll spend today!
These business and technology innovators are leveraging cloud computing, machine learning, APIs, collaborative platforms, among others to advance healthcare, financial services, retail, media and entertainment, manufacturing, and government.
They include some of the biggest brands in the world, as well as some you’ve probably never heard of. But they all have one thing in common: They are astounding in their idea, scope and execution.
Every story has a short–and a longer–version. Scan through them and choose to dive deeper into stories that reflect challenges that your company or your industry is facing. You’ll find solutions to problems you previously thought impossible to solve.
4890
Of your peers have already downloaded this article
1:30 Minutes
The most insightful time you'll spend today!
Artificial intelligence and machine learning are already transforming the technological landscape. From digital assistants to image-recognition software to self-driving cars, what was once the stuff of science fiction is now becoming a reality. But what exactly does it mean for marketing and advertising executives?
It could get us closer to one of advertising’s most-sought goals: relevance at scale. Before then, we’re going to see changes to the way we do business.
Technological advances have always created new opportunities for storytelling and marketing. Just as the advent of TV brought an era of truly mass advertising and reach, and the internet and mobile brought a new level of targeting and context, AI will change how people interact with information, technology, brands, and services.
A big part of the opportunity for marketers is how AI will help us fully realize personalization—and relevance—at scale. With platforms like Search and YouTube reaching billions of people everyday, digital ad platforms finally can achieve communication at scale. This scale, combined with customization possible through AI, means we’ll soon be able to tailor campaigns to consumer intent in the moment. It will be like having a million planners in your pocket.
Find out how you can achieve relevance at scale. Download now!
5360
Of your peers have already downloaded this article
8:15 Minutes
The most insightful time you'll spend today!
To design and implement a successful machine learning (ML) project, you often need to collaborate with multiple teams, including those in business, sales, research, and engineering. Understanding the essentials of gathering and preparing your data is crucial to align teams, and getting a project off the ground.
Building an ML model that achieves your business’ goal, requires the kind of data that a model can learn from. If the goal is to predict a target output based on some input, you’ll need some data consisting of past input-output examples. For example, to detect (that is, to predict) a fraudulent credit card transaction (the target of the prediction), each example must contain information on a past credit card transaction and whether that transaction was fraudulent or not.
So what are the best practices in preparing and curating data for a machine learning initiative? The answer to this question lies within the guide. You’ll learn:
- To evaluate whether your data is suitable for ML
- What types of data you need–and which ones to eliminate
- How to prevent data leakage
- Tricks to speed up an ML project such as Google’s human labeling service, and Cloud Dataprep
- The importance of the granularity of data
6171
Of your peers have already watched this video.
18:00 Minutes
The most insightful time you'll spend today!
An Overview of Google’s Data Cloud
Data access, management and privacy has been at the center of priorities for enterprises that are aiming to be more agile, reliable and data-driven. Google Cloud’s technology innovations spanning products like BigQuery, Spanner, Looker and VertexAI help organizations navigate the complexities related to siloed data in large volumes sprawled across databases, data lakes, data warehouses, and data marts in multiple clouds and on-premises. Watch the video to learn how companies are building data on Google Cloud for better analysis, security and management to achieve bottomline!
6434
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
Value Realization with Google Cloud for Retail SAP Data
Retail engagements have changed drastically over the last few years, and by the virtue of COVID-19 pandemic, retail data and customers’ expectations plummeted. To drive value and transformation across the entire value-chain retailers can make most of Google Cloud’s secure, reliable IaaS by migrating their SAP systems and taking advantage of integrations, insights and innovations. In times of change retail companies can gain maximum visibility of SAP data unlocking Google Cloud’s infrastructure modernization and Big Data and analytics capabilities. Watch the video to understand how Google Cloud and SAP partnership is a golden handshake for retail businesses’ future.
More Relevant Stories for Your Company
Google Cloud Helps LiveRamp Capture, Manage, Process and Visualize Data at Scale
Editor’s note: Today we’re hearing from Sagar Batchu, Director of Engineering at LiveRamp. He shares how Google Cloud helped LiveRamp modernize its data analytics infrastructure to simplify its operations, lower support and infrastructure costs and enable its customers to connect, control, and activate customer data safely and securely. LiveRamp is a data
Google had Enough of Flooding in India. Here’s What It Did About It
Over the last century, floods have become the most common and deadly natural disaster on the planet. Many countries currently lack effective early warning systems and alerts. Today, 20 percent of flood fatalities occur in India. To help, Google sent a team to study the Ghaghara River, near Patna. “In
The New and Upcoming Infrastructure for Google Cloud’s AI and ML Solutions
How does Google manage to provide its customers a differentiated compute platform experience and define ways to fully leverage its infrastructure supporting its cutting-edge AI and ML offerings? Easy-to-use, scalable and ability to create innovative products and services to end-users at low cost of ownership is the narrative behind Google
Rubin Observatory Leverages Google Cloud to Power Astronomical Research
This week, the Vera C. Rubin Observatory is launching the first preview of its new Rubin Science Platform (RSP) for an initial cohort of astronomers. The observatory, which is located in Chile but managed by the U.S. National Science Foundation’s NOIRLab in Tucson, AZ and SLAC in California, is jointly funded by the NSF and the U.S. Department
