The Home Depot (THD) is the world’s largest home-improvement chain, growing to more than 2,200 stores and 700,000 products in four decades. Much of that success was driven through the analysis of data. This included developing sales forecasts, replenishing inventory through the supply chain network, and providing timely performance scorecards.

However, to compete in today’s business world, THD has taken this data-driven approach to an entirely new level of success on Google Cloud, providing capabilities not practical on legacy technologies.

The pressures of contemporary growth that drove much of the work are familiar to many businesses. In addition to everything it was doing, THD needed to better integrate the complexities in its related businesses, like tool rental and home services. It needed to better empower teams, including a fast-growing data analysis staff and store associates with mobile computing devices. It wanted to better use online commerce and artificial intelligence to meet customer needs, while maintaining better security.

Even before addressing these new challenges, THD’s existing on-premises data warehouse was under stress as more data was required for analytics and data analysts were utilizing the data with increasingly complex use cases. This drove rapid growth of the data warehouse, but also created constant challenges for the team in managing priorities, performance, and cost.

In order to add capacity to the environment, it was a major planning, architecture, and testing effort. In one case, adding on-premises capacity took six months of planning and a three-day service outage. Within a year, capacity was again scarce, impacting performance and ability to execute all the reporting and analytics workloads required. The capacity refresh cycles were shrinking, and the expecations for data were growing. There had to be a better way.

Still, THD did not take its move to the cloud lightly. A large-scale enterprise data warehouse migration involves tremendous effort among people, process, and technology. After careful consideration, THD chose Google Cloud’s BigQuery for its cloud enterprise data warehouse.

BigQuery, a scalable serverless data warehouse, was better on cost, infrastructure agility, and analytics capability, driving better insights with improved performance. There are no service interruptions when capacity is added, and that capacity can be added within a week (and soon same day). It doesn’t require complex system administration, and its standard SQL support means people can easily ramp up quickly. Valuable BigQuery products like Identity and Access Management meant THD could create many separate Google Cloud projects, while ensuring that different teams weren’t interfering with each other or accessing protected data.

THD also utilizes BigQuery’s flat-rate monthly pricing model that allows teams to budget their capacity based on need and provides billing predictability. The capacity not being used by a given project is available for enterprise use. This ensures no surprises when the monthly bill arrives and provides all analytical users access to significant computing power.

While THD’s legacy data warehouse contained 450 terabytes of data, the BigQuery enterprise data warehouse has over 15 petabytes. That means better decision-making by utilizing new datasets like website clickstream data and by analyzing additional years of data.

As for performance, look at this chart:

With the cloud EDW migration complete, and the legacy on-premises data warehouse retired, analysts now execute more complex and demanding workloads that they would not have been able to complete before, such as utilizing Datalab for orchestrating analytics through Python Notebooks, utilizing BigQuery ML for machine learning directly against the BigQuery data (no movement of large datasets), and AutoML to help determine the best model for predictions.

Additionally, engineers at THD have adapted BigQuery to monitor, analyze, and act on application performance data across all its stores and warehouses in real time, something that was not practical in the on-premises system.

With over 600 projects that THD now has on Google Cloud, the BigQuery story is just one of the many ways that Google Cloud is working with THD to deliver meaningful business results, every day.

The city of South Bend, Indiana has announced they are working with Google Cloud to connect 500 local job seekers with no-cost access to Google Cloud Skills Boost through Upskill SB. This initiative will train, upskill, and certify residents in cloud computing skills to create pathways to technical jobs.

As companies, nonprofits, and governments increasingly leverage cloud computing technology to strengthen their organizations, skilled cloud talent is in high demand both nationally, in Indiana, and in South Bend. According to Lightcast, there were 27,180 unique cloud roles open in the U.S. as of October 2022. Cloud skills are the most in-demand skill set by IT departments: more than one-third of tech learners said professionals with cloud skills are the most challenging to find.

“This partnership with Google Cloud will help residents get on a no-cost path to a cloud computing career at a time when the field is rapidly growing,” said Caleb Bauer, Executive Director of Community Investment for the City of South Bend. “This will help residents prepare to take the next step in their career with Google Cloud Skills Boost and will pave the way to build in-demand skills.”

Google Cloud Skills Boost is the destination for all Google Cloud learning and training opportunities. Local colleges and universities are also able to offer free access to Google Cloud Skills Boost for students through  the Google Cloud Higher Education program. Students can request access to free online labs, quests, and courses. Faculty can take advantage of free Google Cloud Computing Foundations curriculum and coach their students to prepare for Google Cloud certifications.

“Google is proud to offer no-cost access to Google Cloud Skills Boost for jobseekers in South Bend,” said Alice Kamens, Workforce Development lead at Google Cloud. “Through this partnership, we’re advancing our work to ensure everyone has access to the technical training needed to pursue cloud computing careers.”

Those interested in learning more about the program or applying for Upskill SB should visit the Google Cloud page on Upskill SB. Scholarship recipients need access to a computer, hand-held device or smartphone and the internet.

Kohl’s is an omnichannel retailer focused on driving traffic, operational efficiency and delivering seamless omnichannel customer experiences.

Ratnakar Lavu is Kohl’s Senior Executive Vice President and Chief Technology Officer. He, and Kohl’s, are at the forefront of retail technology innovation, focusing on a frictionless customer journey across digital, mobile and more than 1,150 stores.

As part of this journey to more closely unify its online and offline experiences for customers, the company was looking for supporting cloud services that would continue to drive  best-in-class data center infrastructure; the ability to manage data at a very large scale; and industry leading analytics and machine learning tools to continually understand real time data streams and help personalize experiences for their customers.

Kohl’s recognized the opportunity to take on a cloud partner to help drive the improvement of the speed and reliability of their operations, while they focused on a number of innovations to deepen customer experiences.

“At the time, I was looking for an open and scalable platform to partner with our Kohl’s technology team as we transform our business by shifting to the cloud,” says Ratnakar.

“Google has great engineering talent as well as demonstrated experience solving stability and scale in its own Ads and Search business. At Kohl’s, we need to be bold and innovative in today’s retail environment, and therefore need partners who deeply understand how to manage risk.” 

Kohl’s leveraged several capabilities of Google Cloud. For example:

• They built applications to automate deployment, scaling and operations.
• They used monitoring capabilities to monitor for things like response time.
• Scalable technology provided an infrastructure to elastically scale to site traffic.
• They ran their infrastructure across multiple regions for high availability.

In 2017 and 2018, record-setting numbers of customers visited Kohls.com during the Thanksgiving holiday weekend and the digital platform experienced high double-digit growth both years.

The capabilities provided by Google Cloud Platform (GCP) and Google’s data center infrastructure supported Kohl’s servers and systems during these key timeframes.

In addition, the Kohl’s team partnered together with Google’s core engineering team and services organization to optimize applications and make them more reliable.

Google Cloud’s Customer Reliability Engineers (CREs) worked with them in advance of their peak time frames to test the infrastructure for performance, scaling, and fault tolerance.

“Google CRE and services teams collaborated with us as we ran drills and exercises during each phase of preparation for peak time frames,” Ratnakar said. “As we continued to understand better how to scale, monitor, and support our applications in GCP and we are pleased that we worked with the CRE team as partners on monitoring services, alerting teams, and triaging work.”

Whether they were ready for it or not, the COVID-19 pandemic transformed many retailers into digital businesses. Retailers made huge investments into commerce technologies, customer experience tools, sales and fulfillment technology, and improving digital experiences to continue providing their goods and services to their customers. Now, more than a year into the COVID-19 pandemic, digital retail is the new normal. In fact, many retailers are planning on expanding their digital investments. However, as their digital footprint expands, so do their threats and security concerns. 

As a digital-focused retailer, your website is the most visible part of your attack surface. Your website is where your customers search for goods or services, make payments, or learn more about your brand. However, your website does not operate in isolation. There is an underlying infrastructure as well as the services that run on top of it that need protection from a wide array of attacks that may seek to compromise your data, internal employees, business, and customers. During this week’s Google Cloud Retail Summit, we’ve shared why Google Cloud is built to be the most trusted cloud for retailers. From providing you with control over your data as you move from your own data centers to the public cloud to giving you built-in technology to protect your applications all the way to your end users, Google Cloud helps you safely migrate to and operate in our Trusted Cloud.

Trusted Cloud Gives You Control, Transparency, and Sovereignty

• Access Transparency: We offer the ability to monitor and approve access to your data or configurations by Google Cloud support and engineering based on specific justifications and context, so you have visibility and control over insider access. 
• Certificate Authority Service (CAS): CAS is a highly scalable and available service that simplifies and automates the management and deployment of private CAs while meeting the needs of modern developers and applications. With CAS, you can offload to the cloud time-consuming tasks associated with operating a private CA, like hardware provisioning, infrastructure security, software deployment, high-availability configuration, disaster recovery, backups, and more, allowing you to stand up a private CA in minutes, rather than the months it might normally take to deploy.

• Confidential Computing: We already encrypt data at-rest and in-transit, but customer data needs to be decrypted for processing. Confidential Computing is a breakthrough technology which encrypts data in-use—while it’s being processed. Confidential VMs take this technology to the next level by offering memory encryption so that you can further isolate your workloads in the cloud. With the beta launch of Confidential VMs, we’re the first major cloud provider to offer this level of security and isolation while giving you a simple, easy-to-use option for your newly built and “lift and shift” applications.

• Cloud Key Management: We allow you to configure the locations where your data is stored, where your encryption keys are stored, and where your data can be accessed from. We give you the ability to manage your own encryption keys, even storing them outside Google’s infrastructure. Using our External Key Management service, you have the ability to deny any request by Google to access encryption keys necessary to decrypt customer data at rest for any reason.

Trusted cloud Helps You Prevent, Detect, and Respond to Threats

• BeyondCorp Enterprise is Google’s comprehensive zero trust product offering. Google has over a decade of experience managing and securing cloud applications at a global scale, and this offering was developed based on learnings from our experience managing our own enterprise, feedback from customers and partners, as well as informed by leading engineering and security research. We understand that most customers host resources across different cloud providers. With this in mind, BeyondCorp Enterprise was purpose-built as a multicloud solution, enabling customers to securely access resources hosted not only on Google Cloud or on-premises, but also across other clouds such as Azure and Amazon Web Services (AWS).

• Cloud Armor: We’re simplifying how you can use Cloud Armor to help protect your websites and applications from exploit attempts, as well as Distributed Denial of Service (DDoS) attacks. With Cloud Armor Managed Protection Plus, you will get access to DDoS and WAF services, curated rule sets, and other services for a predictable monthly price.

• Chronicle: Chronicle is a threat detection solution that identifies threats, including ransomware, at unparalleled speed and scale. Google Cloud Threat Intelligence for Chronicle surfaces highly actionable threats based on Google’s collective insight and research into Internet-based threats. Threat Intel for Chronicle allows you to focus on real threats in the environment and accelerate your response time.
• Google Workspace Security: Used by more than five million organizations worldwide, from large banks and retailers with hundreds of thousands of people to fast-growing startups, Google Workspace and Google Workspace for Education include the collaboration and productivity tools found here. Google Workspace and Google Workspace for Education are designed to help teams work together securely in new, more efficient ways, no matter where members are located or what device they use. For instance, Gmail scans over 300 billion attachments for malware every week and prevents more than 99.9% of spam, phishing, and malware from reaching users. We’re committed to protecting against security 1 threats of all kinds, innovating new security tools for users and admins, and providing our customers with a secure cloud service.
• Identity & Access Management IAM: Identity and Access Management (IAM) lets administrators authorize who can take action on specific resources, giving you full control and visibility to manage Google Cloud resources centrally. For enterprises with complex organizational structures, hundreds of workgroups, and many projects, IAM provides a unified view into security policy across your entire organization, with built-in auditing to ease compliance processes.
• reCAPTCHA Enterprise: reCAPTCHA has over a decade of experience defending the internet and data for its network of more than 5 million sites. reCAPTCHA Enterprise builds on this technology with capabilities, such as two-factor authentication and mobile application support, designed specifically for enterprise security concerns. With reCAPTCHA Enterprise, you can defend your website against common web-based attacks like credential stuffing, account takeovers, and scraping and help prevent costly exploits from malicious human and automated actors. And, just like reCAPTCHA v3, reCAPTCHA Enterprise will never interrupt your users with a challenge, so you can run it on all webpages where your customers interact with your services.

• Security Command Center: With Security Command Center (SCC), our native posture management platform, you can prevent and detect abuse of your cloud resources, centralize security findings from Google Cloud services and partner products, and detect common misconfigurations, all in one easy-to-use platform. We have Premium tier for Security Command Center to provide even more tools to protect your cloud resources. It adds new capabilities that let you spot threats using Google intelligence for events in Google Cloud Platform (GCP) logs and containers, surface large sets of misconfigurations, perform automated compliance scanning and reporting. These features help you understand your risks on Google Cloud, verify that you’ve configured your resources properly and safely, and document it for anyone who asks.

• VirusTotal: VirusTotal inspects items with over 70 antivirus scanners and URL/domain blocklisting services, in addition to a myriad of tools to extract signals from the studied content. Any user can select a file from their computer using their browser and send it to VirusTotal.
• Web Risk API: With Web Risk, you can quickly identify known bad sites, warn users before they click infected links, and prevent users from posting links to known infected pages from your site. Web Risk includes data on more than a million unsafe URLs and stays up to date by examining billions of URLs each day.

Trusted cloud Plays an Active Role in Our Shared Fate 

Our trusted cloud provides a shared-fate model for risk management. We stand with retailers from day one, helping them implement best practices for safely migrating to and operating in our trusted cloud.

We hope you enjoy the sessions we’ve created for you with Google Cloud Retail Summit and that they help you understand the ways our trusted cloud can help secure retailers all over the world.

May was a very busy month for data analytics product innovation. If you didn’t have the chance to attend our inaugural Data Cloud Summit, video replays of all our sessions are now available so feel free to watch them at your own pace. 

In this blog, I’d like to share some background behind the innovations we released in May, why we built them the way we did, and the type of value they can bring your company and your team.

But first, a huge thank you!

This week, we had the honor to announce that Google has been named a Leader in The Forrester Wave™: Streaming Analytics, Q2 2021 report. Forrester gave Dataflow a score of 5 out of 5 across 12 different criteria, stating: “Google Cloud Dataflow has strengths in data sequencing, advanced analytics, performance, and high-availability”. 

Google has more than a decade of experience in building real-time and internet-scale systems for its own needs, and we are excited to see that our ability to provide customers with a reliable, scalable, and performant platform is bearing fruit. 

This announcement comes on the back of the release of The Forrester Wave™: Cloud Data Warehouse, Q1 2021 report, which also named Google Cloud as a Leader.

We couldn’t be more excited about the recognition and appreciate all your feedback and trust in the work that we do to support your goal in accelerating data-powered innovation.

Innovation galore

Your feedback and your passion is the fuel that drives our ambition to deliver more and better services to you. That’s why, this year, we didn’t want to wait until Google Cloud Next to share some great products we have been working on. On May 26, our team announced a slew of new products, services and programs. Watch a quick summary below:

https://youtube.com/watch?v=DG1mOPMXJvw%3Fenablejsapi%3D1%26

Meeting you where you are

An important design principle behind all of our services is “meeting you where you are”. This means we aim to provide you with the tools and software you need to innovate on your own terms. Here are three new services that will help you do just that:

Datastream

Datastream, our new serverless change data capture (CDC) and replication service, allows your company to synchronize data across heterogeneous databases, storage systems, and applications reliably and with minimal latency to support real-time analytics, database replication, and event-driven architectures. Datastream delivers change streams from Oracle and MySQL databases into Google Cloud services such as BigQuery, Cloud SQL, Cloud Storage, and Cloud Spanner, saving time and resources while ensuring your data is accurate and up-to-date. 

• Under the hood, Datastream reads CDC events (inserts, updates, and deletes) from source databases, and writes those events with minimal latency to a data destination. It leverages the fact that each database source has its own CDC log—binlog for MySQL and LogMiner for Oracle—which it uses for its own internal replication and consistency purposes. 
• Datastream integrates with purpose-built and extensible Dataflow templates to pull the change streams written to Cloud Storage, and create up-to-date replicated tables in BigQuery for analytics. It also leverages Dataflow templates to replicate and synchronize databases into Cloud SQL or Cloud Spanner for database migrations and hybrid cloud configurations. 
• Datastream also powers a Google-native Oracle connector in Cloud Data Fusion’s new replication feature for easy ETL/ELT pipelining. By delivering change streams directly into Cloud Storage, customers can leverage Datastream to implement modern, event-driven architectures.

Looker and BigQuery Omni on Microsoft Azure

Research on multi cloud adoption is unequivocal — 92% of businesses in 2021 report having a multi cloud strategy. We want to continue supporting your choice by providing the flexibility you need to see your strategy through. 

• This past month, we introduced Looker, hosted on Microsoft Azure. For the first time, you can now choose Azure, Google Cloud, or AWS for your Looker instance. You can also self-host your Looker instance on-premises.
• We also introduced BigQuery Omni for Azure, which along with last year’s introduction of BigQuery Omni for AWS, will help you access and securely analyze data across Google Cloud, AWS, and Azure. 

The cost of moving data between cloud providers isn’t sustainable for many, and it’s still difficult to seamlessly work across clouds. BigQuery Omni represents a new way of analyzing data stored in multiple public clouds, which is made possible by BigQuery’s separation of compute and storage. By decoupling these two, BigQuery provides scalable storage that can reside in Google Cloud or other public clouds, and stateless resilient compute that executes standard SQL queries. 

• Unlike competitors, BigQuery Omni doesn’t require you to move or copy your data from one public cloud to another, where you might incur egress costs. You also benefit from the same BigQuery interface on Google Cloud, enabling you to query data stored in Google Cloud, AWS, and Azure without any cross-cloud movement or copies of data. 
• BigQuery Omni’s query engine runs the necessary compute on clusters in the same region where your data resides. For example, you can query Google Analytics 360 Ads data stored in Google Cloud and query logs data from your ecommerce platform and applications that are stored in AWS S3 and/or Microsoft Azure. 

Then, using Looker, you can build a dashboard that allows you to visualize your audience behavior and purchases alongside your advertising spend. 

Dataplex

We understand that most organizations still struggle to make high-quality data easily discoverable and accessible for analytics, across multiple silos, to a growing number of people and tools within their organization. 

They are often forced to make tradeoffs. For instance, moving and duplicating data across silos to enable diverse analytics use cases or leaving their data distributed but limiting the agility of decisions. 

• Dataplex provides an intelligent data fabric that enables you to centrally manage, monitor, and govern your data across data lakes, data warehouses, and data marts, while also ensuring data is securely accessible to a variety of analytics and data science tools. 
• One of the core tenets of Dataplex is letting you organize and manage your data in a way that makes sense for your business, without data movement or duplication. For that, we provide logical constructs like lakes, data zones, and assets. These constructs enable you to abstract away the underlying storage systems and become the foundation for setting policies around data access, security, lifecycle management, and so on. 
• For example, you can create a lake per department within your organization (e.g. Retail, Sales, Finance, etc.) and create data zones that map to data readiness and usage (e.g. landing, raw, curated_data_analytics, curated_data_science, etc.). 

Once you have your lakes and zones setup, you can attach data to these zones as assets. You can add data from different types of storage (e.g. GCS Bucket and BigQuery dataset) under the same zone. You can also attach data across multiple projects under the same zone. You can ingest data into your lakes and zones using the tools of your choice, including services such as Dataflow, Data Fusion, Dataproc, Pub/Sub, or choose from one of our partner products. Dataplex comes with built-in 1-click templates for common data management tasks. 
To find out more about Dataplex, head to cloud.google.com/dataplex or watch the video below:

https://youtube.com/watch?v=bbFeAt7cw1g%3Fenablejsapi%3D1%26

Helping you innovate everyday

Sharing data is hard. Traditional data sharing techniques use batch data pipelines that are expensive to run, create late arriving data, and can break with any changes to the source data. These techniques also create multiple copies of data, which brings unnecessary costs and can bypass data governance processes. They also fail to offer features for data monetization, such as managing subscriptions and entitlements. Altogether, these challenges mean that organizations are unable to realize the full potential of transforming their business with shared data.

Analytics Hub

To address these limitations, we are introducing Analytics Hub, a new fully managed service that helps organizations unlock the value of data sharing, leading to new insights and increased business value. 

This new service is built on the tremendous experience and feedback we have received over the years. For example, BigQuery has had cross-organizational, in-place data sharing capabilities since its inception in 2010—and the functionality is very popular. Over a 7-day period in April, we had over 3,000 different organizations sharing over 200 petabytes of data. These numbers don’t include data sharing between departments within the same organization.

Analytics Hub takes sharing to the next level, making it easy for you to publish, discover, and subscribe to valuable datasets that you can combine with your own data to derive unique insights. 

This includes: 

• Shared datasets: As a data publisher, you create shared datasets that contain the views of data that you want to deliver to your subscribers. Data subscribers can search through the datasets that are available across all exchanges for which they have access and subscribe to relevant datasets. In addition, the publisher can track subscribers, disable subscriptions, and see aggregated usage information for the shared data.
• Curated, self-service data exchanges: Exchanges are collections used to organize and secure shared datasets. By default, exchanges are completely private, but granular roles and permissions make it easy to deliver data to the right audience—whether internal or public. 

This is just the beginning for Analytics Hub. Please sign up for the preview, which is scheduled to be available in the third quarter of 2021.

Dataflow Prime

At Google Cloud, we have the great privilege of working with some of the most innovative organizations in the world. And this work provides us with a unique perspective into the future of big data processing. Dataflow Prime is a new platform based on a serverless, no-ops, and auto-tuning architecture that brings unparalleled resource utilization and radical operational simplicity to big data processing. This new service introduces a large number of exciting capabilities but I’d like to highlight three key aspects of the product:

• Vertical Autoscaling: Dataflow Prime dynamically adjusts the compute capacity allocated to each worker based on utilization, detecting when jobs are limited by worker resources and automatically adding more resources. Vertical Autoscaling works hand in hand with Horizontal Autoscaling to seamlessly scale workers to best fit the needs of the pipeline. As a result, it no longer takes hours or days to determine the perfect worker configuration to maximize utilization. 
• Right Fitting: Each stage of a pipeline typically has a different resource requirement than the others. Until now, either all workers in the pipeline would have had the higher memory and GPU, or none of them would. Pipelines either had to waste resources or suffer slower workloads. Right Fitting solves this problem by creating stage-specific pools of resources, optimized for each stage. 
• Smart Recommendations: Smart Recommendations automatically detects problems in your pipeline and shows potential fixes. For example, if your pipeline is running into permissions issues, a Smart Recommendation will detect which IAM permissions you need to enable to unblock your job. If you are using an inefficient coder in your job, Smart Recommendations will surface more performant coder implementations that can help you save on costs.

What’s next

We’re excited to hear your thoughts and feedback about all these exciting new services. I would also highly recommend that you connect with members of the community to learn more about their story and journey. A good example to start with is the Data To Value customer panel we produced at our inaugural Data Cloud Summit with the Chief Data Officers of Keybank and Rackspace. You can watch it for free below:

https://youtube.com/watch?v=ITI2Q3MkxuA%3Fenablejsapi%3D1%26