Large Machine Learning (ML) models – such as large language models, generative AI, and vision models – are dramatically increasing the number of trainable parameters and are achieving state-of-the-art results. Increasing the number of parameters results in the model being too large to fit on a single VM instance thus demands distributed compute to spread the model across multiple nodes. Google Kubernetes Engine (GKE) has built-in support for NCCL Fast Socket, to help improve the time to train large ML models with distributed, multi-node clusters.

Enterprises are looking for faster and cheaper performance to train their ML models. With distributed training, communicating gradients across nodes is a performance bottleneck. Optimizing inter-node latency is critical to reduce training time and costs. Distributed training uses collective communication as a transport layer over the network between the multiple hosts. Collective communication primitives such as all-gather, all-reduce, broadcast, reduce, reduce-scatter, and point-to-point send and receive are used in distributed training in Machine Learning. 

The NVIDIA Collective Communication Library (NCCL) is commonly used by popular ML frameworks such as TensorFlow and PyTorch. It is a highly optimized implementation for high bandwidth and low latency between NVIDIA GPUs. Google developed a proprietary version of NCCL called NCCL Fast Socket to optimize performance for deep learning on Google Cloud.

NCCL Fast Socket uses a number of techniques to achieve better and more consistent NCCL performance.

• Use of multiple network flows to attain maximum throughput. NCCL Fast Socket introduces additional optimizations over NCCL’s built-in multi-stream support, including better overlapping of multiple communication requests.
• Dynamic load balancing of multiple network flows. NCCL can adapt to changing network and host conditions. With this optimization, straggler network flows will not significantly slow down the entire NCCL collective operation.
• Integration with Google Cloud’s Andromeda virtual network stack.This increases overall network throughput by avoiding contentions in virtual machines (VMs).

We tested (NVIDIA NCCL tests) the performance of NCCL Fast Socket vs NCCL on various machine shapes with 2 node GKE clusters.

The following chart shows the results. For each machine shape, the NCCL performance without Fast Socket is normalized to 1. In each case, using NCCL Fast Socket demonstrated increased performance in a range of 1.3 to 2.6 times faster internetwork communication speed.

As a built-in feature, GKE users can take advantage of NCCL Fast Socket without changing or recompiling their applications, ML frameworks (such as TensorFlow or PyTorch), or even the NCCL library itself. To start using NCCL Fast Socket, create a node pool that uses the plugin with the --enable-fast-socket and --enable-gvnic flags. You can also update an existing node pool using gcloud container node-pools update.

gcloud container node-pools create NODEPOOL_NAME \
    --accelerator type=ACCELERATOR_TYPE, count=ACCELERATOR_COUNT \
    --machine-type=MACHINE_TYPE \
    --cluster=CLUSTER_NAME \
    --enable-fast-socket \
    --enable-gvnic

To achieve better network throughput with NCCL, Google Virtual NICs (gVNICs) must be enabled when creating VM instances. For detailed instructions on how to use gVNICs, please refer to the gVNIC guide. 

To verify that NCCL Fast Socket has been enabled, view the kube-system pods:

kubectl get pods -n kube-system

And the output should b similar to:

NAME                         READY   STATUS    RESTARTS   AGE
fast-socket-installer-qvfdw  2/2     Running   0          10m
fast-socket-installer-rtjs4  2/2     Running   0          10m
fast-socket-installer-tm294  2/2     Running   0          10m

To learn more visit GKE NCCL Fast Socket documentation. We look forward to hearing how NCCL Fast Socket improves your ML Training experience on GKE.

About three years ago, JCB, one of the biggest Japanese payment companies, launched a project to develop new high-value services with agility. We set up a policy of starting small from scratch without using the existing system, which we call the concept of “Dejima”, where we focused on improving various aspects such as team structure, risk management, and application and platform development process.

Until now, large Japanese enterprises have built decision-making systems focused on eliminating unnecessary business processes and efficiently increasing quarterly profits. As a result, we are seeing more organizational structures that make it difficult to take on new challenges or experiments with trial and error. We wanted to breathe a new life into this situation, and that is how the concept of Dejima came up. In the Edo period, Japan closed its national border to other countries under its national isolation policy. At the time, Dejima was the only area where special rules were applied and allowed people from different cultures to come and go, and trade. This special rule generated the culture of inclusion and led to Dejima’s prosperity. Like Dejima, we believe that creating an organization that is independent from other business practices can be effective in enabling digital transformation for the organization. 

We have been able to make this transformation with the direct help of the Google Cloud and its products such as Google Kubernetes Engine (GKE), Cloud Spanner and Anthos Service Mesh, applying domain-driven design and microservice architecture. We named this the “JCB Digital Enablement Platform (JDEP),” which now hosts multiple business critical production services.

A key benefit of GKE is that the team can easily add resources and release them when they are finished, allowing them to be flexible to accommodate busy periods and off-seasons. Meanwhile, Anthos Service Mesh helps us manage complex environments easily. With containerization and managed services, we are prepared for the future for when more services go into production, as it would be easy to maintain and provide version upgrade support. At the same time, Cloud Spanner ensures that we maintain a 99.99% availability at all times.

Our initial motivation for introducing SRE practices was to break proverbial walls between business, development and operations, which was a success. Now we are focused on ensuring its reliability and maintaining customer satisfaction with our SRE practices.

To ensure the success of SRE practices that we implemented, there were a few categories we needed to address, from defining the organizational culture and practices to ensuring the policies attached to the new models created were practical enough to be implemented on the ground level. This is so that the Dejima concept remains sustainable for the long run.  

Instilling a culture of measurement 

Here, “appropriate” reliability is the key. According to the conventional way of thinking at JCB, “service failure must not occur” and “SLA should be maintained as high as possible.” We started by discussing what was the “appropriate reliability” that our customers really needed, but it was not as easy as we thought because the level of reliability for user satisfaction differed from application to application. 

Eventually, the business, development and operations teams formulated specific SLIs and SLOs together, something we would never have been able to do if we discussed separately. This is because the business is required to compromise on lower service levels, since our reliability standard used to be too high. The collaboration of development and operations teams is necessary to understand how our system works upon our users’ interactions.

After Google Cloud helped us run a series of workshops where all teams participated, we saw change within the organization. The business team started evangelizing SRE to other members in the business department, and the development and operations teams started collaborating autonomously. We felt like we were working at Google speed, accomplishing so much in a short amount of time. 

Understanding SRE as an entire company is necessary to progress. We are now working on creating internal training materials to spread the SRE concept throughout the company.

Eliminating ambiguity

With the cooperation of Google Cloud, we have created a Team Charter that defines the team’s mission, values and engagement models. We also created policy documents that include Incident Response Policy, Postmortem Policy, On-call Policy, Toil Policy and Error Budget Policy, to eliminate ambiguity in day-to-day operations.

For example, when an incident occurs, we can identify exactly the level of importance, the roles that are assigned to each person, and in what order they need to follow. When to do a postmortem, who owns it? What to do if the error budget is exhausted? How do other teams reach out to SRE when they have problems? The written policy documents will dramatically improve efficiency and motivate teams to adopt a culture of learning from failures.

The format for such policies are written in Google’s SRE book, but when we adopt it, it needs to take into account the circumstances specific to our company. Simply copying an existing policy won’t work, which is why it’s important to formulate a policy that fits the situation each team is in.

Reformalizing teams

Based on these policies, JCB’s SRE team has two sub-teams. One is called Sheriff which works as the platform SRE, and provides infrastructure services for the application team. The other is called the Diplomat which works as the embedded SRE, and participates in the application team to lead productionisation. There is also a team called Architecture that is separate from the SRE teams whose role is to consult SRE on system design and review architecture.

The SRE team was a single role when it was first launched, but now has two sub-teams. This is because as the number of application teams increases, the number of support tasks for the teams also increases, which can result in a shortage of resources to work on overall improvement. Securing people who are not interrupted from day-to-day support tasks and focusing on the main task improves efficiency.

Whereas both sub-teams share the on-call duty, some engineers are not allowed to do it by contract as they are not allowed to get paged. For those who cannot participate in on-call duties, we created what’s called a Toil Shift, which allows them to focus on resolving tickets in our backlogs instead.

This works well so far, but we will keep evolving as our business grows.

As consumer expectations grow for more personalized, relevant, and assistive experiences, machine learning is becoming an invaluable tool to help meet those demands.

It’s helping marketers create smarter customer segmentations, deliver more relevant creative campaigns, and measure performance more effectively. In fact, 85% of executives believe AI will allow their companies to obtain or sustain a competitive advantage.¹

We created this guide to help you optimize your machine learning marketing efforts — whether you’re just starting out or you want to discover more benefits of machine learning.

Go to the Guide Now!

Editor’s note: To kick off the new year and in preparation for NRF The Big Show, we invited partners from across our retail ecosystem to share stories, best practices, and tips and tricks on how they are helping retailers transform during a time that continues to see tremendous change. Please enjoy this entry from our partner.

If you put a pot of water on the stove, it doesn’t heat up instantly. It simmers slowly at first, eventually picking up steam to reach a rolling boil. The retail landscape is not so different. Capgemini’s research shows the sector has evolved over four generations, from the early days of fragmented outlets to omnichannel and customer-centric focuses, with a fifth generation on the horizon that promises to be centered on consumption.

• Generation 1: Fragmented outlets
• Generation 2: Chain concentration
• Generation 3: Omnichannel
• Generation 4: Consumer-centric
• Generation 5: Consumption-centric

Although incremental change allows companies to experiment and iterate during their digital journeys, the COVID-19 pandemic rapidly accelerated the evolution of online and contactless shopping. Evolution became a revolution, with most Consumer Product and Retail (CPR) companies still mastering the omnichannel generation of their digital transformation to create a seamless shopping experience. Companies that are more digitally mature are already aspiring to the consumer-centric phase, embracing opportunities made possible by technology such as personalization and automation.

What consumers want

Customers have more choices in how they shop and engage with brands. This has made it harder for brands to predict and anticipate needs across customer journeys. And that’s convincing some companies to innovate more quickly as consumer demand drives the need for speed and scale.

Think about your own online behavior. Say you’re shopping for an item and searching online for the closest store in your neighborhood. Google is likely your go-to for finding that information. Looking to troubleshoot an issue with a product or seek out a service? Again, you’ll likely hit up Google first, not even considering going directly to a brand’s website for answers.

Both scenarios point to a disconnect between virtual and physical worlds, a gap technology can bridge in numerous ways such as breaking down silos and integrating fragmented media channels. Interestingly, though, not everything will be centered online all the time. In our recent study on consumer behavior, The great consumer reset, Capgemini discovered 57 percent of shoppers plan to return to brick-and-mortar stores post-pandemic, which is basically unchanged from the 59 percent who often interacted with physical stores before.

But business as usual? Not even close. Consumers have come to expect a frictionless shopping experience (buy online, pick up in store) or an immersive one (products displayed online using augmented reality), and are not content to return to in-store lineups, empty shelves, or a one-size-fits-all approach. Moreover, customers want personalized interactions while ensuring their data and privacy are protected.

So the role of the store is changing. In fact, many online-only brands are opening brick-and-mortar establishments to drive customer experience. In our research on “smart stores,” we found the majority of consumers (66 percent) believe automation can improve their shopping experience by solving the challenges they face at retail stores.

From personalization to serendipity

Retailers must recognize that they have to win consumer trust and confidence. Many consumers believe retailers’ use of tech is focused on reducing costs rather than easing friction. And they’re right. That same Capgemini research found that only one-third (35 percent) of retailers consider “solving customer pain points” as the most important criteria when deciding which automation use cases to implement.

“Retailers are largely in the early stages of adopting automation, and that’s an opportunity to rethink how they’re using technology, not just to smooth out friction and engender consumer trust but to build unexpected consumer benefits,” says Neerav Vyas, Head of Customer First, Co-Chief Innovation Officer, Insights & Data, North America, Capgemini. “We’re trying to move towards this idea of delivering serendipitous experiences to bridge the physical and digital divide.”

The focus is not solely on shoppers seeking out a specific product. “When consumers are in an exploratory mood, retailers can recommend products and services customers didn’t even know they wanted,” says Vyas. For example, business teams that use personalization platforms as part of an integrated media strategy can optimize algorithms against outcomes such as improving conversion and driving engagement.

Vyas says the elevated experience from “personalization to serendipity” fosters trust in the ability of recommendation architectures to persuade and influence consumers’ choices in beneficial ways. A case in point: our research found that half (52 percent) of spending by millennials goes towards experience-related purchases. As always, the key is to meet consumers where they are. Even better, according to Vyas, is to anticipate and understand when signals like customer intent are changing.

How to create value throughout the customer journey

One solution companies can implement right now is an integrated media spend platform that incorporates reporting, planning, and strategy across the entire customer journey. This offers value throughout the customer journey by using technology to reduce friction along the way. Think of it as starting with the customer looking for a product (search and discovery), moving on to the purchase (omnichannel basket, “shoppable” screens) and pick up/delivery (QR code scan in store), and through to post-purchase engagement with the retailer (Google Contact Center AI).

Such a holistic approach also accelerates data acquisition, integration, and reporting using advanced analytics to break down silos and emphasize the importance of privacy and first-party data. This in turn guides end-to-end interventions across customer journeys that enable optimized media spend, empowers businesses to analyze their spend distribution, fine-tunes owned and paid tactics with agencies, and promotes stewardship to support audit efficacy.

A culture of experimentation

With this data-driven focus, CPRs can create a 360-degree perspective of the customer. That intelligence can be used to enhance and humanize automated shopping experiences by putting the customer in control, whether online, in store, or across company brands. Moreover, by using Google Cloud’s emerging technologies such as artificial intelligence (AI), we help companies accelerate value across the spectrum, from supply-chain optimization and customer innovation to consumer experience.

Building a culture of experimentation is a team effort. “It’s not ever just one person who had a big idea. It’s all incremental steps,” says Jennifer Marchand, Google Cloud COE Leader, Capgemini. “Finding the right use case and timing is everything.” Take Google Glass Enterprise, she continues. For greater consumer experience, it can enable in-store associates to better serve with hands-free checkout, customer personalization and recommendations, and special offers. At the same time, Computer Vision and Smart Shelves can help prioritize tasks for employees, notifying them of low stock or a spill in the store.

Marchand points out that companies and consumers alike might not be ready to fully adopt some technology like facial recognition, but since almost everyone has a smartphone these days, they can benefit from automation with ease. “What’s interesting,” she adds, “is the way Google thinks about these types of problems, solving them for the long term.”

The store of tomorrow

Imagine a truly frictionless shopping experience, where state-of-the-art computer vision and AI identifies the products you pick up, put back, and keep, allowing you to head home, completely bypassing the checkout, with a 99 percent accuracy rate and receipts sent directly to your mobile app. That utopian experience is already taking shape at CornerShop, Capgemini’s live experimental store in London, UK.

Jamie MacLoud, Transformation & Strategy Consultant at frog, part of Capgemini, describes the retail space, which runs on Google Cloud, as the store of tomorrow, and not the distant future. It’s an experiential space where retailers and brands can explore, develop, and test technological shopping innovations in real-time. The outcome is a clearer understanding of how digital innovation can enable new ways to progress the customer experience, improve in-store operations, and help consumers to rediscover the joy of in-person retail through new ways to shop and engage with brands.

“We build, test, and learn about store concepts of tomorrow that we believe could be implemented into actual stores in the next one to two years. Getting these experiences in front of real customers in the CornerShop allows us to generate tangible learnings that we can share with our clients and use to shape future store strategy” says MacLeod. CornerShop was opened to the public in two eight-week stints, which allowed real-time testing to see what technologies resonated with customers, which brands can adapt and scale. Frictionless checkout, not surprisingly, was a big win for customers, but the technology underpinning the “virtual try-on” of clothing was deemed more suitable for the store of the future.

So, unlike an innovation lab, CornerShop lets companies experiment risk free, speeding up the process from hypothesis to full-scale implementation. It’s also another step toward solving the challenges customers face, while delivering those serendipitous experiences that build brand loyalty and longevity.

Learn more about how Capgemini is partnering with Google Cloud to help retailers create next-generation shopping experiences today.

We would like to acknowledge Jamie MacLoud, Transformation & Strategy Consultant at frog, part of Capgemini and Neerav Vyas, Head of Customer First, Co-Chief Innovation Officer, Insights & Data, North America at Capgemini who supported with invaluable insights and subject matter expertise in the writing of this blog post.

For over two decades, Google has helped lead the invention of modern cloud systems—defining, designing and deploying warehouse-scale computing as the foundation for reliable, performant, and secure global-scale information services delivered to billions of users around the world. This leadership involves significant innovation across a broad range of systems technologies, including distributed systems, storage systems, databases, analytics, operating systems, wide area and data center networking, cluster computing, ML, video acceleration and more.

Today, we are announcing a significant step in continuing Google’s tradition of innovation and charting its path into the future: the formation of SystemsResearch@Google (SRG). SRG will be a new research team, positioned in the heart of Google’s Cloud and Infrastructure engineering organization, with the mission of shaping the future of hyperscaler systems design for Google and its ecosystem. It is focused on inventing, incubating, and infusing new concepts, designs, and technologies into Google’s applications, systems, and data centers. The team’s position will allow seamless engagement with engineering and product teams, enabling joint exploration in concert with transformative workloads. Beyond Google, the SRG team will look to forge strong relationships with external research communities working on the most pressing systems-research problems.

Critical research at a pivotal time

We are at a time of enormous transition and opportunity, as nearly all large-scale computing is moving to cloud infrastructure, classical technology trends are hitting limits, new programming paradigms and usage patterns are taking hold, and most levels of systems design are being restructured. We are seeing wholesale change with the introduction of new applications around ML training and real-time inference to massive-scale data analytics and processing workloads fed by globally connected edge and cellular devices. This is all happening while the performance and efficiency gains we’ve relied on for decades are slowing dramatically from generation to generation. And while reliability is more important than ever as we deploy societally-critical infrastructure, we are challenged by increasing hardware entropy as underlying components approach angstrom scale manufacturing processes and trillions of transistors.

In the last twenty years, much of the world’s population has gained real-time access to the world’s information and to one another in ways that were previously the stuff of science fiction. The next decade will see computing and associated capabilities undergo an even more profound transformation, bringing real-time insights, sensing, and actuation to trillions of network-connected devices spanning all of the world’s population. Doing so will require fundamental advances in security, reliability, programming models, data analysis, systems for machine learning, networking, storage systems, hardware architecture, and software systems.

SystemsResearch@Google will be co-led by David Culler and Hank Levy, who bring a combination of academic and industrial experience, plus a long history of successful and impactful research in computer systems. Culler is the former Chair of EECS at UC Berkeley, where he worked to create the Division of Data Sciences and became its founding Dean. His research has focused on parallel architectures, clusters, embedded wireless networks, planetary-scale internet services, and sustainability design. He was the founding faculty director of Intel Research Berkeley, co-founded two startups, and worked with Sun Microsystems for a decade. Levy is the former Chair of Computer Science & Engineering at University of Washington, where he worked to create the Paul G. Allen School and became its founding Director. His research has focused on operating systems, distributed systems, computer architecture, and hardware multithreading. Before UW, Levy spent a decade at Digital Equipment Corporation (DEC), where he worked on operating systems and early-generation clustered computer systems; he has also co-founded two startups. Culler and Levy are both Members of the National Academy of Engineering and Fellows of the IEEE and the ACM.

SRG will be located across sites in Google’s Bay Area and Seattle facilities. We are currently building the SRG team, bringing together leading networked systems thinkers from around the world and inside Google. If you are interested in learning more please reach out to us at systemsresearch@google.com.