Today, manufacturers are advancing on their factory digitalization journey, betting on innovative technologies to strengthen competitiveness, deliver sustainable growth, and offer new services. Macroeconomic factors – such as high energy costs, increasing labor, and raw material shortages – drive the need for urgent operational optimizations and automation.

Cloud capabilities have matured at an accelerated pace, giving manufacturers practical avenues to achieve these goals. Manufacturers are finding new ways to bring AI and machine learning (ML) to practical use cases, like predictive maintenance, anomaly detection, and asset utilization management. However, manufacturers struggle to adopt AI at scale due to challenges around data accessibility, infrastructure, and technology.

Google Cloud created purpose-built tools and solutions to organize manufacturing data, make it accessible and useful, and help manufacturers quickly take significant steps on this journey by reducing the time to value. In this post, we will explore a practical example of how manufacturers can use Google Cloud manufacturing solutions to train, deploy and extract value from ML-enabled capabilities to predict asset utilization and maintenance needs.

The journey to machine learning insights starts with accessible data

The first step to a successful machine learning project is to unify necessary data in a common repository. For this, we will use Manufacturing Connect, the factory edge platform co-developed with Litmus Automation, to connect to manufacturing assets and stream the asset telemetries to Pub/Sub.

After the telemetry messages are published to Pub/Sub, Dataflow will identify each message based on its structure and apply corresponding normalizations and transformations, which are preconfigured in Manufacturing Data Engine. Once the messages are processed, the messages will be routed to Cloud Storage, BigQuery, and/or Cloud BigTable based on user configuration.

To train a machine learning model, manufacturers can use Vertex AI AutoML to build a no-code model based on the training data stored in the Manufacturing Data Engine.

Then, users can trigger a batch prediction job in Vertex AI or export the AutoML model to run on the edge component of Manufacturing Connect for real-time prediction. Regardless of the model deployment methods, the prediction and explanation results will be ingested into Manufacturing Data Engine, which can be analyzed and visualized in Looker.

A blueprint for classifying asset condition

The following scenario is based on a hypothetical company, Cymbal Materials. This company is a factitious discrete manufacturing company that runs 50+ factories in 10+ countries. 90% of Cymbal Materials manufacturing processes involve milling, which are accomplished using industrial computer numerical control (CNC) milling machines. Although their factories implement routine maintenance checklists, there are unplanned and unknown failures that happen occasionally. However, many of the Cymbal Materials factory workers lack the experience to identify and troubleshoot failures due to labor shortage and high turnover rate in their factories. Hence, Cymbal Materials is working with Google Cloud to build a machine learning model that can identify and analyze failures on top of Manufacturing Connect, Manufacturing Data Engine, and Vertex AI.

For the pilot, Cymbal Materials forms a team of manufacturing engineers and data scientists to evaluate the feasibility of solving the tool wear detection problem. To avoid compliance concern, the Cymbal Materials team chooses to start with a public tool wear detection dataset hosted on Kaggle. This dataset is collected from running machining experiments on 2″ x 2″ x 1.5″ wax blocks in a CNC milling machine. The dataset contains measurements from the 4 motors (X,Y, Z axes and spindle) and program values in the CNC machine, which maps well to the data that Cymbal Materials collects for their CNC milling machines.

To start, the Cymbal Materials data scientists download the tool wear detection dataset from Kaggle and upload the dataset to Cloud Storage. Then, the data scientists use Vertex AI to:

• Perform exploratory data analysis using Vertex AI Workbench
• Train machine learning models using Vertex AI AutoML
• Deploy machine learning models to perform batch prediction and export AutoML models for edge deployment
• Interpret predictions using Vertex Explainable AI

After seeing great performance of the AutoML tabular model, the Cymbal Materials data scientists decide to use the AutoML model to predict on the actual CNC milling machine telemetries from their factories and validate the generalizability of the AutoML model. They ask the manufacturing engineers to deploy Manufacturing Connect in a Cymbal Materials factory and stream telemetries for one CNC milling machine to the Manufacturing Data Engine.

Manufacturing Connect includes an edge component that can gather data from manufacturing assets via an extensive library of 250+ communication protocols. The edge component of Manufacturing Connect comes with built-in Node-RED and Docker runtime, which support running custom workflows and machine learning models at the edge.

Using pre-defined hierarchies, Manufacturing Connect pushes asset telemetries and states to Pub/Sub.

After the factory operational data are ingested into Pub/Sub, Cymbal Materials uses Manufacturing Data Engine to:

• Normalize, transform, and contextualize real-time operational data with slowly changing metadata
• Batch ingest historical operational data and prediction results
• Route data dynamically to Cloud Storage, BigQuery, and/or Cloud BigTable

Using the trained AutoML tabular model and real-time telemetries from CNC milling machines, the data scientists trigger a batch prediction job on the CNC milling machine telemetries in BigQuery. The data scientists configure the batch prediction to output prediction results in Cloud Storage such that Manufacturing Data Engine can batch ingest the prediction results after the batch prediction job completes.

To consume the prediction results, the Cymbal Materials manufacturing engineers use Looker to create visualizations. The dashboard allows the manufacturing engineers to:

• Visualize the CNC milling machine actual and predicted tool conditions over time
• Explain the prediction results by summarizing the top attributing features
• Create alerts based on the predicted tool condition for their assets
• Take actions by contacting the supplier and/or scheduling maintenance for their assets

From edge to cloud, improving production efficiency for manufacturers

To support the entire factory digitalization value journey, manufacturers are looking for capabilities from simple visualizations to predictive ML models. Robust solutions, such as the one covered here, provide rapid paths for engineers to extract insight from their factory data.

Having a common data repository for manufacturing data, industry-leading machine learning platform, and versatile dashboard components accelerate manufacturer’s digital transformation.

This solution brings the best of Google Cloud’s data analytics and artificial intelligence capabilities in an industrial environment. Manufacturing Connect creates the link between industrial machinery and Manufacturing Data Engine, the cloud platform where the manufacturing data are processed, normalized, contextualized, and stored in a ready-to-consume format. Vertex AI can build, deploy, and scale machine learning models using data stored in the Manufacturing Data Engine. Vertex AI includes AutoML and Workbench for training models without code and training custom models with code-first experience respectively.

Learn more about how Google Cloud is transforming manufacturing to meet changing customer expectations at our Google Cloud Next Manufacturing playlist.

What’s next

1. Introducing new Google Cloud manufacturing solutions: smart factories, smarter workers
2. Manufacturing Data Engine | Solutions | Google Cloud
3. GitHub – GoogleCloudPlatform/mfg-ml-examples

It’s no secret that many organizations and job seekers find the hiring process exhausting. It can be time consuming, costly, and somewhat risky for both parties. Those are just some of the experiences we wanted to change when we started Gyfted, a pre-vetted talent marketplace for people who complete tech training or degree programs and are looking for the right career move. At the same time, we’re helping businesses save time and improve recruiting outcomes with our automated candidate screening and sourcing tools.

Our vision is clear: To take a candidate through one structured hiring process, and then put them in front of thousands of companies. It’s similar to the common app system in higher education. Sounds simple, but it is a herculean technical and UX task. To succeed we had to combine advanced psychometric testing, machine learning, the latest in behavioral design, and develop the highest quality structured, relational dataset to represent candidate and manager profiles and preferences on our network. Fortunately, we were cofounded by world-leading experts in these areas including Dr. Michal Kosinski, one of the world’s top computational psychologists, and Adam Szefer, a gifted young technologist. We’ve been joined by a group of equally talented employees, most of whom work remotely in Poland, US, Switzerland, UK, Israel, and Ukraine.

The influence of dating platforms and matching

When seeking inspiration, we were influenced by the success of dating platforms, especially Bumble with its focus on commitment. These platforms have done a great job using design to match people together.

We like to think we’re doing the same for recruiters and candidates in terms of not only role and culture fit matching, but also through a fundamental feature of Gyfted, which is that our job-seekers are anonymous. This helps recruiters meet one of their goals today, which is to minimize bias in the hiring process and enable objective, diversity-oriented recruiting.

Another of our unique selling points is that we conduct candidate screening that is gamified and automated, with a structured interview, where the interview remains with the candidate’s profile. We roughly estimate that just for the 15 million open jobs on LinkedIn, if companies fill in 50% of those via external recruiting, and conduct a screening interview with 10 candidates per job filled at $50/hour paid to an employee to do the screening, that’s $3.75 billion and 75 million hours in direct costs. This is on top of applying for jobs, selecting CVs, and coordinating the process, which takes an even bigger financial and time toll on both applicants and recruiters. Instead, it would be better to take one interview for 1000 companies. The impact of what we want to achieve with our vision is enormous.

We also offer career discovery and career search tools for job candidates. This includes free, personalized feedback for every job-seeker. Right now, we’re aiming the service at students, bootcamp graduates and juniors, helping them to land jobs in tech and the creative industry at large. Next, we’ll expand into mid and senior roles. In the long run we want to reshape how recruiting happens through a common app that saves everyone in the market significant time and resources, helping people find jobs not only faster, but jobs that truly fit them.

Developing advanced AI applications with Google Cloud

We obtained our original funding from angel and institutional investors, and we were selected into the Fall 2021 batch of StartX, the non-profit start-up accelerator and founder community associated with Stanford University. But like most startups our budgets are tight, and we need to find ways to operate as efficiently as possible, especially when building out our technology stack and developer environment.

That’s where Google Cloud comes in. It’s a lot more affordable and flexible than competing solutions, and our developers love it. We use Google App Engine for the hosting and development of our applications giving us enormous flexibility. Vertex AI enables us to build, deploy, and scale machine learning models faster, within a unified artificial intelligence platform. On top of that we use Google Vertex AI Workbench as the development environment for the data science workflow, which allows us to have everything that we need to host and develop innovative AI-based applications.

BigQuery, Google Cloud’s serverless data warehouse, is another stand-out solution for us. We use it to crunch big data from all our systems and the UX is very intuitive and easy to use, allowing us to use it across the business and get insights from a wide range of employees, not just technical experts.

Above all, Google Cloud helps us solve the main platform challenges facing Gyfted including scalability and identity management, so we are perfectly positioned for growth. Right now, we handle about 2 million candidate interactions, a volume we expect to grow exponentially. As that number grows, we rely on Google Cloud to help us scale securely and with reliability.

Eliminating bias from the hiring process

Our technology partners have also been integral to helping us get to an advanced stage of our beta program. MongoDB on Google Cloud takes the data burden off our teams and reduces time to value of our applications. We can stay nimble and can scale database capacity at the push of a button.

Our collaboration with the Google team has been fantastic. Our Startup Success Manager is an expert when it comes to Google Cloud solutions, and he also understands our business from his own experience as an entrepreneur and an investor. It’s great to have an internal point of contact who can help us navigate all of Google’s resources.

I’d also stress the extent to which Google Cloud values align with ours. For example, a key benefit for our customers is the ability to strip unconscious bias out of the hiring process. Google Cloud tools support this commitment to diversity, especially when we are building out our AI models.

On a team level, we also appreciate the support that Google Cloud has shown through its Google Support Fund for Start-ups in Ukraine. This has helped many Ukrainian businesses to continue to operate at a very challenging time, including startups with remote, distributed teams in Poland where most of us stem from.

If I had to sum up Google Cloud and our collaboration with Google for Startups in a phrase, I’d say that it adds enormous value to our business while removing much of the risk when scaling up a start-up. We’ve seen the addition of many new tools and features in the past two years and our Google mentors are always looking at the best way these can be integrated with Gyfted’s own roadmap. That means that we can continue to transform recruiting and hiring processes with the support of one of the world’s most advanced tech companies as a strategic growth partner.

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

Last week we announced the private preview of Duet AI for Google Cloud, an always-on AI collaborator that uses generative AI to provide help to developers and cloud users. This article gives you a detailed look at Duet AI for developers, showing how Duet AI can help provide developers with real-time code suggestions, chat assistance, and enterprise-focused customization. You can sign-up here to join our waitlist for Google Cloud’s AI trusted tester program.  

We believe that addressing these use case​​s with large language models (LLMs) will usher in a major leap in productivity in enterprise development. Duet AI uses Codey, a family of code models built on PaLM 2. 

Developers continuously seek ways to improve their productivity and, over the past few decades, these efforts have resulted in massive productivity leaps due to technological changes. From advanced debuggers and online developer communities, to modern IDEs/notebooks and cloud computing, each advance brought massive changes in productivity. Despite these improvements, developers still face numerous challenges, some of which are unique to cloud development:

• Disruptive context switching and friction when integrating a new tool or service
• Excessive time spent on repetitive tasks
• Time required to understand a new code base or project
• Large cognitive workload when working on large code bases or complex APIs

Duet AI for developers focuses on challenges and tasks across the development lifecycle:

Code/Boilerplate Generation — Developers can describe the tasks they have in mind as a comment or function name, such as creating a Cloud Pub/Sub topic. Duet AI will generate a reference implementation that can be reviewed and modified, so developers don’t need to spend time reading through multiple documentation pages.

Inline Code Completion — To reduce the time spent on repetitive tasks and minimize the cognitive workload of tasks such as writing repetitive code or retrieving variable names, Duet AI provides intelligent, context-aware code completion, helping reduce the time spent on coding and enhancing the quality of the written code.

Enterprise Customization — Organizations frequently have massive code bases and specific recommended frameworks and best practices, which generic code assistance solutions may not be best positioned to support. With Vertex AI, developers will be able to tune and customize the underlying models and connect them to the Duet AI experience, allowing for assistance optimized to the needs of the organization.

Code Explanation — Developers spend significant time and effort reading and understanding code written by their peers or external contributors. To help assist in this process, Duet AI for code assistance provides an “Explain this code” option available whenever a developer selects their code, allowing them to more quickly understand, map, and navigate unfamiliar code bases.

Code security guardrails — Code generated by Duet AI can also be scanned for vulnerable dependencies via Source Protect, helping surface known public vulnerabilities impacting code, along with suggested fixes when available, bringing additional security.

By harnessing the power of AI-driven developer assistance such as the one provided by Duet AI for developers, businesses can unlock unprecedented levels of productivity and efficiency in software development, paving the way for a new era of innovation and growth.

These early features of Duet AI for Google Cloud will be available for limited users and we will be expanding access very soon. Sign up here to join Google Cloud’s AI Trusted Tester Program.

Organizations are currently modernizing their businesses in order to meet the increasing complexity of today’s business landscape. In effect, business leaders must evaluate the best way to mitigate the challenges which plague their cloud operations, all while meeting customers’ growing expectations around digital experience (DX) through agility, automation, and proactive incident avoidance. 

In this commissioned study, “Modernize With AIOps To Maximize Your Impact”, Forrester Consulting surveyed organizations worldwide to better understand how they’re approaching artificial intelligence for IT operations (AIOps) in their cloud environments, and what kind of benefits they’re seeing. 

Within this July 2021 study, you’ll see that AIOps systems and principles are here to help. It covers how AIOps increases efficiency and productivity across day-to-day operations, and how businesses are taking note. In fact, 91% of respondents have implemented AIOps to address at least one cloud operations issue, and expansion is set to skyrocket. Those that wait to act, risk losing out on the efficacy of their cloud investment and falling behind their more efficient competitors.

As you can see in the image above, there is a plethora of great information in this complimentary study. So, if you’re looking to enhance your cloud operations and/or adopt AIOps within your organization, be sure to download this free study today.

Editor’s note: Today’s guest post comes from Indian social media platform ShareChat. Here’s the story of how they improved performance, app development, and analytics for serving regional content to millions of users using Google Cloud. 

How do you create a social network when your country has 22 major official languages and countless active regional dialects? At ShareChat, we serve more than 160 million monthly active users who share and view videos, images, GIFs, songs, and more in 15 different Indian languages. We also launched a short video platform in 2020, Moj, which already supports over 80 million monthly active users. 

Connecting with people in the language they understand

As mobile data and smartphones have become more affordable in India, we noticed a large new segment of people, many in rural areas, being welcomed onto the internet. However, many of them didn’t speak English, and when it comes to accessing content and information—language plays a significant role. Instead of joining other social media sites where English reigned supreme, new internet users chose to join language or dialect-specific Whatsapp groups where they felt more comfortable instead.

So, we set out to build a platform where people can share their opinions, document their lives, and make new friends, all in their native language. ShareChat simplifies content and people discovery by using a personalized content newsfeed to deliver language-specific content to the right audience.

Given the high-intensity data and high volume of content and traffic, we rely heavily on IT infrastructure. On top of that, a large number of our users rely on 2G networks to post, like, view, or follow each other. Our platform needs to deliver great experiences to people who are spread out across the country and different networks without any reduction in performance.

The right cloud partner to support future growth

ShareChat was born in the cloud—we already knew how to scale systems to serve a large customer base with our existing cloud provider. But like many companies, we struggled with over-provisioning compute and storage to accommodate unpredictable traffic and avoid running out of storage. With demand rising for local language content and an increase in online interactions in response to the COVID-19 crisis, we realized that we would need a more efficient way to scale dynamically and allocate resources as needed.

Google Cloud was a natural choice for us. We wanted to partner with a technology-first company that would make it easy (and cost-effective) to manage a strong technology portfolio that would allow us to build whatever we wanted. Google is at the forefront of technology innovation and provided everything we needed to build, run, and manage our applications (including creating an efficient DevOps pipeline to fix and release new features quickly). 

We had a few issues in mind at the start of discussions with the Google Cloud team, but over time, as we got information and support from them, we realized that these were the partners we wanted in our corner when it came time to tackle our most challenging problems. In the end, we decided to take our entire infrastructure to Google Cloud.

To support millions of users, we deploy and scale using Google Kubernetes Engine. While we analyze our data using a combination of managed data cloud services, such as Pub/Sub for data pipelines, BigQuery for analytics, Cloud Spanner for real-time app serving workloads, and Cloud Bigtable for less-indexed databases. We also rely on Cloud CDN to help us distribute high-quality and reliable content delivery at low latency to our users. 

We now use just half the total core consumption of our legacy environment to run ShareChat’s existing workloads.

Google Cloud delivers better outcomes at every level 

By moving to Google Cloud, we saw major benefits in several key areas: 

Zero-downtime migration for users

At the time of migration, we had over 70 terabytes of data, consisting of 220 tables—some of which were up to 14 terabytes with nearly 50 billion rows. Due to our data’s interdependencies, moving services over one at a time wasn’t an option for us. 

Even though we were migrating such large volumes of data, we didn’t want to impact any of our customers. Latency spikes for out-of-sync data might affect message delivery. For instance, if a message or notification was delayed, we didn’t want to risk a bad user experience causing someone to abandon ShareChat. 

To prepare for the move, we ran a proof-of-concept cluster for over four months to test database performance in a real-world scenario for handling more than a million queries per second. Using an open-source API gateway, we replicated our legacy data environment into Google Cloud for performance testing and capacity analysis. As soon as we were confident Google Cloud could handle the same traffic as our previous cloud environment, we were ready to execute.

Using wrappers, we were able to migrate without having to change anything in our existing application code. The entire migration of 60 million users to Google Cloud took five hours—without any data loss or downtime. Today, ShareChat has grown to 160 million users, and Google Cloud continues to give us the support we need.

Scaling globally to meet unexpected demand

We rely on real-time data to drive everything on ShareChat by tracking everything that goes on in our app—from messages and new groups to content people like or who they follow. Our users create more than a million posts per day, so it’s critical that our systems can process massive amounts of data efficiently. 

We chose to migrate to Spanner for its global consistency and secondary index. Unlike our legacy NoSQL database, we could scale without having to rethink existing tables or schema definitions and keep our data systems in sync across multiple locations. It’s also cost-effective for us—moving over 120 tables with 17 indexes into Cloud Spanner reduced our costs by 30%.

Spanner also replicates data seamlessly in multiple locations in real time, enabling us to retrieve documents if one region fails. For instance, when our traffic unexpectedly grew by 500% over just a few days, we were able to scale horizontally with zero lines of code change. We were also launching our Moj video app simultaneously, and we were able to move it to another region without a single issue. 

Simplifying development and deployment

On average, we experience about 80,000 requests per second (RPS) –nearly 7 billion RPS per day. That means daily push notifications sent out to the entire user base about daily trending topics can often result in a spike of 130,000 RPS in just a few seconds. 

Instead of over-provisioning, Google Kubernetes Engine (GKE) enables us to pre-scale for traffic spikes around scheduled events, such as holidays like Diwali, when millions of Indians send each other greetings. 

Migrating to GKE has also enabled us to adopt more agile ways of work, such as automating deployment and saving time with writing scripts. Even though we were already using container-based solutions, they lacked transparency and coverage across the entire deployment funnel. 

Kubernetes features, such as sidecar proxy, allows us to attach peripheral tasks like logging into the application without requiring us to make code changes. Kubernetes upgrades are managed by default, so we don’t have to worry about maintenance and stay focused on more valuable work. Clusters and nodes automatically upgrade to run the latest version, minimizing security risks and ensuring we always have access to the latest features.

Low latency and real-time ML predictions

Even though many of our users may be accessing ShareChat outside of metropolitan areas, it doesn’t mean they’re more patient if the app loads slowly or their messages are delayed. We strive to deliver a high-performance experience, regardless of where our users are. 

We use Cloud CDN to cache data in five Google Cloud Point of Presence (PoP) locations at the edge in India, allowing us to bring content as close as possible to people and speeding up load time. Since moving to Cloud CDN, our cache hit ratio has improved from 90% to 98.5%—meaning our cache can handle 98.5% of content requests. 

As we expand globally, we’d like to use machine learning to reach new people with content in different languages. We want to build new algorithms to process real-time datasets in regional languages and accurately predict what people want to see. Google Cloud gives us an infrastructure optimized to handle compute-intensive workloads that will be useful to us both now—and in the future.  

The confidence to build the best platform

Our current system now performs better than before we migrated, but we are continuously building new features on top of it. Google’s data cloud has provided us with an elegant ecosystem of services that allows us to build whatever we want, more easily and faster than ever before. 

Perhaps the biggest advantage of partnering with Google Cloud has been the connection we have with the engineers at Google. If we’re working to solve a specific problem statement and find a specific solution in a library or a piece of code, we have the ability to immediately connect with the team responsible for it. 

As a result, we have experienced a massive boost in our confidence. We know that we can build a really good system because we not only have a good process in place to solve problems—we have the right support behind us.