This Diagnostic Company is Revolutionising Healthcare Delivery with AI

5416
Of your peers have already read this article.
5:30 Minutes
The most insightful time you'll spend today!
Dr. Elliot Smith cannot be accused of lacking ambition. A high achiever with a Ph.D. in Electrical Engineering and a specialist in magnetic resonance imaging (MRI) systems, Smith aims to deliver top quality healthcare to anyone in the world — regardless of their location or wealth.
Dr. Smith has already made strides on this journey with his Brisbane, Queensland-headquartered business, Maxwell MRI. “I saw there was a big gap in the market around automating the diagnosis of health conditions,” he says. “Existing processes were typically manual and involved a lot of people.”
Artificial intelligence (AI) and machine learning can remove a key obstacle to scaling out medicine and improve the efficiency and accuracy of diagnosing conditions, the healthcare entrepreneur believes.
“Our grand vision is to build an AI doctor that anyone can receive affordable support from and connect to in order to obtain results,” explains Dr. Smith.
Maxwell MRI presently enables clinicians to submit anonymised MRI scans to a machine learning enabled AI platform to help diagnose prostate cancer. The service is sold to clinicians who can then charge a per-session fee to clients. As well as obtaining results for individual cases, the MRI scans and associated information is used to ‘train’ the platform to deliver accurate diagnoses faster and in a more affordable way than existing systems do.
Dr. Smith and his team started by running a number of functions and processes on a single server with graphics processing units (GPUs) and sizable hard disk capacity. However, this infrastructure could not scale to support the planned growth of the business. Each case Maxwell MRI processes involves about 200MB of data in MRI scans alone. Once supplementary data, blood test result, pathology results and genetic information is included, this load can reach more than 1GB of data per patient.
The business aimed to process 150,000 cases by the end of 2018. This required a service that could deliver massive scale in data storage and compute, and could easily be accessed from any location. “We wanted to move from three GPUs to 30 GPUs without having to buy more servers or other associated equipment, so the cloud was the natural next step,” says Dr. Smith.
Maxwell MRI evaluated Google Cloud Platform (GCP) and determined that the managed services component of GCP would remove the burden of infrastructure deployment and administration. In addition, Google Cloud Machine Learning Engine would enable the business to scale to as many GPUs as needed to meet demand.
Maxwell MRI started with some small experiments to determine that GCP met all its requirements and completed its migration to the platform in February 2017. “We really started to scale up the data we had and consequently our computing requirements at that time,” Dr. Smith says.
The Maxwell MRI platform features an upload service that enables clinicians to upload imaging and associated data. This service triggers several different upload pipelines that clean and standardise data. They then write imaging data to Google Cloud Storage, and more structured data to a combination of Google Cloud Datastore and Google Cloud Spanner.
The platform then converts the information into records that can be used to ‘train’ new machine learning configurations or run evaluations through existing machine learning pipelines.
“The tasks we perform including segmenting various anatomical regions for analysis and sending those results back into Google Cloud Storage,” says Dr. Smith. “This then commences that repeated process of running Google Cloud Dataflow pipelines and machine learning algorithms, and presenting those outcomes back to the clinicians.”
Existing Literature Validated
The data processed and analysed to date has, Dr Smith says, enabled Maxwell MRI to help validate existing literature that indicates clinicians lack confidence in existing early-stage testing procedures for prostate cancer. This prompts them to move quickly to the biopsy stage to assure themselves their diagnosis is valid. “New technologies have a lot of potential to rectify this situation and guide treatment to be more accurate, specific and cost-effective,” he says.
Results Delivered in 10-15 Minutes
More specifically, using GCP has enabled Maxwell MRI to guarantee to clinicians that results will be delivered within minutes. “Clinicians are used to getting results back in two days to a week,” says Dr. Smith. “We’re saying that with our platform running on GCP we’ll deliver you results in 10 to 15 minutes, regardless of the number of patients coming in.”
Running on GCP has enabled the business to accelerate its development cycles, test new ideas easily on a subset of data, test in parallel and deliver new services considerably faster than in another environment. In addition, the flexible GCP charging model aligned with the ability to scale compute capabilities quickly and easily has enabled the fledgling business to control its costs.
Google technologies are poised to play an integral role in the business’s future. “With Google available, it doesn’t make sense for us to use our own infrastructure,” Dr. Smith says. “Our expertise in AI, machine learning and clinical engagement complements cloud platform specialties of infrastructure, managed services and ease of use. We see a bright future ahead in helping to transform healthcare globally.”
Enabling Real-time AI with Streaming Ingestion in Vertex AI

2519
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Many machine learning (ML) use cases, like fraud detection, ad targeting, and recommendation engines, require near real-time predictions. The performance of these predictions is heavily dependent on access to the most up-to-date data, with delays of even a few seconds making all the difference. But it’s difficult to set up the infrastructure needed to support high-throughput updates and low-latency retrieval of data.
Starting this month, Vertex AI Matching Engine and Feature Store will support real-time Streaming Ingestion as Preview features. With Streaming Ingestion for Matching Engine, a fully managed vector database for vector similarity search, items in an index are updated continuously and reflected in similarity search results immediately. With Streaming Ingestion for Feature Store, you can retrieve the latest feature values with low latency for highly accurate predictions, and extract real-time datasets for training.
For example, Digits is taking advantage of Vertex AI Matching Engine Streaming Ingestion to help power their product, Boost, a tool that saves accountants time by automating manual quality control work.“Vertex AI Matching Engine Streaming Ingestion has been key to Digits Boost being able to deliver features and analysis in real-time. Before Matching Engine, transactions were classified on a 24 hour batch schedule, but now with Matching Engine Streaming Ingestion, we can perform near real time incremental indexing – activities like inserting, updating or deleting embeddings on an existing index, which helped us speed up the process. Now feedback to customers is immediate, and we can handle more transactions, more quickly,” said Hannes Hapke, Machine Learning Engineer at Digits.
This blog post covers how these new features can improve predictions and enable near real-time use cases, such as recommendations, content personalization, and cybersecurity monitoring.

Streaming Ingestion enables real-time AI
As organizations recognize the potential business impact of better predictions based on up-to-date data, more real-time AI use cases are being implemented. Here are some examples:
- Real-time recommendations and a real-time marketplace: By adding Streaming Ingestion to their existing Matching Engine-based product recommendations, Mercari is creating a real-time marketplace where users can browse products based on their specific interests, and where results are updated instantly when sellers add new products. Once it’s fully implemented, the experience will be like visiting an early-morning farmer’s market, with fresh food being brought in as you shop. By combining Streaming Ingestion with Matching Engine’s filtering capability, Mercari can specify whether or not an item should be included in the search results, based on tags such as “online/offline” or “instock/nostock.”

Mercari Shops: Streaming Ingestion enables real-time shopping experiment
- Large-scale personalized content streaming: For any stream of content representable with feature vectors (including text, images, or documents), you can design pub-sub channels to pick up valuable content for each subscriber’s specific interests. Because Matching Engine is scalable (i.e., it can process millions of queries each second), you can support millions of online subscribers for content streaming, serving a wide variety of topics that are changing dynamically. With Matching Engine’s filtering capability, you also have real-time control over what content should be included, by assigning tags such as “explicit” or “spam” to each object. You can use Feature Store as a central repository for storing and serving the feature vectors of the contents in near real time.
- Monitoring: Content streaming can also be used for monitoring events or signals from IT infrastructure, IoT devices, manufacturing production lines, and security systems, among other commercial use cases. For example, you can extract signals from millions of sensors and devices and represent them as feature vectors. Matching Engine can be used to continuously update a list of “the top 100 devices with possible defective signals,” or “top 100 sensor events with outliers,” all in near real time.
- Threat/spam detection: If you are monitoring signals from security threat signatures or spam activity patterns, you can use Matching Engine to instantly identify possible attacks from millions of monitoring points. In contrast, security threat identification based on batch processing often involves potentially significant lag, leaving the company vulnerable. With real-time data, your models are better able to catch threats or spams as they happen in your enterprise network, web services, online games, etc.
Implementing streaming use cases
Let’s take a closer look at how you can implement some of these use cases.
Real-time recommendations for retail
Mercari built a feature extraction pipeline with Streaming Ingestion.

The feature extraction pipeline is defined with Vertex AI Pipelines, and is periodically invoked by Cloud Scheduler and Cloud Functions to initiate the following process:
- Get item data: The pipeline issues a query to fetch the updated item data from BigQuery.
- Extract feature vector: The pipeline runs predictions on the data with the word2vec model to extract feature vectors.
- Update index: The pipeline calls Matching Engine APIs to add the feature vectors to the vector index. The vectors are also saved to Cloud Bigtable (and can be replaced with Feature Store in the future).
“We have been evaluating the Matching Engine Streaming Ingestion and couldn’t believe the super short latency of the index update for the first time. We would like to introduce the functionality to our production service as soon as it becomes GA, ” said Nogami Wakana, Software Engineer at Souzoh (a Mercari group company).
This architecture design can be also applied to any retail businesses that need real-time updates for product recommendations.
Ad targeting
Ad recommender systems benefit significantly from real-time features and item matching with the most up-to-date information. Let’s see how Vertex AI can help build a real-time ad targeting system.

The first step is generating a set of candidates from the ad corpus. This is challenging because you must generate relevant candidates in milliseconds and ensure they are up to date. Here you can use Vertex AI Matching Engine to perform low-latency vector similarity matching, generate suitable candidates, and use Streaming Ingestion to ensure that your index is up-to-date with the latest ads.
Next is reranking the candidate selection using a machine learning model to ensure that you have a relevant order of ad candidates. For the model to use the latest data, you can use Feature Store Streaming Ingestion to import the latest features and use online serving to serve feature values at low latency to improve accuracy.
After reranking the ads candidates, you can apply final optimizations, such as applying the latest business logic. You can implement the optimization step using a Cloud Function or Cloud Run.
What’s Next?
Interested? The documents for Streaming Ingestion are available and you can try it out now. Using the new feature is easy: For example, when you create an index on Matching Engine with the REST API, you can specify the indexUpdateMethod attribute as STREAM_UPDATE.
{
displayName: "'${DISPLAY_NAME}'",
description: "'${DISPLAY_NAME}'",
metadata: {
contentsDeltaUri: "'${INPUT_GCS_DIR}'",
config: {
dimensions: "'${DIMENSIONS}'",
approximateNeighborsCount: 150,
distanceMeasureType: "DOT_PRODUCT_DISTANCE",
algorithmConfig: {treeAhConfig: {leafNodeEmbeddingCount: 10000, leafNodesToSearchPercent: 20}}
},
},
indexUpdateMethod: "STREAM_UPDATE"
}After deploying the index, you can update or rebuild the index (feature vectors) with the following format. If the data point ID exists in the index, the data point is updated, otherwise, a new data point is inserted.
{
datapoints: [
{datapoint_id: "'${DATAPOINT_ID_1}'", feature_vector: [...]},
{datapoint_id: "'${DATAPOINT_ID_2}'", feature_vector: [...]}
]
}It can handle the data point insertion/update at high throughput with low latency. The new data point values will be applied in any new queries within a few seconds or milliseconds (the latency varies depending on the various conditions).
The Streaming Ingestion is a powerful functionality and very easy to use. No need to build and operate your own streaming data pipeline for real-time indexing and storage. Yet, it adds significant value to your business with its real-time responsiveness.
To learn more, take a look at the following blog posts for learning Matching Engine and Feature Store concepts and use cases:
Unified, Flexible and Accessible: How Companies’ Data Help Them Achieve More on Google Cloud

4888
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
As the volume of data that people and businesses produce continues to grow exponentially, it goes without saying that data-driven approaches are critical for tech companies and startups across all industries. But our conversations with customers, as well as numerous industry commentaries, reiterate that managing data and extracting value from it remains difficult, especially with scale.
Numerous factors underpin the challenges, including access to and storage of data, inconsistent tools, new and evolving data sources and formats, compliance concerns, and security considerations. To help you identify and solve these challenges, we’ve created a new whitepaper, “The future of data will be unified, flexible, and accessible,” which explores many of the most common reasons our customers tell us they’re choosing Google Cloud to get the most out of their data.
For example, you might need to combine data in legacy systems with new technologies. Does this mean moving all your data to the cloud? Should it be in one cloud or distributed across several? How do you extract real value from all of this data without creating more silos?
You might also be limited to analyzing your data in batch instead of processing it in real-time, adding complexity to your architecture and necessitating expensive maintenance to combat latency. Or you might be struggling with unstructured data, with no scalable way to analyze and manage it. Again, the factors are numerous—but many of them accrue to inadequate access to data, often exacerbated by silos, and insufficient ability to process and understand it.
The modern tech stack should be a streaming stack that scales with your data, provides real-time analytics, incorporates and understands different types of data, and lets you use AI/ML to predictively derive insights and operationalize processes. These requirements mean that to effectively leverage your data assets:
- Data should be unified across your entire company, even across suppliers, partners, and platforms., eliminating organizational and technology silos.
- Unstructured data should be unlocked and leveraged in your analytics strategy.
- The technology stack should be unified and flexible enough to support use cases ranging from analysis of offline data to real-time streaming and application of ML without maintaining multiple bespoke tech stacks.
- The technology stack should be accessible on-demand, with support for different platforms, programming languages, tools, and open standards compatible with your employees’ existing skill sets.
With these requirements met, you’ll be equipped to maximize your data, whether that means discerning and adapting to changing customer expectations or understanding and optimizing how your data engineers and data scientists spend their time. In coming weeks, we’ll explore aspects of the whitepaper in additional blog posts—but if you’re ready to dive in now, and to steer your tech company or startup towards success by making your data better work for you, click here to download your copy, free of charge.
Manhattan Associates and Google Cloud: How the Partnership Accelerates Future of Digital Retail

4977
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
While the shift to digital business and the cloud has been well under way for some years now, organizations today have a new sense of urgency due to COVID-19. Delivering digital transformation is no longer a ‘nice to have’ option, rather, it is an operational imperative. Taking advantage of the infrastructure, platform and solution gains that cloud and microservices architecture provide is a must for brands today.
At Google Cloud, we understand the pressures and challenges organizations of all sizes, across all industries are facing. The pandemic has dramatically impacted global commerce at-large, exposing (for many organizations across multiple sectors) gaps in omnichannel capabilities, business continuity and forecasting plans, not to mention spots in supply chain agility, resilience and responsiveness.
A rapidly evolving consumer-driven commerce landscape has put innovation squarely in the spotlight for supply chain teams all over the world, with the effects of the global pandemic making it increasingly difficult for manufacturers, wholesalers, third party logistics providers and retailers (in particular) to weather the perfect storm of fast-moving consumer trends and a need for ‘always on’ digital innovation.
These same effects have driven increasing interest and uptake of technology like the Manhattan Active® suite of solutions, as well as our own cloud platform; both of which afford organizations the levels of agility, flexibility and scalability needed to insulate their people, processes and long-term business strategies against unforeseen future obstacles such as global pandemics or international trade disputes.
An excellent example of this agility, flexibility and scalability in action is PVH’s response to the global pandemic. One of the most admired fashion and lifestyle companies with such iconic brands as Calvin Klein, TOMMY HILFIGER, Van Heusen, and IZOD, PVH was forced to temporarily close its physical stores and, as a result, experienced a sudden massive increase in online sales. The retailer was able to quickly pivot by adjusting its business rules in Manhattan Distributed Order Management (part of Manhattan Active Omni) to expose store inventory to online consumers and reroute its fulfillment processes. Thanks to Manhattan’s solution delivered through Google Cloud, in a matter of days, PVH was able to leverage both its distribution centers and vast store network to fulfill its online orders.
“The events of 2020 have accelerated retail and ecommerce operations forward,” said David Herridge, executive vice president of Global Value Chain Technologies for PVH. “With quick, creative thinking and the right partner, we were able to pivot operations, satisfy our customers and prepare for the future.”
Manhattan’s products have been recognized for their ability to solve real-world challenges through innovation, and used by many of the world’s top brands to solve some of their most complex commerce and supply chain challenges: the latest recognition is Manhattan’s position as sole leader in the 2021 Forrester Wave™ for Order Management Solutions.
Since December 2018, Google Cloud has been collaborating closely with the team at Manhattan and its ‘always on’, versionless approach to innovation. And, during the last two and a half years, Manhattan has significantly accelerated its cloud-first solutions and market adoption, resulting in tremendous growth in its overall cloud business efforts.
By building cloud native solutions on Google Cloud, the teams at Manhattan continue to deliver the high-performance, elastic, high-redundancy, secure solutions their customers rely on. Moreover, it means both Google Cloud and Manhattan continue to innovate and push the boundaries of what is possible in terms of the supply chain and omnichannel innovations that underpin global commerce – innovation that is needed more now than maybe ever before.
Our commitment to distributed cloud solutions and ongoing innovation, not to mention the fact Google Cloud operates a net carbon-neutral cloud, means that the working partnership between both industry leading teams continues to be a perfect match of brand values; not just from a technology perspective, but also a long-term sustainability and environmental one too.
More information on the partnership can be found here.
Supercharging Security with Generative AI

1524
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
At Google Cloud, we continue to invest in key technologies to progress towards our true north star on invisible security: making strong security pervasive and simple for everyone. Our investments are based on insights from our world-class threat intelligence teams and experience helping customers respond to the most sophisticated cyberattacks. Customers can tap into these capabilities to gain perspective and visibility on the most dangerous threat actors that no one else has.
Recent advances in artificial intelligence (AI), particularly large language models (LLMs), accelerate our ability to help the people who are responsible for keeping their organizations safe. These new models not only give people a more natural and creative way to understand and manage security, they give people access to AI-powered expertise to go beyond what they could do alone.
At the RSA Conference 2023, we are excited to announce Google Cloud Security AI Workbench, an industry-first extensible platform powered by a specialized, security LLM, Sec-PaLM. This new security model is fine-tuned for security use cases, incorporating our unsurpassed security intelligence such as Google’s visibility into the threat landscape and Mandiant’s frontline intelligence on vulnerabilities, malware, threat indicators, and behavioral threat actor profiles.
Google Cloud Security AI Workbench powers new offerings that can now uniquely address three top security challenges: threat overload, toilsome tools, and the talent gap. It will also feature partner plug-in integrations to bring threat intelligence, workflow, and other critical security functionality to customers, with Accenture being the first partner to utilize Security AI Workbench.
The platform will also let customers make their private data available to the platform at inference time; ensuring we honor all our data privacy commitments to customers. Because Security AI Workbench is built on Google Cloud’s Vertex AI infrastructure, customers control their data with enterprise-grade capabilities such as data isolation, data protection, sovereignty, and compliance support.

Preventing threats from spreading beyond the first infection
We already provide best-in-class capabilities to help organizations immediately respond to threats. But what if we could not just identify and contain initial infections, but also help prevent them from happening anywhere else? With our AI advances, we can now combine world class threat intelligence with point-in-time incident analysis and novel AI-based detections and analytics to help prevent new infections. These advances are critical to help counter a potential surge in adversarial attacks that use machine learning and generative AI systems. That’s why we’re excited to introduce:
- VirusTotal Code Insight uses Sec-PaLM to help analyze and explain the behavior of potentially malicious scripts, and will be able to better detect which scripts are actually threats.
- Mandiant Breach Analytics for Chronicle leverages Google Cloud and Mandiant Threat Intelligence to automatically alert you to active breaches in your environment. It will use Sec-PaLM to help contextualize and respond instantly to these critical findings.
These new updates build on the existing AI in Google’s industry-leading solutions. For example, Chronicle Security Operations already uses frontline intelligence, integrated reasoning, and machine learning to identify initial infections, prioritize impact, and contain threats. Another example is reCAPTCHA Enterprise, which uses image noising capabilities to help protect your site from adversaries that leverage novel AI advances, greatly enhancing our defenses against bots.
Adding intelligence to reduce toil
At Google Cloud, we help organizations modernize security wherever they are, in part by simplifying their security tools and controls whenever possible. Advances in generative AI can help reduce the number of tools organizations need to secure their vast attack surface areas and ultimately, empower systems to secure themselves. This will minimize the toil it takes to manage multiple environments, to generate security design and capabilities, and to generate security controls. Today, we’re announcing:
- Assured OSS will use LLMs to help us add even more open-source software (OSS) packages to our OSS vulnerability management solution, which offers the same curated and vulnerability-tested packages that we use at Google.
- Mandiant Threat Intelligence AI, built on top of Mandiant’s massive threat graph, will leverage Sec-PaLM to quickly find, summarize, and act on threats relevant to your organization.
These announcements build on existing capabilities that help customers centralize visibility and control, detect targets, and improve security across their platform. For example, Security Command Center (SCC) uses always-on machine learning to detect malicious scripts executing in the customer container environment and immediately alert the customer. In addition, Cloud Data Loss Prevention leverages machine learning to find and classify data, and with Confidential Computing you can collaborate on, train, and deploy sensitive and regulated AI models in the cloud, all while preserving confidentiality.
Evolving how practitioners do security to close the talent gap
At Google, we believe that to truly democratize security, we need to first acknowledge that AI will soon usher in a new era for security expertise that will profoundly impact how practitioners “do” security. Most people who are responsible for security — developers, system administrators, SRE, even junior analysts — are not security specialists by training.
Imagine a world where novices and security experts are paired with AI expertise to free themselves from repetition and burnout, and accomplish tasks that seem impossible to us today. To help power this evolution, we’re embedding Sec-PaLM-based features that can make security more understandable while helping to improve effectiveness with exciting new capabilities in two of our solutions:
- Chronicle AI: Chronicle customers will be able to search billions of security events and interact conversationally with the results, ask follow-up questions, and quickly generate detections, all without learning a new syntax or schema.
- Security Command Center AI: Security Command Center will translate complex attack graphs to human-readable explanations of attack exposure, including impacted assets and recommended mitigations. It will also provide AI-powered risk summaries for security, compliance, and privacy findings for Google Cloud.
These new releases bolster our existing efforts to tackle these issues through capabilities like IAM Recommender, which suggests permissions better suited to actual usage patterns. We will soon be augmenting this capability to cover organizational policies, further enabling the administrator to help improve the security posture of their organization. In addition, Mandiant Automated Defense applies machine learning to help reduce the repetitive Tier 1 alert triage problem and address alert fatigue.
Offering availability
VirusTotal Code Insight, available now in Preview, is our first example of putting Security AI Workbench to work for our customers. We will be rolling out other offerings to trusted testers in coming months, and they will be available in Preview more broadly this summer. Click here for the demo.
Security AI Workbench, including Sec-PaLM and partner integrations, in addition to the product innovations described in our demo, are all building blocks for a larger effort to elevate security across the ecosystem. So far, that effort:
- Provides assistive functions to rapidly develop IT generalist talent to Tier 1 security operator status in a way that wasn’t previously feasible. Security Command Center now can summarize threat intelligence insights and findings for Google Cloud, and Chronicle can quickly generate YARA-L rules or other detections.
- Provides advanced functions such as iterative query and multivariate detection generation, conversational filtering and interaction with results, and smart case awareness to empower advanced Tier 2 and 3 security operators to focus on threat analysis instead of struggling with process and toil. Mandiant Threat Intelligence users now can elevate their core competencies to hunt, investigate, and remediate threats — using the same tools our own Mandiant experts use.
- Fuses threat intelligence and AI-based analytic capabilities, which are unsurpassed in the market. VirusTotal Code Insight enables security teams to help gain insights and identify threats in suspicious code. This can significantly enhance their ability to detect and mitigate potential attacks.
However, this is just an initial step. We’ll continue to iterate and innovate, and we encourage customers and partners to leverage Security AI Workbench in new and exciting ways. Moving forward, we anticipate many new use cases to emerge over time.
Building a safer future
While generative AI has recently captured the imagination, Sec-PaLM is based on years of foundational AI research by Google and DeepMind, and the deep expertise of our security teams. This work includes new efforts to expand our partner ecosystem to provide businesses with security capabilities at every layer of the cybersecurity stack. We have only just begun to realize the power of applying generative AI to security, and we look forward to continuing to leverage this expertise for our customers and drive advancements across the security community.
UKG Ready: Meeting the Needs of Complex Machine Learning Models and Distributed Data Sets

3989
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Business Problem
UKG Ready primarily operates in the Small and Medium Business (SMB) space, so inherently many customers are forced to operate and make key business decisions with less Workforce Management (WFM) / Human Capital Management (HCM) data. In addition to volume, SMB lacks the variety of data needed to create a dynamic and agile organization. This puts SMB at a major disadvantage compared to larger segments.
Project Goals
People Insights module is committed to surfacing insights to customers in the context of their day-to-day duties and aid in decision making. With the SMB customer data limitations mentioned above, the goal of this project was to create a global dataset that augments individual customer data to bring light to less obvious, yet important information.
Challenges
UKG Ready is a highly configurable application that gives customers the opportunity to build solutions on a platform that meets their specific business needs. High configurability gives high flexibility to customers in their usage of the software. However, it becomes nearly impossible to create a global dataset for machine learning and data insights. UKG Ready manages just under 4 million of the US workforce and some 30,000+ customers. Despite the large employee dataset size, machine learning models that are specific to customers are starved for data because the individual customers have a relatively small employee population. Does that mean we cannot support our SMB customers’ decision making with ML?
Result
Partnering with Google, we were able to develop an approach that allowed us to standardize various domain entities (pay categories, time off codes, job titles, etc.) so that we could build a global dataset to augment SMB customer data. Using machine learning we were able to build a common vocabulary across our customer base. This common vocabulary encapsulates the nuances of how our customers manage their business and yet is generalized and standardized such that the data can be aggregated over the variety of customer configurations. This allows us to serve up practical insights to customers through various use cases. Our partnership allowed us to leverage Google Cloud Services to meet the needs of our complex machine learning models, distributed data sets and CI/CD processes.
How
UKG Ready decided to partner with Google for an end-to-end solution for the analytics offering. This allowed us to focus on our core business logic without having to worry about the platform, environment configurations, performance and scalability of the entire solution. We make use of various Google Cloud services such as Cloud Triggers, Cloud Storage, Cloud Functions, Cloud Composer, Cloud Dataflow, Big Query, Vertex AI, Cloud Pub/Sub… to host our analytics solution. Jenkins manages the entire CI/CD pipelines and cloud environments are configured and deployed using Terraform.
The standardization of business entities problem was solved in three distinct steps:

Step 1: Collecting aggregated data
We needed an approach to collect aggregated data from our highly distributed, sharded, multi-tenant data sources. We developed a custom solution that allows us to extract data aggregated at source for PII and GDPR considerations and transfer to Google Cloud Storage in the fastest manner possible. Data is then transformed and stored in Big Query. Services used: GCS, Cloud Functions, DataFlow, Cloud Composer and Big Query. All processes are orchestrated using Cloud Composer and detailed logging is available in Cloud Logging (Stackdriver).
Step 2: Applying NLP (Natural Language Processing)
Once we had the variety of customer configurations or the business entities available, we then applied NLP algorithms to categorize and standardize these in buckets. This approach assumes that customers use natural language for configurations like job titles, pay codes etc.

String Preparation
The input data for string preparation process is an entity string or several strings, that describe one entity object (like name-description pair or code-name pair). The output represents set of tokens that may be used to run a classification/clustering model. The process of string preparation tokenizes strings, replaces shortcuts, handles abbreviations, translates tokens, handles grammatical errors and mistypes
ML Models
Statistical
The idea of the model is to use defined target classes (clusters) and assign several tokens (anchors) to each of them an entity that has any of those tokens would be “attracted” to appropriate class. All other tokens are weighted according to frequencies of usage of theses tokens in the entities with anchor tokens:
Using anchor tokens, we are building kind-of Word2Vec - dimensionality of vector is equal to number of target classes. The higher the specific dimension (cluster) value, the higher the probability of entity to be included in appropriate cluster. Final prediction for entity tokens list for specific class is sum of weights of all the tokens included. Predicted cluster is a cluster that has maximal prediction score.
Lexical Model
We managed to generate reasonable amount of labeled data during statistical model implementation and testing. That opens a possibility to build “classical” NLP model that uses labeled data to train classification neural network using pretrained layers to produce token embeddings or even string embeddings. We started experimentation with pre-trained models like GloVe and got good results with single words and bi-grams but started getting issues in handling of n-grams. Our Google account team came to our rescue and recommended some white papers that helped formulate our strategy. We now use Tensorflow nnlm-en-dim128 model to produce string embeddings – it was trained on 200B records English Google News corpus and produces for each input string 128-dimensional vector. After that we use several Dense and Dropout layers to build a classification model.
Ensembling
To perform ensembling all the model results for each class are cast to probabilities using softmax transformation with scale normalization. Final predicted probability is maximal average score of both models among all the classes scores – appropriate class is predicted class.
The machine learning models are deployed on Vertex AI and are used in batch predictions. Model performance is captured at every prediction boundary and monitored for quality in production.
Step 3: Making available common vocabulary
Having the standardized vocabulary, we then needed a mechanism to have the results be available in UKG Ready reports and customer specific models like Flight Risk and Fatigue. For this we again used Google Services for orchestration, data transformation and data storage.
Once the modeling is complete, we made the customer specific models leveraging the above architecture be available in Reports. We utilized our proven existing technology choices in GCP for orchestration, data transformation and data storage
Results
We are able to build a common vocabulary of our customers’ business entities with good confidence. And be an expert advisor to our SMB customers in their decision-making using machine learning. With the advice of our Google account team and using Google services we can add value to our product in a relatively short amount of time. And we are not done! We continue to use this platform for new use cases, complex business problems and innovative machine learning solutions.
Sample result:

Special thanks to Kanchana Patlolla , AI Specialist, Google for the collaboration in bringing this to light
More Relevant Stories for Your Company

What Drives Your Organization to be Data-driven?
Every organization has its own unique data culture and capabilities. Yet each is expected to use technology trends and solutions in the same way as everyone else. Your organization may be built on years of legacy applications, you may have developed a considerable amount of expertise and knowledge, yet you

Google is a Leader in the 2019 Gartner Magic Quadrant for Data Management Solutions for Analytics
As organizations continue to produce vast quantities of data, they increasingly need platforms that allow them to analyze, store, and extract meaningful insights from that data. And research from analyst firms like Gartner offer an important way for organizations to evaluate and compare cloud data warehouse providers. Earlier this year,
Data Science Teams: A Practitioners Guide to MLOps
Across industries, DevOps and DataOps have been widely adopted as methodologies to improve quality and reduce the time to market of software engineering and data engineering initiatives. With the rapid growth in machine learning (ML) systems, similar approaches need to be developed in the context of ML engineering, which handle

Measuring Deforestation in Extractive Supply Chains With ML
Introduction In my experience, I have observed that it’s common in machine learning to surrender to the process of experimenting with many different algorithms in a trial and error fashion, until you get the desired result. My peers and I at Google have a People and Planet AI YouTube series







