Building a strong brand in today’s hyper-competitive business environment takes vision. It also requires a flexible, easily managed approach to digital asset management (DAM), so marketing professionals and other stakeholders can easily share, store, track, and manipulate assets to build the brand.

Many of today’s leading companies, including JetBlue, Slack, TripAdvisor, Lyft, and HealthONE, rely on Brandfolder to deliver consistent, organized, and efficient brand experiences. Brandfolder provides an easy-to-use platform that can scale across an entire company with little end-user training, empowering customers to distribute digital assets wherever they are needed. Customers also gain much greater insight into how those assets are used, and how to use them more effectively in marketing campaigns and brand messaging.

“Google Cloud made it easy to build an ML platform to quickly iterate through different brand intelligence use cases and release data-driven product features into the Brandfolder platform.”

—Ajay Rajasekharan, Head of Data Science, Brandfolder

Brandfolder is constantly advancing its development efforts to introduce new data-driven features without complicating the user experience. Big data, artificial intelligence (AI), and machine learning (ML) are key to meeting customers’ unique business needs, and essential for Brandfolder to compete in the fast-moving DAM industry. To enhance these capabilities, Brandfolder sought a public cloud provider that could help it scale its data pipeline cost effectively while providing access to advanced AI technologies.

After graduating from the Techstars startup accelerator program in 2013, Brandfolder tried two other cloud providers before standardizing on Google Cloud Platform (GCP).

“We saw a difference with Google Cloud from the very beginning because the interactions felt like a strategic relationship,” says Jim Hanifen, Head of Product at Brandfolder. “Google gave us startup credits and a lot of face-to-face support, which we hadn’t experienced with other cloud providers. We decided to move our entire infrastructure to Google Cloud Platform.”

Building an ML platform for brand intelligence

After performing an initial lift-and-shift migration of virtual machines (VMs) onto Compute Engine, Brandfolder built an ML platform using GCP managed services to seamlessly deliver its data products. The platform leverages Cloud SQL, Cloud Storage as the data lake, Cloud Dataproc for cloud-native Apache Spark computing clusters, Cloud Composer as the batch job scheduler, Cloud Pub/Sub as the backbone data pipeline, Container Registry to store Docker images, and Google Kubernetes Engine (GKE) as the application orchestrator. Cloud Dataflow brings data into the data lake and into BigQuery for analysis.

“Google Cloud made it easy to build an ML platform to quickly iterate through different brand intelligence use cases and release data-driven product features into the Brandfolder platform,” says Ajay Rajasekharan, Head of Data Science at Brandfolder, who describes the architecture in a detailed blog. “We simply ingest raw application and event data on one end and output an ML service on the other.”

“Moving to Google Cloud Platform allows us to complete more sophisticated data analysis and ML models much faster, and at a much lower cost. We can create brand-specific ML models 12x faster and get them into production quickly to address our customers’ unique business needs.”

—Brett Nekolny, Head of Engineering, Brandfolder

For many general use cases, Brandfolder does not need to build custom ML models, and instead relies on pre-trained API models from GCP. For example, it uses Vision API and Video Intelligence API to auto-tag creative assets on import to enable fast, intuitive searches across images and videos. When more product- and brand-specific modeling is required to address unique customer use cases, Brandfolder builds and trains custom ML models using its GCP pipeline or Cloud AutoML, a suite of products built on Google transfer learning and neural architecture search technology. For example, if a Brandfolder customer makes different types of grills, Brandfolder can use AutoML Vision to train a model to recognize the different grills.

“Moving to Google Cloud Platform allows us to complete more sophisticated data analysis and ML models much faster, and at a much lower cost,” explains Brett Nekolny, Head of Engineering at Brandfolder. “We can create brand-specific ML models 12x faster and get them into production quickly to address our customers’ unique business needs.”

Industry-leading security and performance

Google Cloud’s security model helps Brandfolder give existing and prospective customers peace of mind that their data will be protected. Cloud Identity & Access Management (Cloud IAM) provides enterprise-grade access control, while Cloud Identity-Aware Proxy (Cloud IAP) enables remote users to work more securely without the hassles of a VPN client. GCP also isolates cloud resources into projects, making it easy to assign permissions and keep data and VMs organized and segregated.

“With Google Cloud, everything begins and ends with security, which makes things very easy for us,” says Jim. “If we’re under a security review, we can submit a Google security white paper. If a potential customer has security concerns, we tell them we are hosted on GCP, and those concerns go away.”

To give customers even better application performance for accessing their brand assets, Brandfolder uses Cloud Memorystore, an in-memory data store service for Redis, to cache data and provide sub-millisecond data access for production applications.

“It was much easier for us to use Cloud Memorystore versus running Redis on our compute instances,” says Brett. “The high availability, replication across zones, and automatic failover with no data loss are big for us.”

Global private network interconnects between Google Cloud and the Fastly content delivery network (CDN) dramatically reduce latency, allowing Brandfolder’s customers to deliver and update even very large creative assets quickly around the world.

“What’s beautiful about the relationship between Google and Fastly is that if one of our customers uploads a new version of an asset, we can propagate that out to Fastly, and the new version will automatically show up in all the places where it’s referenced,” says Brett.

“The ability to quickly solve problems with AI has a substantial impact on our revenue, and that’s more apparent every quarter. Few of our competitors are doing product- or brand-specific modeling because it takes a lot of time and resources. We overcame those hurdles with Google Cloud.”

—Jim Hanifen, Head of Product, Brandfolder

Improving employee and customer productivity

Brandfolder also uses Google solutions for real-time collaboration and productivity, using G Suite to connect employees with intuitive, cloud-based apps. Teams use Gmail, Calendar, Docs, Drive, Sheets, Slides, and Hangouts Meet every day to move the business forward. Many of Brandfolder’s customers are also G Suite users, and Brandfolder offers a plug-in that allows them to view their creative assets inside of Docs and pull images in as needed. Customers can also log into Brandfolder with their G Suite credentials, making the solution even easier to use.

“We’ve been using G Suite since the beginning, and it’s helped us collaborate efficiently to build a successful, growing company,” says Jim. “Our teams expect to have that kind of close collaboration, and everyone here enjoys the G Suite experience.”

Driving 99 percent annual business growth

With automated tagging and other innovative AI-based features, Brandfolder is helping customers locate and distribute assets faster. As a result, Brandfolder is building customer loyalty and increasing sales, growing its business by 99 percent year-over-year. Since moving to GCP, Brandfolder has been able to scale its analytics and data pipeline 50x without a corresponding increase in costs and has not had to expand its development team.

“The ability to quickly solve problems with AI has a substantial impact on our revenue, and that’s more apparent every quarter,” says Jim. “Few of our competitors are doing product- or brand-specific modeling because it takes a lot of time and resources. We overcame those hurdles with Google Cloud.”

Digital technology offers increasing flexibility and choice to consumers. As a result, the retail industry is dramatically shifting toward more tailored and personalized experiences for shoppers, and businesses are rethinking how they deliver value to customers.

This couldn’t be more true for the beauty retailing industry where leading companies are turning to digital technology to create customized shopping experiences.

At Google Cloud, we’re particularly excited about our work with Ulta Beauty, the largest beauty retailer in the United States with more than 1196 stores in all 50 states, and how the company is using Google Cloud technology solutions to power personalization and redefine beauty retailing.

Established in 1990, Ulta Beauty has had incredible success as a company, and as customers become more discerning and curious about their purchases, the company is finding new ways to meet their changing needs.

Recently, leaders at Ulta Beauty recognized a huge opportunity to complement and enhance the shopping experience by helping beauty enthusiasts navigate through more than 500 brands and 25,000 products carried in their stores and online channel.

They decided to leverage the data from Ulta Beauty’s successful Ultamate Rewards loyalty program to create and offer more unique and personalized user experiences.

With more than 30 million members generating data through sales, transactions, product reviews, and social media engagement, Ulta Beauty’s Loyalty Program creates a comprehensive data set, and the company sought the right technology partner to help organize, analyze and transform that data into valuable insights for its customers.

Ulta Beauty’s leaders knew they had an opportunity to leverage data analytics and machine learning to reach customers in new ways, enhance the guest experience, and continue to grow their active loyalty member base. After considering a number of cloud providers, they chose to expand their existing partnership with Google Cloud.

“Google Cloud listened to our needs and worked in tandem with our engineering team to address our challenges,” said Michelle Pacynski, vice president of digital innovation at Ulta Beauty. “The ease of working with the Google Cloud team and their breadth of experience made the decision a no-brainer, laying the foundation for a great partnership.”  

In 2019, Ulta Beauty announced it was working with Google Cloud Platform to unify and organize its data, using:

• BigQuery to perform data analysis and generate dynamic content, personalized product recommendations, and event-based messages for customers.
• Cloud Storage to provide highly available, secure, resilient and cost-effective access to data across the entire enterprise.
• Compute Engine for the high-performance scalability needed to grow with customer demand while painlessly migrating existing applications to the cloud.
• Anthos to build a hybrid cloud foundation that allows their applications to take advantage of all this data, combining the power and flexibility of GKE with the ability to leverage their existing investment in secure infrastructure on-premises.

Our partnership with Ulta Beauty has enabled increased engagement with customers in store and online, and the creation of new tools and capabilities, including a new Virtual Beauty Advisor tool to deliver tailored recommendations and help shoppers choose the right products, and a Customer Conversation Platform that’s enabling deeper connections with guests, ultimately driving customer loyalty.

“It’s been a really efficient process so far due in part to the ease of working with the Google team,” said Michelle Pacynski, vice president of digital innovation at Ulta Beauty. “They’re experienced, approachable, and their can-do style makes for a great partnership. They listened to our needs and worked in tandem with our engineering team, figuring things out, and getting it done.”

Editor’s note: Today’s post is by Mike Francomb, Senior Vice President of Technology, RecruitMilitary and a U.S. Army Veteran. RecruitMilitary is a wholly owned subsidiary of Bradley-Morris, Inc. (BMI), the largest military-focused recruiting company in the United States. RecruitMilitary uses Google Cloud Talent Solution to power its job search experience and connect more organizations with veteran talent.

For seven years, I served in the U.S. Army as a Field Artillery Officer, Military Occupation Code 13A. My time in service included a deployment to Operation Desert Shield / Desert Storm with the 24th Infantry Division out of Fort Stewart, GA, and a variety of front line artillery leadership roles, serving as a logistics officer for my unit and as an instructor teaching new officers how to be professional artillerymen. My day-to-day entailed leading teams of highly trained soldiers and managing logistics and materials to help those soldiers perform at a high level in stressful, fast-paced environments. It was my job to ensure we were ready to handle any circumstance. 

The hardest part about transitioning out of the Army in May 1996 as a highly trained artillery veteran was the fact that, though I felt prepared for any challenge ahead, I wasn’t sure I was making the right choice. I made a common mistake of transitioning veterans, I jumped right into an entrepreneurial venture. Looking back, I wish I’d had access to resources that displayed career options that would help translate my skills for the corporate world, it would have helped me be better prepared and know what my options were. I wasn’t ready to jump from the Army into running a business, and it was a long two years. 

Though my first job out of the Army was challenging, it taught me that I loved the start-up environment, and I joined RecruitMilitary in October 1998 when it was five months old. For the past 21 years, I have been fortunate enough to play an important role in helping RecruitMilitary grow to what it is today, the industry leader in connecting military veterans with organizations.  

RecruitMilitary connects organizations with veteran talent through over 30 products and services, all of which are fueled by our job board. Our job board, with over 1,400,000 members, is core to our business. In fact, if we don’t have an active and growing job board population, we don’t have the supply of veteran talent we need to deliver to our clients across our suite of services. 

With veteran unemployment at a 50-year low, it became increasingly challenging for RecruitMilitary to grow our veteran job seeker database and keep those veterans actively applying to client jobs. Being a data-driven company, we saw our existing search functionality was no longer producing the desired results for clients and began to receive client feedback about decreased candidate activity.

It was clear to us that we needed to begin adopting machine learning and more advanced search capabilities into our products and operations. The HR Tech space is shifting that way fast, and we want to be at the forefront. As we researched paths to take and learned of Google’s operating philosophy leading with AI, and that they were developing a tool for veteran job search, it made a lot of sense to go with a leader.

When Grow with Google announced its commitment to support veterans, we learned that we could add their military occupation code (MOS) translation feature to our job board through Cloud Talent Solution. This feature lets transitioning service members enter their military occupation codes (MOS, AFSC, NEC, or rating) directly into our search bar to see relevant civilian jobs available at client companies. We’re also using Cloud Talent Solution’s remote work functionality to provide an improved job search experience that allows our customers to make remote work opportunities in the U.S. more discoverable on their career sites. We’re excited about this feature, as it enhances our ability to deliver meaningful jobs to important members of our military community, military spouses, and veterans with limited mobility. 

The results of Cloud Talent Solution compared to our previous search are tremendous. Our job seekers are getting a much better experience, and to us that means more veterans are connected to jobs with our clients. We know this because our number of daily job applications has grown by 78 percent. And knowing that we now have a tool that’s going to learn and get better as more of our job seekers use it means that we will reap benefit for work done over time, and so will our clients and veterans who use our job board. That’s tremendous ROI to receive for a lean development staff. 

These are just a few of the types of tools I wish I’d had access to when I was considering my transition in 1996. With the help of technology and resources, like those from RecruitMilitary and Grow with Google, people in the military community, including veterans like myself, can prepare for and build meaningful careers.

Today, to accelerate research in the bio-pharma space, from the creation of treatments for diseases to the production of new synthetic biomaterials, we are announcing a new Vertex AI solution that demonstrates how to use Vertex AI Pipelines to run DeepMind’s AlphaFold protein structure predictions at scale.

Once a protein’s structure is determined and its role within the cell is understood, scientists can develop drugs that can modulate the protein function based on its role in the cell. DeepMind, an AI research organization within Alphabet, created the AlphaFold system to advance this area of research by helping data scientists and other researchers to accurately predict protein geometries at scale.

In 2020, in the Critical Assessment of Techniques for Protein Structure Prediction (CASP14) experiment, DeepMind presented a version of AlphaFold that predicted protein structures so accurately, experts declared the “protein-folding problem” solved. The next year, DeepMind open sourced the AlphaFold 2.0 system. Soon after, Google Cloud released a solution that integrated AlphaFold with Vertex AI Workbench to facilitate interactive experimentation. This made it easier for many data scientists to efficiently work with AlphaFold, and today’s announcement builds on that foundation.

Last week, AlphaFold took another significant step forward when DeepMind, in partnership with the European Bioinformatics Institute (EMBL-EBI), released predicted structures for nearly all cataloged proteins known to science. This release expands the AlphaFold database from nearly 1 million structures to over 200 million structures—and potentially increases our understanding of biology to a profound degree. Between this continued growth in the AlphaFold database and the efficiency of Vertex AI, we look forward to the discoveries researchers around the world will make.

In this article, we’ll explain how you can start experimenting with this solution, and we’ll also survey its benefits, which include offering lower costs through optimized selection of hardware, reproducibility through experiment tracking, lineage and metadata management, and faster run time through parallelization.

Background for running AlphaFold on Vertex AI

Generating a protein structure prediction is a computationally intensive task. It requires significant CPU and ML accelerator resources and can take hours or even days to compute. Running inference workflows at scale can be challenging—these challenges include optimizing inference elapsed time, optimizing hardware resource utilization, and managing experiments.Our new Vertex AI solution is meant to address these challenges.

To better understand how the solution addresses these challenges, let’s review the AlphaFold inference workflow:

1. Feature preprocessing. You use the input protein sequence (in the FASTA format) to search through genetic sequences across organisms and protein template databases using common open source tools. These tools include JackHMMER with MGnify and UniRef90, HHBlits with Uniclust30 and BFD, and HHSearch with PDB70. The outputs of the search (which consist of multiple sequence alignments (MSAs) and structural templates) and the input sequences are processed as inputs to an inference model. You can run the feature preprocessing steps only on a CPU platform. If you’re using full-size databases, the process can take a few hours to complete.
2. Model inference. The AlphaFold structure prediction system includes a set of pretrained models, including models for predicting monomer structures, models for predicting multimer structures, and models that have been fine-tuned for CASP. At inference time, you independently run the five models of a given type (such as monomer models) on the same set of inputs. By default, one prediction is generated per model when folding monomer models, and five predictions are generated per model when folding multimers. This step of the inference workflow is computationally very intensive and requires GPU or TPU acceleration.
3. (Optional) Structure relaxation. In order to resolve any structural violations and clashes that are in the structure returned by the inference models, you can perform a structure relaxation step. In the AlphaFold system, you use the OpenMM molecular mechanics simulation package to perform a restrained energy minimization procedure. Relaxation is also very computationally intensive, and although you can run the step on a CPU-only platform, you can also accelerate the process by using GPUs.

The Vertex AI solution

The AlphaFold batch inference with the Vertex AI solution lets you efficiently run AlphaFold inference at scale by focusing on the following optimizations:

• Optimizing inference workflow by parallelizing independent steps.
• Optimizing hardware utilization (and as a result, costs) by running each step on the optimal hardware platform. As part of this optimization, the solution automatically provisions and deprovisions the compute resources required for a step.
• Describing a robust and flexible experiment tracking approach that simplifies the process of running and analyzing hundreds of concurrent inference workflows.

The following diagram shows the architecture of the solution.

The solution encompasses the following:

A strategy for managing genetic databases. The solution includes high-performance, fully managed file storage. In this solution, Cloud Filestore is used to manage multiple versions of the databases and to provide high throughput and low-latency access.

An orchestrator to parallelize, orchestrate, and efficiently run steps in the workflow. Predictions, relaxations, and some feature engineering can be parallelized. In this solution, Vertex AI Pipelines is used as the orchestrator and runtime execution engine for the workflow steps.

Optimized hardware platform selection for each step. The prediction and relaxation steps run on GPUs, and feature engineering runs on CPUs. The prediction and relaxation steps can use multi-GPU node configurations. This is especially important for the prediction step because the memory usage is approximately quadratic with the number of residues. Therefore, predicting a large protein structure can exceed the memory of a single GPU device.

Metadata and artifact management. The solution includes management for running and analyzing experiments at scale. In this solution, Vertex AI Metadata is used to manage metadata and artifacts.

The basis of the solution is a set of reusable Vertex AI Pipelines components that encapsulate core steps in the AlphaFold inference workflow: feature preprocessing, prediction, and relaxation. In addition to those components, there are auxiliary components that break down the feature engineering step into tools, and helper components that aid in the organization and orchestration of the workflow.

The solution includes two sample pipelines: the universal pipeline and a monomer pipeline. The universal pipeline mirrors the settings and functionality of the inference script in the AlphaFold Github repository. It tracks elapsed time and optimizes compute resources utilization. The monomer pipeline further optimizes the workflow by making feature engineering more efficient. You can customize the pipeline by plugging in your own databases.

Next steps

To learn more and to try out this solution, check our GitHub repository, which contains the components and universal and monomer pipelines. The artifacts in the repository are designed so that you can customize them. In addition, you can integrate this solution into your upstream and downstream workflows for further analysis. To learn more about Vertex AI, visit our product page.

Acknowledgements

We would like to thank the following people for their collaboration: Shweta Maniar, Sampath Koppole, Mikhail Chrestkha, Jasper Wang, Alex Burdenko, Meera Lakhavani, Joan Kallogjeri, Dong Meng (NVIDIA), Mike Thomas (NVIDIA), and Jill Milton (NVIDIA).

Finally and most importantly, we would like to thank our Solution Manager Donna Schut for managing this solution from start to finish. This would not have been possible without Donna.

Who would have known that today technology would enable us with the ability to use machine learning to track vessel activity, and make pattern inferences to help address IUU (illegal, unreported, and unregulated) fishing activities. What’s even more noteworthy is that we now have the computing power to share this information publicly in order to enable fair and sustainable use of our ocean. 

An amazing group of humans at the nonprofit Global Fishing Watch took on this massive big data challenge and succeeded. You can immediately access their dynamic map on their website globalfishingwatch.org/map that is bringing greater transparency to fishing activity and supporting the creation and management of marine protected areas throughout the world.

In our second episode of our People and Planet AI series we were inspired by their ML solution to this challenge, and we built a short video and sample with all the relevant code you need to get started with building a basic time-series classification model in Google Cloud, and visualize it in an interactive  map. 

Architecture

These are the components used to build a model for this sample:

• Global Fishing Watch GitHub: where we got the data
• Apache Beam: (open source library) runs on Dataflow. 
• Dataflow: (Google’s data processing service) creates 2 datasets; 1 for training a model and the other to evaluate its results.
• TensorflowKeras: (high level API library) used to define a machine learning model, which we then train in Vertex AI.
• Vertex AI: (a platform to build, deploy, and scale ML models) we train and output the model.

Pricing and steps

The total cost to run this solution was less than $5. 

There are seven steps we went through with their approximate time and cost:

Why do we use a time series classification model? 

Vessels in the ocean are constantly moving, which creates distinctive patterns from a satellite view.

 We can train a model to recognize the shapes of a vessel’s trajectory. Large vessels are required to use the automatic identification system, or AIS. The GPS-like transponders  regularly broadcast a vessel’s maritime mobile service identity, or MMSI, and other critical information to nearby ships, as well as to terrestrial and satellite receivers. While AIS is designed to prevent collisions and boost overall safety at sea, it has turned out to be an invaluable system for monitoring vessels and detecting suspicious fishing behavior globally.

One tricky part is that the MMSI data location signal (which includes a timestamp, latitude, longitude, distance from port, and more) is not emitted at regular intervals. AIS broadcast frequency changes with vessel speed (faster at higher speeds), and not all AIS messages that are broadcast are received – terrestrial receivers require line-of-sight, satellites must be overhead, and high vessel density can cause signal interference. For example, AIS messages might be received frequently as a vessel leaves the docks and operates near shore, then less frequently as they move further offshore until satellite reception improves.  This is challenging for a machine learning model to interpret. There are too many gaps in the data, which makes it hard to predict.

A way to solve this is to normalize the data and generate fixed-sized hourly windows. Then the model can predict if the vessel is fishing or not fishing for each hour.

It could be hard to know if a ship is fishing or not by just looking at its current position, speed, and direction. So we look at the data from the past as well, looking at the future could also be an option if we don’t need to do real time predictions. For this sample, it seemed reasonable to look 24 hours into the past to make a prediction. This means we need at least 25 hours of data to make a prediction for a single hour (24 hours in the past + 1 current hour). But we could predict longer time sequences as well. In general, to get hourly predictions, we need (n+24) hours of data.

Options to deploy and access the model

For this sample specifically we used Cloud Run to host the model as a web app so that other apps can call it to make predictions on an ongoing basis; this is our favorite in terms of pricing if you need to access your model from the internet over an extended period of time (charged per prediction request). You can also host it directly from Vertex AI where you trained and built the model, just note there is an hourly cost for using those VMs even if they are idle. If you do not need to access the model over the internet, you can make predictions locally or download the model onto a microcontroller if you have an IoT sensor strategy.

Want to go deeper?

If you found this project interesting and would like to dive deeper either into the specifics of the thought process behind each step of this solution or even run through the code in your own project (or test project); we invite you to check out our interactive sample hosted on Colab, which is a free Jupyter notebook.  It serves as a guide with all the steps to run the sample, including visualizing the predictions on a dynamically moving map using an open source Python library called Folium. 

There’s no prior experience required! Just click “open in Colab” which is linked at the bottom of GitHub.

You will need a Google Cloud Platform project. If you do not have a Google Cloud project you can create one with the free $300 Google Cloud credit, you just need to ensure you set up billing, and later delete the project after testing the desired sample.

🌏🌎🌍 We hope to inspire you to build other beautiful climate-related solutions.