VM End-to-end: Series Transcript on Conversation on VMs and their Role to Cloud-native Future - Build What's Next
Podcast

VM End-to-end: Series Transcript on Conversation on VMs and their Role to Cloud-native Future

3294

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

Missed the first episode of VM End-to-End, a conversation between a VM enthusiast and a skeptic? Here's the transcript. Read through it to see what can get you excited about VMs and their relevance in the cloud-native future!

VMs and their relevance to a cloud-native future: A conversation

Last week, we published the first episode of VM End-to-End, a series of curated conversations between a “VM skeptic” and a “VM enthusiast”. Join Brian and Carter as they explore why VMs are some of Google’s most trusted and reliable offerings, and how VMs benefit companies operating at scale in the cloud. Here’s a transcript of the first episode:


Carter Morgan (VM skeptic): My team asked me to research VMs and see how they compare to other cloud-native approaches, like microservices, containers, and serverless. And to be honest with you, I am not excited about it at all. So I’ve brought in someone who is, Brian, resident VM expert. Welcome, Brian!

Brian Dorsey (VM enthusiast): Hello. I’m super happy to be here, because I am really excited about VMs.

Carter: How? See this is why I wanted to bring you here. I don’t understand how you can be so excited about VMs, already?

Brian: How can you not? It is like the best of both worlds. You’ve got a stable, reliable system you can run anything on, and you’re in the cloud, close to all of the new features, and you get new automation tooling.

Carter: See, that’s what I’m kind of skeptical about. When I think about the features of a modern system, I’m not sure that a VM can provide me with those.

Brian: Okay. Well, let’s be kind of specific there. What do you mean by a modern system?

Carter: When I think about a modern system, I think about things like modularity. I think about scalability and reliability. I want automation. I don’t want to have to do everything manually. I even think about portability and being able to move my workloads wherever they need to go. I also don’t want to implement everything by hand. I don’t want to have to do everything myself. Is that something I can get with a VM?

Brian: Yeah. Great, because I actually think we can get most of that, and I wonder, why not? Let’s go one level deeper and then we’ll come back out. In your mind, what is a VM?

Carter: Oh, you’re putting me on the spot, Brian?!

Brian: Yep.

Carter: Okay. A VM, it’s a computer, but it’s a virtualization. It’s a slice of a computer. And so, what it lets you do is it lets you run multiple operating systems on one machine, so it looks like you have multiple machines running on one physical machine.

Brian: Yep. Absolutely. And in the cloud kind of not.

Carter: Oh, what?

Brian: Yeah. So here’s why I say not. The obstructions are all there, so you’ve got memory CPU, disk, networking. And instead of from one computer, in the cloud, that’s coming from all of the computers in a data center. So the CPU is coming from a lot of machines. The networking is from the whole data center. And so, I like to think about it as instead of a slice of a computer, it’s a slice of the data center

Carter: Instead of a slice of a computer, it’s a slice of a data center. That sounds interesting. Impressive, even. But also abstract. Do you have some examples of features that you can get from a cloud VM that you can’t get from a traditional VM on one machine?

Brian: My turn to be specific. Yeah. I think one example is like bin packing. You talk about these VMs, and their different shapes. So you have one that needs a lot of CPU and another that needs a lot of memory. Maybe you’ve got a bunch of them that need a lot of CPU and they don’t fit so well in the same box without orphaning some of the memory or CPU. And if you’re running those in a whole data center, or you can basically just leave it to Google to solve that problem, you can just have whatever shape machine you want and we’ll figure out where to put it. Basically, that lets you customize your machines to exactly what you need.

Carter: Okay. That’s very interesting, because that’s a hard problem. And so, if you can just let Google handle where your workloads are going to go, that’s a good benefit. If that’s the only benefit of cloud VMs though, I’m not sure I’m sold on them over other approaches. Is there anything else we got?

Brian: Absolutely. There’s a ton of stuff we could talk about in terms of automation and other things. But I think another really concrete example is disks, and it’s kind of my favorite there, because you think about a physical disk and you read and write blocks from it, right? It’s a block device. In the data center level, those blocks could be on hundreds or thousands of different machines, and so all of them are working together to give you more reliability and make things smoother, more predictable in terms of performance. So what you get out of it is something that looks a lot like a SAN, you can take backups of disks that are running even, or if you’ve run out of space, like I think almost all of us have, you can just make the disk bigger. So, things like that.

Carter: That’s impressive, especially like you’re saying being able to run and just scale up, scale down or resize. Okay. Then another very targeted question. It’s going to sound like a dig. I don’t mean it to you. Google’s putting a lot of effort and resources into Google Kubernetes Engine (GKE). And so, is Google even still investing in VMs?

Brian: Absolutely. Where do you think these containers run? Every Kubernetes cluster is running on top of a whole bunch of VMs. And so, everything you learned about VMs applies to those clusters that you’re running. Also, another example is our managed databases, so Cloud SQL. So if you’re running Postgres or MySQL on a Cloud SQL, that’s running on VMs. And there’s a bunch of other examples, too.

Carter: I can’t get away from VMs, even if I tried, it sounds like.

Brian: Nope.

Carter: Okay. The way you’re saying that is making me think that maybe I need to rethink my idea that VMs are just old, dusty pieces of technology. So I want to be clear, definitively, are you saying that VMs have a place in a cloud-native future?

Brian: Absolutely. I want to take old and dusty and turn that into mature and reliable. Then the future part, all these things are built on top of it and we’re building new things over time. So we’ve got tools for scaling clusters of machines up and down. That’s using Kubernetes, but you can use it directly. And we keep investing and doing more and more things there. So, absolutely, part of the future as well.

Carter: Okay. Then what about if I wanted to switch to them? I’m pretty familiar with Kubernetes. It’s fairly easy to get started. What about with VM? Is it going to take me years to get started on these?

Brian: No. It’s just a computer. Basically, anything that’s already running on a computer somewhere, even if you don’t have the team who built it nearby, you can run that on a VM and in turn you can run it on a cloud VM and get a bunch of the cloud benefits as well.

Carter: All right. I must admit that you’ve swayed me a little bit. I’m still skeptical, but what you said made a lot of sense. I still have a lot of questions. I want to know about keeping costs down. I want to know how to update VMs. I have this idea in my head that they’re really slow to start and stop. Stateful data, I’m curious about that.

Brian: Awesome. How much time do you have?

Carter: All right. You know what, let’s have this convo another day, and maybe, just maybe we can agree that VMs do matter in a cloud-native future.

How-to

What You Can Learn from ‘Up or Out’ Framework for Cloud Adoption

3231

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

When it comes to cloud adoption, there can be no 'one-size-fits-all' approach. Google Cloud's detailed white paper explores the up, our and both framework to help your organization decide the most palatable way to benefit from cloud migration.

In times of significant disruption, organizations are faced with three choices: Retrench within legacy solutions, pause and do nothing while waiting for more data or different circumstances, or press ahead, potentially even accelerating to realize the desired outcome. In such an environment, it is critical to ensure you’re delivering the greatest possible impact to the business. 

In Google Cloud’s Office of the CTO, or OCTO, we have the privilege of co-innovating with customers to explore what’s possible and how we can re-imagine and solve their most strategic challenges. These collaborative innovation engagements are often core to critical transformational projects, which often include the rehosting, evolution, and at times re-architecture of existing business solutions. 

We are happy to offer this brand new white paper, where we have distilled the conversations we’ve had with CIOs, CTOs, and their technical staff into several frameworks that can help cut through the hype and the technical complexity, to help devise the strategy that empowers both the business and IT. We called one such framework “up or out.” (And we don’t mean some consulting firm’s hard-nosed career philosophy.) 

One model that we found can help enterprises chart their cloud adoption journey delineates cloud migration along two axes—up and out, and we’ll cover this in much greater detail in the white paper itself.

cloud migration.jpg
Click to enlarge

As you can see, there isn’t a single path to the cloud—not for individual enterprises and not even for individual applications. The up or out framework can help an IT organization and its leadership characterize how they can best benefit from migrating their services or workloads. The framework acts as a general pattern that highlights the continuum of approaches to explore, and you can learn all about it by downloading this detailed white paper.

Or, if you’re really ready to jump start your migration today, you can take advantage of our current offer by signing up for a free discovery and assessment.

Research Reports

AI in Manufacturing Already A Mainstream: Google Cloud Study

5680

Of your peers have already read this article.

3:00 Minutes

The most insightful time you'll spend today!

Google Cloud's latest research unveiled nearly 76 percent of the manufacturers across 7 countries turned to AI and other digital enablers during the pandemic. The study also found that 66 percent of manufacturers relied on AI for daily operations.

While the promise of artificial intelligence transforming the manufacturing industry is not new, long-ongoing experimentation hasn’t yet led to widespread business benefits. Manufacturers remain in “pilot purgatory,” as Gartner reports that only 21% of companies in the industry have active AI initiatives in production

However, new research from Google Cloud reveals that the COVID-19 pandemic may have spurred a significant increase in the use of AI and other digital enablers among manufacturers. According to our data—which polled more than 1,000 senior manufacturing executives across seven countries—76% have turned to digital enablers and disruptive technologies due to the pandemic such as data and analytics, cloud, and artificial intelligence (AI). And 66% of manufacturers who use AI in their day-to-day operations report that their reliance on AI is increasing.

1 AI acceleration in manufacturing.jpg
Click to enlarge

The top three sub-sectors deploying AI to assist in day-to-day operations are automotive/OEMs (76%), automotive suppliers (68%), and heavy machinery (67%).

2 AI acceleration in manufacturing.jpg
Click to enlarge

In fact, Bryan Goodman, Director of Artificial Intelligence and Cloud, Ford Global Data & Insight and Analytics shares, “Our new relationship with Google will supercharge our efforts to democratize AI across our business, from the plant floor to vehicles to dealerships. We used to count the number of AI and machine learning projects at Ford. Now it’s so commonplace that it’s like asking how many people are using math. This includes an AI ecosystem that is fueled by data, and that powers a ‘digital network flywheel.’”

Moving from edge cases to mainstream business needs

Why are manufacturers now turning to AI in increasing numbers? Our research shows that companies who currently use AI in day-to-day operations are looking for assistance with business continuity (38%), helping make employees more efficient (38%), and to be helpful for employees overall (34%). It’s clear that AI/ML technology can augment manufacturing employees’ efforts, whether by providing prescriptive analytics like real-time guidance and training, flagging safety hazards, or detecting potential defects on the assembly line.

3 AI acceleration in manufacturing.jpg
Click to enlarge

In terms of specific AI use cases called out by the research, two main areas emerged: quality control and supply chain optimization. In the quality control category, 39% of surveyed manufacturers who use AI in their day-to-day operations use it for quality inspection and 35% for product and/or production line quality checks. At Google Cloud, we often speak with manufacturers about AI for visual inspection of finished products. Using AI vision, production line workers can spend less time on repetitive product inspections and can instead focus on more complex tasks, such as root cause analysis. 

In the supply chain optimization category, manufacturers said they tapped AI for supply chain management (36%), risk management (36%), and inventory management (34%).

4 AI acceleration in manufacturing.jpg
Click to enlarge

In our day-to-day work, we’re seeing many manufacturers rethink their supply chains and operating models to better accommodate for the increased volatility that has been brought about by the pandemic and support the secular trend of consumers asking for increasingly individualized products. We’ll share more on deglobalization in the third installment of our manufacturing insights series.

AI use differs by geography, but not for the reasons you may think

The extent to which AI is already being used today varies quite strongly between geographies, according to our research. While 80% and 79% of manufacturers in Italy and Germany respectively report using AI in day-to-day operations, that percentage plummets in the United States (64%), Japan (50%) and Korea (39%).

5 AI acceleration in manufacturing.jpg
Click to enlarge

It’s tempting to state this disparity is due to an “AI talent gap.” Although the most common barrier, just a quarter (23%) of manufacturers surveyed believe they don’t have the talent to properly leverage AI. Cost, too, does not appear to be a roadblock (21% of those surveyed). Rather, from our observations, the missing link appears to be having the right technology platform and tools to manage a production-grade AI pipeline. This is obviously the focus of our efforts and others in the space, as we believe the cloud can truly help the industry make a step change.

Looking ahead: The Golden Age of AI for manufacturing

The key to widespread adoption of AI lies in its ease of deployment and use. As AI becomes more pervasive in solving real-world problems for manufacturers, we see the industry moving away from “pilot purgatory” to the “golden age of AI.” The manufacturing industry is no stranger to innovation, from the days of mass production, to lean manufacturing, six sigma and, more recently, enterprise resource planning. AI promises to bring even more innovation to the forefront. 

To learn more about these findings and more, download our infographic here and our full report here


Research methodology
The survey was conducted online by The Harris Poll on behalf of Google Cloud, from October 15 – November 4, 2020, among 1,154 senior manufacturing executives in France (n=150), Germany (n=200), Italy (n=154), Japan (n=150), South Korea (n=150), the UK (n=150), and the U.S. (n=200) who are employed full-time at a company with more than 500 employees, and who work in the manufacturing industry with a title of director level or higher. The data in each country were weighted by number of employees to bring them into line with actual company size proportions in the population. A global post-weight was applied to ensure equal weight of each country in the global total.

3800

Of your peers have already watched this video.

8:00 Minutes

The most insightful time you'll spend today!

Case Study

Target Leverages Google Cloud to Create Market-defining Online Experience

In the hyper-competitive world of online retail sales, ease-of-use and transaction speed can make or break business outcomes. However, a few years ago US Retail giant Target was going through a period of uncertainty.

While the company had over 1800 stores across the US with an estimated 85% of US consumers shopping at a Target store and over 25 million people visiting the Target website or using its app each month, it was still losing ground.

In spite of having millions of loyal customers, the company was dangerously late on digital and its technology wasn’t keeping pace with unstable systems to boot. The company faced the twin challenges of trying to operate today’s business as efficiently as possible and creating tomorrow’s business as quickly as possible. On the one hand it needed productivity and stability and on the other it wanted speed and disruption. Not an easy task to accomplish.

That’s when Target decided to use Google Cloud to solve its challenges. See how Target leveraged Google Cloud to create a market-defining online experience that has made customers happier and more loyal.

Case Study

IT Team Figures Out Easiest Way to Build Data Pipelines and Create ML Models

14107

Of your peers have already read this article.

5:15 Minutes

The most insightful time you'll spend today!

To give brands greater agility to rapidly create high-impact customer experiences and increase its own competitive edge, Brandfolder moved to Google Cloud Platform, using AI-powered solutions and fully managed cloud services to enable an efficient and focused development team to improve customer experiences.

Building a strong brand in today’s hyper-competitive business environment takes vision. It also requires a flexible, easily managed approach to digital asset management (DAM), so marketing professionals and other stakeholders can easily share, store, track, and manipulate assets to build the brand.

Many of today’s leading companies, including JetBlue, Slack, TripAdvisor, Lyft, and HealthONE, rely on Brandfolder to deliver consistent, organized, and efficient brand experiences. Brandfolder provides an easy-to-use platform that can scale across an entire company with little end-user training, empowering customers to distribute digital assets wherever they are needed. Customers also gain much greater insight into how those assets are used, and how to use them more effectively in marketing campaigns and brand messaging.

“Google Cloud made it easy to build an ML platform to quickly iterate through different brand intelligence use cases and release data-driven product features into the Brandfolder platform.”

Ajay Rajasekharan, Head of Data Science, Brandfolder

Brandfolder is constantly advancing its development efforts to introduce new data-driven features without complicating the user experience. Big data, artificial intelligence (AI), and machine learning (ML) are key to meeting customers’ unique business needs, and essential for Brandfolder to compete in the fast-moving DAM industry. To enhance these capabilities, Brandfolder sought a public cloud provider that could help it scale its data pipeline cost effectively while providing access to advanced AI technologies.

After graduating from the Techstars startup accelerator program in 2013, Brandfolder tried two other cloud providers before standardizing on Google Cloud Platform (GCP).

“We saw a difference with Google Cloud from the very beginning because the interactions felt like a strategic relationship,” says Jim Hanifen, Head of Product at Brandfolder. “Google gave us startup credits and a lot of face-to-face support, which we hadn’t experienced with other cloud providers. We decided to move our entire infrastructure to Google Cloud Platform.”

Building an ML platform for brand intelligence

After performing an initial lift-and-shift migration of virtual machines (VMs) onto Compute Engine, Brandfolder built an ML platform using GCP managed services to seamlessly deliver its data products. The platform leverages Cloud SQLCloud Storage as the data lake, Cloud Dataproc for cloud-native Apache Spark computing clusters, Cloud Composer as the batch job scheduler, Cloud Pub/Sub as the backbone data pipeline, Container Registry to store Docker images, and Google Kubernetes Engine (GKE) as the application orchestrator. Cloud Dataflow brings data into the data lake and into BigQuery for analysis.

“Google Cloud made it easy to build an ML platform to quickly iterate through different brand intelligence use cases and release data-driven product features into the Brandfolder platform,” says Ajay Rajasekharan, Head of Data Science at Brandfolder, who describes the architecture in a detailed blog. “We simply ingest raw application and event data on one end and output an ML service on the other.”

“Moving to Google Cloud Platform allows us to complete more sophisticated data analysis and ML models much faster, and at a much lower cost. We can create brand-specific ML models 12x faster and get them into production quickly to address our customers’ unique business needs.”

Brett Nekolny, Head of Engineering, Brandfolder

For many general use cases, Brandfolder does not need to build custom ML models, and instead relies on pre-trained API models from GCP. For example, it uses Vision API and Video Intelligence API to auto-tag creative assets on import to enable fast, intuitive searches across images and videos. When more product- and brand-specific modeling is required to address unique customer use cases, Brandfolder builds and trains custom ML models using its GCP pipeline or Cloud AutoML, a suite of products built on Google transfer learning and neural architecture search technology. For example, if a Brandfolder customer makes different types of grills, Brandfolder can use AutoML Vision to train a model to recognize the different grills.

“Moving to Google Cloud Platform allows us to complete more sophisticated data analysis and ML models much faster, and at a much lower cost,” explains Brett Nekolny, Head of Engineering at Brandfolder. “We can create brand-specific ML models 12x faster and get them into production quickly to address our customers’ unique business needs.”

Industry-leading security and performance

Google Cloud’s security model helps Brandfolder give existing and prospective customers peace of mind that their data will be protected. Cloud Identity & Access Management (Cloud IAM) provides enterprise-grade access control, while Cloud Identity-Aware Proxy (Cloud IAP) enables remote users to work more securely without the hassles of a VPN client. GCP also isolates cloud resources into projects, making it easy to assign permissions and keep data and VMs organized and segregated.

“With Google Cloud, everything begins and ends with security, which makes things very easy for us,” says Jim. “If we’re under a security review, we can submit a Google security white paper. If a potential customer has security concerns, we tell them we are hosted on GCP, and those concerns go away.”

To give customers even better application performance for accessing their brand assets, Brandfolder uses Cloud Memorystore, an in-memory data store service for Redis, to cache data and provide sub-millisecond data access for production applications.

“It was much easier for us to use Cloud Memorystore versus running Redis on our compute instances,” says Brett. “The high availability, replication across zones, and automatic failover with no data loss are big for us.”

Global private network interconnects between Google Cloud and the Fastly content delivery network (CDN) dramatically reduce latency, allowing Brandfolder’s customers to deliver and update even very large creative assets quickly around the world.

“What’s beautiful about the relationship between Google and Fastly is that if one of our customers uploads a new version of an asset, we can propagate that out to Fastly, and the new version will automatically show up in all the places where it’s referenced,” says Brett.

“The ability to quickly solve problems with AI has a substantial impact on our revenue, and that’s more apparent every quarter. Few of our competitors are doing product- or brand-specific modeling because it takes a lot of time and resources. We overcame those hurdles with Google Cloud.”

Jim Hanifen, Head of Product, Brandfolder

Improving employee and customer productivity

Brandfolder also uses Google solutions for real-time collaboration and productivity, using G Suite to connect employees with intuitive, cloud-based apps. Teams use GmailCalendarDocsDriveSheetsSlides, and Hangouts Meet every day to move the business forward. Many of Brandfolder’s customers are also G Suite users, and Brandfolder offers a plug-in that allows them to view their creative assets inside of Docs and pull images in as needed. Customers can also log into Brandfolder with their G Suite credentials, making the solution even easier to use.

“We’ve been using G Suite since the beginning, and it’s helped us collaborate efficiently to build a successful, growing company,” says Jim. “Our teams expect to have that kind of close collaboration, and everyone here enjoys the G Suite experience.”

Driving 99 percent annual business growth

With automated tagging and other innovative AI-based features, Brandfolder is helping customers locate and distribute assets faster. As a result, Brandfolder is building customer loyalty and increasing sales, growing its business by 99 percent year-over-year. Since moving to GCP, Brandfolder has been able to scale its analytics and data pipeline 50x without a corresponding increase in costs and has not had to expand its development team.

“The ability to quickly solve problems with AI has a substantial impact on our revenue, and that’s more apparent every quarter,” says Jim. “Few of our competitors are doing product- or brand-specific modeling because it takes a lot of time and resources. We overcame those hurdles with Google Cloud.”

Case Study

How One Company Uses AI and Data Analysis to Boost Revenue

9469

Of your peers have already read this article.

4:30 Minutes

The most insightful time you'll spend today!

With Google Cloud, ViSenze has created a data platform that can ingest and process 500 million records per day and store data for up to one year, while giving non-technical team members the ability to generate detailed, insightful reports.

AI, deep learning, and image recognition is transforming the shopping experience. These technologies enable consumers to use product images or screenshots rather than text to search for similar products. This improves the customer experience and enables retailers with online and offline outlets to provide a genuine omnichannel experience.

The lack of complexity and the ease of use of BigQuery has enabled ViSenze to reduce its data infrastructure and management costs by 30%–50%, and scale up without incurring downtime.

—Renjie Yao, Data Platform Lead, ViSenze

Visual commerce provider ViSenze is helping some of the world’s leading retailers improve conversion rates through image-based search.

The business’s products also enable media companies to use the platform to turn images and videos into engagement opportunities—driving new and incremental revenues.

Created through NExT—a research center established by the National University of Singapore and Tsinghua University of China—ViSenze now operates in the United States, United Kingdom, India, China, and Singapore. The business is backed by Japan-based internet and ecommerce company Rakuten and cross-border investment specialist WI Harper Group.

Growth in SMB and Mobiles

Renjie Yao, Data Platform Lead at ViSenze, sees opportunities for growth in the small-to-medium business sector, where companies do not have the resources to build similar technologies, and with mobile device OEMs to integrate ViSenze natively on smartphones.

Phone owners can activate a “shopping lens” on camera and gallery apps to capture an image of a product. They then receive matching results from more than 800 partner merchants and retailers and can then click through to product pages on partner apps or mobile websites. Alternatively, they may use a photo to compare products sold on different sites or shop matching styles.

“Our research found Google Cloud provided a complete, integrated ecosystem rather than a disparate collection of tools and components, and so was ideal for our needs.”

—Renjie Yao, Data Platform Lead, ViSenze

The ViSenze API analyzes the contents of a selected or clicked image and sends the information back to the organization’s visual commerce platform. The platform feeds back similar results based on that information.

The ViSenze offering also extends to image analysis for the tagging of product attributes—such as a white turtleneck cardigan with full sleeves—to provide an improved search experience.

Data Vital to ViSenze

Capturing and analyzing large volumes of data is integral to ViSenze. “We have to understand how consumers interact with our customers’ ecommerce websites and apps,” says Yao. “For example, we need to know who has looked at a particular pair of jeans on a website and whether that visit led to a conversion. We can then tell that customer whether they need to make more stock available.”

ViSenze also relies on data to provide high-quality training for its image recognition models and its domain-specific models for online retail.

Protect Customer Data

Data is vital to ViSenze—but customer privacy is most important. “All the data we collect is transparent to our customers, meaning they can decide what they do not want us to collect. In addition, all personal data processing complies with privacy protection regulations in each region, such as the General Data Protection Regulation in Europe.”

A Quick Move to the Cloud

ViSenze started operations using servers, storage, networking, and associated systems in an on-premises data center operated by NExT.

However, to support rapid growth, the business decided to move its workloads to the cloud. ViSenze opted for a multi-cloud architecture, using in part a Google Cloud data infrastructure.

“Our research found Google Cloud provided a complete, integrated ecosystem rather than a disparate collection of tools and components, and so was ideal for our needs,” says Yao. “We could connect different components with the click of a mouse.” Further, the business found it could easily configure rules and pipelines to route data logs to relevant Google Cloud services.

The review found Google Cloud’s extensive managed services would also remove administration and maintenance tasks from ViSenze’s in-house technology team—freeing team members to focus on more valuable tasks.

In addition, Google Cloud provided the security features—including custom hardware running hardened operating systems and file systems and encryption of data at rest and in transit—needed to protect sensitive information. Finally, the location of Google Cloud regions in several countries would enable the business to meet regulatory and data sovereignty requirements.

A Three-Month Implementation

ViSenze opted to move to Google Cloud in mid-2017 and completed a three-month implementation using internal resources. “The process was very smooth and intuitive, and we had no problems building our entire data platform within Google Cloud,” says Yao.

The business now uses an architecture comprising Google Kubernetes Engine to manage and orchestrate Docker containers running in Google Cloud Platform; BigQuery to provide an analytics data warehouse, with Google Data Studio providing customizable visualization and reports; Stackdriver to monitor and manage virtual machine instances and services inside Google Cloud; Cloud SQL to manage its relational databases for real-time analytics; Compute Engine to provide compute resources; Cloud Storage to store files and objects; Cloud Pub/Sub to provide real-time messaging between applications; and Cloud Functions to build event-driven applications.

After collecting the request logs of users in virtual machine instances and Docker containers, ViSenze distributes them in three directions. “We export raw logs into Cloud Pub/Sub for indexing inside an Elasticsearch search engine, and to a BigQuery data warehouse for further analytics,” explains Yao. “We also use Cloud Functions-created applications to obtain the logs from Cloud Pub/Sub to perform some real-time calculations.”

“We are currently using Airflow workflow management on Compute Engine as our hosted ETL platform, but are likely to move to Cloud Composer in future.”

500 Million Records Per Day

With Google Cloud providing its data infrastructure, ViSenze is well positioned to meet internal and customer demands for more granular insights. The business is now processing 500 million records per day through BigQuery and saves up to one year’s aggregated data—excluding any personal data—in the data warehouse for analysis.

The nature of ViSenze’s business means most reports are generated for data processed on an hourly, daily, or monthly basis. “BigQuery is extremely stable and performance optimized, regardless of the volume of data it processes,” says Yao. “Across BigQuery and other Google Cloud Platform services, we’ve recorded 99.99% availability over the past year.”

The lack of complexity and the ease of use of BigQuery has enabled ViSenze to reduce its data infrastructure and management costs by 30%–50%, and scale up without incurring downtime.

“With BigQuery, we have saved the equivalent of two full-time engineers and now need only half of one person’s time to maintain our whole data platform,” says Yao.

“In addition, BigQuery integrates closely with Data Studio, enabling non-technical people in our product and business teams to create dynamic, detailed analysis dashboards. We now use Data Studio to create nearly 50 separate reports.”

The business has now grown to offer access to more than 1 billion users and a listing of more than 400 million purchasable products.

Next Steps

ViSenze is now researching the potential of the Cloud AutoML suite of machine learning products to improve the training of its models and run a fully managed NoSQL database through Cloud Datastore.

“A NoSQL database service is the only missing piece of our architecture for now, and using Cloud Datastore would enable us to focus almost exclusively on our business,” says Yao. “With Google Cloud Platform, we are ideally positioned to continue providing support to our business team and help them continue expanding into new markets.

“In addition, we can help retailers and consumers to unlock the potential of the web and apps to transform the purchasing experience.”

More Relevant Stories for Your Company

Blog

Google Cloud & Optiva Partnership Cements the Future of Telecom for Driving Strong Customer Experience

Editor’s note: Coming out of Mobile World Congress 2022, we are excited to share key learnings from our partner ecosystem on how to leverage advancements in 5G technologies to power customer experiences that seamlessly blend our physical and virtual world into one. The original version of this blog was published

Blog

Google’s Latest ‘Carbon Footprint’ can Flag Users about Carbon Emission Levels from their Cloud Usage

Google Cloud is proud to support our customers with the cleanest cloud in the industry. For the past four years, we’ve matched 100% of our electricity use with renewable energy purchases, and we were the first company of our size to commit going even further by running on carbon-free energy 24/7

Blog

A New Milestone: Istio Comes Closer to Cloud-native Ecosystem

Today we are excited to announce that Google and the Istio Steering Committee have submitted the Istio project for consideration as an incubating project within the Cloud Native Computing Foundation (CNCF). This is a significant milestone for Istio and its community, and we are thrilled to reach this next step

Blog

Google’s Research and Data Insights Solutions to Power Drug Development and Clinical Research

In order to be successful, research needs to be replicable, so scientists can build on past work and insights. However, an article in Nature warned that as much as 50% of published drug development research could not be reproduced in subsequent trials. As a result, promising drug candidates sometimes led to disappointment,

SHOW MORE STORIES