When moving software applications from on-premise hardware to the cloud, it often “just works,” but it’s never guaranteed. This is especially the case for applications that are hardware intensive. This blog post examines what happened when a media company took software for real-time video broadcasts into the cloud. We’ll share how we, Google Cloud, collaborated with media software provider Vizrt to meet the demanding requirements of an eSports broadcaster. Together, we delivered a solution that not only met but exceeded the expected performance from a cloud-based deployment.

Video broadcasting in the cloud

Video broadcasts are a very hardware-intensive workflow. By needing to process and store data streams in near-real-time, broadcasts stress GPU, memory, disk, and CPU. In addition, the performance requirements quickly increase as producers add additional video streams into the mix, such as in this case, an eSports broadcast.

Vizrt’s customer wanted to increase their broadcast production by doubling the amount of camera feeds from 8 to 16, to have a more compelling and elaborate production.

At the heart of the eSports broadcasters’ production was Viz Vectar Plus, Vizrt’s software-based 4K switcher. While the client wanted to move more of its production into the cloud, Viz Vectar Plus was designed initially for on-premise hardware. So, when they tried a straightforward “lift and shift ” to the cloud, it surprised no one that the software didn’t run as well. They turned to Google Cloud and Vizrt to make it run the way they needed it to. 

Troubleshooting lift and shift

Initially, we suspected that the issue could be in the design of the cloud deployment, i.e., the configuration of the VM hosting the software. So the focus of our troubleshooting was to find a cloud configuration that 1) made sure the software worked to spec and 2) did so optimally considering costs, robustness, and performance. Furthermore, we wanted to ensure that all components met performance specifications, particularly throughput, IOPS, and network bandwidth, as this was a video media application. Only after we validated the cloud deployment would we ask Vizrt to investigate the code itself. We would:

1. Set up a test environment.
2. Benchmark the environment.
3. Test various configurations.
4. Validate that the vendor software was optimally using the configuration.  

This high-level methodology is straightforward. However, we approached the details in a particular order, considering we were optimizing for broadcast video. We honed in on the optimal cloud configuration by testing the following elements, prioritized in order of expected impact:  

1. VM type: The VM type largely dictates the available memory and CPU configurations. However, because this was a VM workflow, we had to pick N1’s. Today, they are the only VM type that can be attached to GPUs, which are practically a requirement for broadcast video.  
2. Disk type: Video broadcasts require high I/O speeds to handle high-quality video streams. We went from an HDD to a much-faster SSD. 
3. CPU size: We increased the VM CPU cores from 16 in increments up to 32. Increasing CPU size indeed increased performance, but did not return the level of performance we needed.
4. SSD size: We increased the SSD size (and the accompanying higher IOPS and throughput that comes with increasing the size) enabling more simultaneous recordings. Again, this only partially worked. 
5. Disk count. We noticed read/write problems when the application was reading/writing with a single drive. There are two typical ways to approach this: 1) Separate read/writes tasks among two discs and 2) striping the data streams across discs. Implementing these had improved but marginal improvements in performances.  

After our testing, we arrived at the following optimal configuration:

• VM: n1-standard-32 instance w/ 500Gb boot drive
• GPU: 1 T4 GPU 
• SSD: 1 persistent disk with 1TB*

* We would later determine that two SSDs for separate read and write operations would be more optimal

This configuration was able to produce between 6 – 12 streams. Compared to the on-premise target of 8 streams, this was about as good but was not the customer’s target of 16. So we would need Vizrt to take the ball from here to optimize the software itself.

Optimizing media applications for cloud

We provided Vizrt our recommended configuration, performance notes, and the following best practices that are generally applicable to cloud-based video workloads:

• Separate read and write operations to two different disks to enable higher performance for both operations. 
• A second 1 TB SSD persistent disk can be attached to the VM instance to increase performance.

With this information, Vizrt engineers worked their magic, providing daily patches to test; with each daily iteration the overall solution was found quickly. Not only were they able to meet the broadcaster’s request of 16 feeds, but they were also able to go even further to 44. Over a 5x improvement by optimizing for the cloud! 

Teamwork in troubleshooting 

Because of the specialized nature of media and entertainment, workflow situations across multiple companies are common as specialized applications hand their work from one to another. 

“By working in partnership with Google Cloud we managed to build a system that can scale in ways that probably none of us thought would be possible. This allowed Viz Vectar Plus to run fully in the cloud using NDI and opened up amazing possibilities for making shows,” Dr. Andrew Cross, President R&D, Vizrt Group. “We ended up with great feedback from the customer, who were appreciative of how Google Cloud and Vizrt collaborated on a solution.”

The results speak for themselves: A satisfied customer with over a 5x improvement in results. That’s what we call a good game.

About Grofers

Grofers is an India-based local mobile e-commerce platform that delivers groceries, fruits and vegetables, cosmetics and electronics. Customers use Grofers’ mobile app to shop for products from local stores in 17 cities in India.

Industries: Services

Location: India

India-based Grofers is a hyperlocal mobile platform that delivers groceries, fruits and vegetables, cosmetics and electronics. Customers use Grofers’ mobile app to shop for products from local stores in 17 cities in India.

In 2015, the number of deliveries Grofers fulfilled was growing at the rate of 10 times a month, and its delivery system struggled under the stress. When traffic spiked on weekends, its database frequently crashed. Fulfilling and delivering orders depended on manual processes such as telephone calls. Grofers needed a solution that would allow it to quickly scale and improve its delivery process.

Building an entire backend in two weeks

When Grofers launched in 2013, the logistics system was entirely manual. Staff at the headquarters called delivery drivers with order details and delivery addresses. Drivers then decided what order to pick up goods from local merchants. With a small number of customers and orders, that wasn’t a problem. But the manual system couldn’t handle growth. Grofers recognized it had to automate the system.

To do that, it turned to Google Cloud Platform. Grofers built a new backend for its delivery system using Google App Engine. When an order arrives, the new system looks at the status of all drivers — including their current location, the order he’s completing, and the pickup and delivery locations on the route. Based on that information, the system calculates the optimal driver for each order and sends the driver the order via a mobile app.

The system also takes advantage of Google Cloud Endpoints, which generates libraries for Javascript, iOS and Android. Grofers used Google Cloud Endpoints to create mobile apps to communicate with its delivery staff.

Scaling fast, dramatically improving delivery metrics

With the new backend and mobile apps, Grofers can scale quickly to keep up with its rapid growth. Grofers now delivers 70,000 orders per day, up from 10,000 before the GCP deployment, and has had 100 percent uptime. Each driver handles an average of 15 orders per day, compared to only five previously, letting Grofers handle more work with less staff. All this has been accomplished at very low cost: Prakash says that GCP costs account for only two percent of the company’s infrastructure spending. Without GCP, Prakash says the backend would be far more expensive and account for up to 50 percent of the company’s infrastructure expenses.

Is Cloud an option for Canadian Healthcare healthcare and medical research organizations?

Yes, Canadian healthcare and medical research organizations are moving to the cloud. The cloud market is expected to grow in Canada significantly through 2027.

There are several reasons why Canadian healthcare and medical research organizations are moving to the cloud. 

• Reduce costs by eliminating the need to invest in and maintain on-premises infrastructure. 
• Enable the healthcare research community to drive their research more expediently to clinical outcomes
• Improve patient satisfaction by making it easier for patients to access their health information and communicate with their providers.
• Improve the quality of care by providing access to patient data and records from anywhere in the country. 

Overall, the transition to the cloud is a positive development for Canadian healthcare and medical research organizations. 

Canadian healthcare providers face many challenges before they can move to the cloud, such as addressing security and privacy concerns, data sovereignty issues, and ensuring interoperability. To help them overcome these challenges, it is important to provide Healthcare Data Custodians, Infrastructure Architects, and Research Leads with clear guidance on how the cloud can align with Canadian Healthcare Regulations. This will allow them to have a practical understanding of what is required to enhance their cloud journey and facilitate a smoother transition to the cloud.

iSecurity and MD+A Health are actively assisting Canadian healthcare and medical research organizations in comprehending the risks and exploring pathways to embrace the cloud. Through extensive research and analysis, iSecurity and MD+A Health have evaluated Google Cloud as a suitable platform for healthcare. Their diligent efforts have resulted in the production of comprehensive documents that detail their findings via a Threat Risk Assessment (TRA) and a Privacy Impact Report (PIA).

Why a Threat Risk Assessment? 

A threat risk assessment is a process of identifying and evaluating threats to an organization and then determining the likelihood and impact of those threats. The goal of a threat risk assessment is to identify the most serious threats and develop mitigation strategies to reduce the likelihood and impact of those threats.

A threat risk assessment typically involves the following steps:

1. Identify threats: The first step is to identify all potential threats to the organization. This can be done by brainstorming, interviewing experts, or reviewing historical data.
2. Evaluate threats: Once the threats have been identified, they need to be evaluated in terms of their likelihood and impact. The likelihood of a threat is the probability that it will occur, while the impact of a threat is the severity of the consequences if it does occur.
3. Prioritize threats: The threats need to be prioritized based on their likelihood and impact. The most serious threats should be addressed first.
4. Develop mitigation strategies: Once the threats have been prioritized, mitigation strategies need to be developed to reduce the likelihood and impact of those threats. Mitigation strategies can include things like implementing security controls, training employees, and developing contingency plans.
5. Implement mitigation strategies: The mitigation strategies need to be implemented and tested to ensure that they are effective.
6. Monitor and review: The threat risk assessment should be monitored and reviewed regularly to ensure that it is still effective.

Why a Privacy Impact Assessment?

A Privacy Impact Assessment (PIA) is a process that organizations use to identify and assess the privacy risks associated with a new or changed information technology (IT) system or project. The goal of a PIA is to help organizations protect the privacy of individuals whose personal information is collected, used, or disclosed by the IT system or project.

PIAs typically include the following steps:

1. Identifying the purpose of the IT system or project and the types of personal information that will be collected, used, or disclosed.
2. Identifying the privacy risks associated with the IT system or project.
3. Assessing the likelihood and severity of the risks.
4. Developing and implementing controls to mitigate the risks.
5. Monitoring the effectiveness of the controls.

PIAs are an important tool for organizations to help them follow privacy laws and regulations. They can also help organizations build trust with their patients, employees and the research community by demonstrating their commitment to protecting privacy.

The benefits of conducting a PIA:

• Helps organizations identify and assess privacy risks
• Helps organizations develop and implement controls to mitigate privacy risks
• Helps organizations comply with privacy laws and regulations
• Helps organizations build trust with customers and employees

Why Google Cloud?

Google Cloud is committed to providing Canadian healthcare organisations with an environment to expand both their clinical and research environments. Google Cloud has invested significant resources into building out a cloud environment based on best practices coming from Google’s experience running some of the world’s largest platforms.  

Some highlights include:

• Built-in security features that help protect your data and applications from unauthorized access, use, disclosure, disruption, modification, or destruction.
• A comprehensive security management platform that helps you assess, prioritize, and address security risks across your organization.
• A team of security experts who can help you design, implement, and manage your security solutions.
• A wide range of security training and resources to help you learn about and stay up-to-date on the latest security threats and best practices.