About Tencent Africa

Tencent Africa is dedicated to making every day easier for customers and brands. Opening its doors in 2013 as WeChat Africa, the company is responsible for WeChat operations on the continent.

Industries: Technology

Location: South Africa

About Siatik Systems

Founded in 2008, South Africa-based Siatik Systems is a leading Google Cloud Platform solutions provider with a focus on cloud integration services, information technology consulting services, and bespoke software engineering solutions.

Industries:

Location:

As one of Africa’s leading technology companies, Tencent Africa is responsible for WeChat operations on the continent. WeChat Africa has successfully launched a number of features including the WeChat Wallet, a mobile payment service for smartphones, which enables seamless and secure transactions for friends to send money to one another, cash money out at Standard Bank ATMs, use at accredited retailers, and make purchases from any SnapScan merchant in South Africa. Under the new name, Tencent Africa recently launched two new mobile products, JOOX, a mobile music streaming application, and VOOV, a social live streaming application.

When China’s Tencent brought its hugely popular instant messaging service WeChat to South Africa, it faced a near saturated market. To gain traction, Tencent Africa had to showcase WeChat’s versatility as a platform that goes far beyond just instant messaging. Engaging with local financial partners helped WeChat launch a WeChat Wallet to facilitate electronic payments, while providing exclusive content from South African cultural figures helped bring the platform extra publicity.

As WeChat started to grow its customer base, its on-premises infrastructure was stretched beyond its limits and the company looked for a cloud-based solution. After evaluating all its options, Tencent Africa found Google Cloud Platform best suited its needs.

“We’re a growing business and constantly upgrading our infrastructure. Getting new servers was becoming quite expensive so we wanted a cloud-based solution to reduce costs,” says Christoff Albertyn, Head of Technology at Tencent Africa. “When we did some latency testing, Google came out on top, so it made sense to go with them. Everybody loves Google, it’s a great brand. Just knowing a company like Google is behind it gives you peace of mind.”

Building a platform to scale

Instant messaging services need a stable infrastructure, the ability to scale at speed, and lightning fast performance. When Tencent Africa built an Official Accounts API to run a number of services on WeChat, to help companies engage with users on the platform, latency was its main concern. The Official Accounts services had to deliver replies over WeChat in under five seconds or the connection would drop.

Between the API on-premises servers in South Africa and WeChat’s main servers in China, Tencent Africa’s services regularly hit the limits of the permissible response time. In addition, by 2015, WeChat was starting to garner more customers across South Africa and Tencent Africa had to provision more servers. The on-premises servers were simply being asked to do too much, and expansion was proving expensive, so the company looked for a cloud-based solution.

Before committing to anything, Tencent Services did a thorough latency test of several cloud-based solutions. Google Cloud Platform came top. Tencent also engaged the help of Siatik, a Google Partner, to gauge what solutions best fit the company’s needs and help implement them. Siatik customers span both the public and private sectors and its consultants are regularly engaged in the health care, education, financial and technology industries. In order to provide leading technology solutions, it partners with industry leaders, such as Google, to provide best-of-breed infrastructure and cloud platform solutions to its customers.

“Siatik was a great partner to work with,” says Christoff. “Any questions I had, I could just pick up the phone and whatever it was they’d help me out. They were also able to get some of the team from Google to come to our offices and see how we worked, which was really good.”

With a simple, clean interface in Google Compute Engine, the company was able to spin up virtual instances very quickly and begin migrating its stack to Google. When it became more comfortable with Google Cloud Platform features, Tencent used Google Cloud SQL and Google BigQuery to manage its data streams, and Google Cloud Storage to store them safely.

From day one, it was very smooth. The interface was amazing—Google must have spent a long time perfecting it because it was so easy to use says Christoff. “I set up the first virtual machine in no time without any struggles.”

In addition to powerful hardware from Google Compute Engine, the fast, global network from Google helped to speed up performance. With the infrastructure ported over to Google, Tencent also brought in G Suite as an office productivity solution, which eased the pressure on its servers even more. Throughout the process, Siatik were on hand to advise and provide technical guidance whenever any issues came up.

What I really like is how quickly you can spin servers up. There’s no messing about and the interface is really easy to use, says Christoff. “You can have a virtual machine running in just a few minutes. It’s very impressive.”

Cutting costs, raising expectations

Google Cloud Platform helped Tencent Africa to slash its latency. Before the migration, pings to the Chinese servers and back would regularly take 180 milliseconds. Afterwards, this fell to around 35 milliseconds, helping to ensure smooth service to Tencent’s South African customers. Google offers innovative pricing, such as per-minute billing and sustained use discounts, which helped significantly cut server costs compared to an equivalent on-premises solution.

As its customer base grew, and Tencent Africa began to release new products, the ease of use of Google Compute Engine meant that spinning up new servers took minutes instead of hours and allowed the company to scale up and down at speed. Exploring products such as Kubernetes for rapid deployment or TensorFlow for machine learning, Tencent Africa continues to evolve and experiment with new products to keep WeChat a growing success in South Africa.

“Initially, we just used per-minute billing, but when I saw how much work we were doing with Google BigQuery and Google Cloud SQL, we started to generate sustained-use discounts, which helped save us a lot of money,” says Christoff. “The other great thing is, Google Cloud Platform monitors how you’re using it and tells you what else you can do to reduce costs.”

In the fragmented world of U.S. healthcare, patients often have to wait in line or on hold, navigate multiple patient portals, and fill out numerous request forms—all in pursuit of their own medical history. Healthcare technology startup PicnicHealth is on a mission to put control back with the patient, where it belongs.

PicnicHealth’s growth, from closing successful venture rounds to winning machine learning (ML) competitions, speaks to not only improvements and opportunities in healthcare, but also how startups are leveraging Google Workspace and Google Cloud services to accelerate momentum.

The company does the heavy lifting of collecting records and leverages human-in-the-loop ML to transcribe and validate them with an abstraction team of medical professionals. The records are then structured into a complete medical history that patients can access and share with providers to get better care.

But PicnicHealth helps to improve patient health on more than one front. It allows patients to contribute their data to de-identified medical research, building high-quality, anonymized datasets that researchers and life sciences companies can use to better understand disease progression and treatment in the real world.

Google Workspace has been part of PicnicHealth from day one, helping the founders collaborate and shape the company’s vision using cloud-synced documents and spreadsheets to collaborate and model predictions. “I’ve had a Gmail account since 2006, and in college 100% of people worked out of Google Docs. Workspace continues to be the best choice for online collaboration, and that’s why it’s still the default standard for startups,” said Troy Astorino, CoFounder & CTO of PicnicHealth. “When we started PicnicHealth, my co-founder Noga Leviner was in San Francisco and I was in Southern California, and of course we used Workspace.”

Today, Google Workspace continues to play a central role in the company’s collaboration. “We create design documents in Google Docs, primarily for engineering and product changes, and get really healthy, vibrant discussions through comments,” noted Astorino. “This practice has grown beyond engineering and is used for everything from how the company operates to communication norms.”

Google Workspace offers everything the team needs to collaborate, no matter where employees are. PicnicHealth’s team has spread from San Francisco to being distributed across the country and around the world. Instant collaboration is crucial.

“We work in a complex domain where people need a lot of information to make good decisions,” said Astorino. “Google Workspace allows us to operate in a mode of default transparency, where people can easily get the information they need even if it wasn’t intentionally or directly shared with them. Whether it’s working in Docs or scheduling in Calendar, we can operate much more effectively than we could otherwise.”

By any measure, PicnicHealth’s trajectory is one of record success. The startup is a 2014 alumni of Y Combinator, a program that helped launch household names like Airbnb, DoorDash, and Dropbox. Three years later, the team went on to win the $1 million grand prize at Google’s Machine Learning Competition. And the momentum has continued— PicnicHealth has recently announced a $60 million Series C round, bringing the amount raised to date to over $100 million. With the Series C, PicnicHealth is investing in expanding its reach to more patients across over 30 diseases.

As a healthcare startup, PicnicHealth faced a very particular set of challenges, especially when working with and accessing data. Data fragmentation and interoperability are only some of the challenges of realizing the value of big data in the cloud. The healthcare industry is notoriously difficult to navigate due to sensitive data protection laws and regulations like the Health Insurance Portability and Accountability Act (HIPAA).

PicnicHealth started in the cloud on Amazon Web Services (AWS). However, after migrating over to Kubernetes and facing an expanding list of requirements for HIPAA compliance, the company started to explore alternatives.

“We needed to be HIPAA compliant, which was going to be painful on AWS, and we wanted to get away from managing and operating our own Kubernetes clusters,” recalled Astorino. “We had heard good things about GKE (Google Kubernetes Engine). And particularly valuable for us, — many technical requirements you need for HIPAA compliance are configured by default on Google Cloud.”

PicnicHealth would have had to implement a lot of changes and get specialized instance types to get their existing configuration to work. So, they began experimenting with Google Cloud and discovered a much smoother experience.

“It was a lot easier to manage in terms of product setup and developer experience,” said Astorino. “There is a sane product hierarchy of resources you can access and use through Google Cloud and the relationships between them, from coordinated IAM (identity and access management) to using Google Groups for granting permissions. Overall, it’s cleaner.”

Astorino added that the move has also opened the doors to taking advantage of other services in the Google Cloud ecosystem like Cloud SQL, BigQuery, and Cloud Composer. PicnicHealth also uses Security Command Center because it easily integrates with everything but also helps meet various compliance frameworks’ requirements, providing visibility, near-real-time asset discovery, and security information and event management.

But most importantly, the integrated ecosystem has simplified the work needed for PicnicHealth to create a secure environment for employees to use when working with sensitive medical records while still providing all the tools they need. For example, abstractors not only use Google Workspace but also have Chromebooks because they are easy to manage and secure.

Altogether, Google Cloud helps form a technology stack that has enabled the startup to build a massive labeled dataset containing over 100 million labeled medical data concepts. In turn, it accelerates PicnicHealth’s ability to generate highly-performant AI models and feed other ML pipelines, which has been vital for processing and reviewing data at scale.

To learn more about how Google Workspace and Google Cloud help startups like PicnicHealth accelerate their journey, visit our startups solutions pages for Google Workspace and Google Cloud.

Editor’s Note: In a previous blog, we discussed how home improvement retailer Lowe’s was able to increase the number of releases it supports by adopting Google’s Site Reliability Engineering (SRE) framework on Google Cloud. Lowe’s went from one release every two weeks to 20+ releases daily, helping meet its customer needs faster and more effectively. Today, the Lowe’s SRE team shares how they used SRE principles to decrease their mean-time-to-recovery (MTTR) by over 80 percent.

The stakes of managing Lowes.com have never been higher, and that means spotting, troubleshooting and recovering from incidents as quickly as possible, so that customers can continue to do business on our site. 

To do that, it’s crucial to have solid incident engineering practices in place. Resolving an incident means mitigating the impact and/or restoring the service to its previous condition. The average time it takes to do this is called mean time to recovery (MTTR). Tracking this metric helps us stay on top of the overall reliability of our systems at Lowe’s, while simultaneously improving the speed with which we recover. Our goal is to keep the MTTR metric as low as possible, so that failures don’t negatively impact our business. Here are the four areas we addressed to drive holistic improvement in our MTTR.

Lowe’s incident reporting process

To reduce MTTR, we created a seamless incident reporting process following SRE principles. Our incident reporting process is a workflow that starts at the time an incident occurs, and ends with an SRE captain who closes the action items after a postmortem report. With this approach, we are able to limit the number of critical incidents. The reporting process involves three core components: monitoring, alerting, and blameless postmortems.

Monitoring and alerting

Having proper monitoring and alerting in place is crucial when it comes to incident management. Monitoring and alerting tools let you detect issues as soon as they occur, and notify the right person in the shortest possible time to take action. From a measurement standpoint, we track this as our mean time to acknowledge (MTTA). This is the average time it takes from when an alert is triggered, to when work on the issue begins.

At the time of an incident, our monitoring and alerting tools notify the on-call SRE first responder via PagerDuty in the form of a phone call, text message and email. Our SRE software engineering team has done a lot of automation to enable various Service Level Indicator (SLI) alerts and Service Level Agreement (SLA) notifications. The on-call SRE then initiates a triage call with our service/domain stakeholders to resolve the incident. As a result, we reduced our MTTA from 30 minutes in 2019, to one minute – a 97 percent decrease. 

Blameless postmortems: learning from incidents

A postmortem is a written record of an incident, its impact, the actions taken to resolve it, the root cause and the follow-up actions to prevent the incident from recurring (see example here). A blameless postmortem builds on that and is a core part of an SRE culture, and our culture at Lowe’s. We ensure that individuals are not singled out, and the outcome for all postmortems are directed toward learnings and process improvement.

For us, the postmortem process is the biggest part of our incident workflow. When an SRE creates a new postmortem report, the first step is to conduct a postmortem session with domain stakeholders to review the report. The postmortem then goes into the review stage and gets reviewed by more stakeholders in our weekly postmortem meeting. In the final stage of this process, the SRE captain will close the report once everyone in the weekly meeting agrees that the report is complete.

To conduct a successful postmortem, it is critical to keep the focus on identifying gaps and issues with the system and operations processes, rather than an individual, and generate concrete actions to address the problems we’ve identified. To ensure this, we follow a couple of best practices:

1. We start by gathering the facts from the person who identified the problem, and each SLI owner has to identify a gap or the next SLI upstream owner who created the impact for them.
2. Every SLI owner is provided full opportunity to present their case, and identifying the issue is done as a community exercise. 
3. Once action items and process changes are identified, an owner is nominated to complete the actions, or they will volunteer. 
4. For easy reference, we publish and store postmortems in our incident knowledge base. This process helps SREs continuously improve as future incidents arise. 

Continuous Improvement 

Encouraging a culture of honest, transparent and direct feedback that you need for blameless postmortems is often an iterative process that needs sponsorship from executives, empowering incident captains to lead the entirety of the discussion and outcomes. Running successful postmortems, and completing action items from them, needs to be recognized and accounted for in SRE performance objective assessment. As shared in Google’s SRE book, the best practice is to ensure that writing effective postmortems is a rewarded and celebrated practice, with leadership’s acknowledgement and participation. This is possibly the hardest part to accomplish in an effective postmortem during a cultural transformation unless you have full buy-in from leadership.

However, it’s all well worth it. This process is a key part of how we were able to improve our MTTR over time—from two hours in 2019 to just 17 minutes! 

Our SRE incident reporting process has also transformed how our company solves issues. By streamlining this workflow from alerting, to solving an issue, to blameless postmortems, we have reduced our MTTR by 82 percent and our MTTA by 97 percent. Most importantly, our team is learning from every incident and becoming better engineers as a result. Visit the SRE Google Cloud website to learn more about implementing SRE best practices in the cloud.

Acknowledgement

Special thanks to Rahul Mohan Kola Kandy, Vivek Balivada, and the Digital SRE team at Lowe’s for contributing to this blog post.

Climate scientists face a deluge of environmental data to analyze and interpret from real-time sensors and satellites across the globe. The stakes are as high as our planet’s long-term future, but rapidly changing conditions are already impacting communities through extreme events like floods and wildfires as well as management of everyday resources. In this context, detailed environmental maps are key sources for urgent global issues like food security, water quality, and vegetation levels.

Scientists, researchers, and developers rely on state-of-the-art cloud computing tools like Google Earth Engine (EE) to detect changes, map trends, and quantify differences on the Earth’s surface. EE leverages the compute power of Google Cloud to combine a multi-petabyte catalog of satellite imagery and geospatial datasets with planetary-scale analysis capabilities.

To find out more about how climate scientists address these complex research challenges, I spoke to Dr. Richard Fernandes, Research Scientist at Natural Resources Canada (NRCan). Dr. Fernandes is also a 2022-23 member of the Google Research Innovators Program, which provides technical and professional support to a global cohort of leading researchers. I asked him to describe how cloud computing and machine learning (ML) tools can support climate resilience research and drive awareness about climate sustainability.

Dr. Fernandes, can you give me an overview of your research in climate sciences?

My research focuses on mapping the status and trends of vegetation over Canada. Every month we generate maps of vegetation parameters, such as canopy cover at 20m resolution, to support environmental monitoring and assessment. These maps contribute to global datasets that are used to reduce uncertainty in weather and climate forecasts.

Canada has approximately 10 million square kilometers of land and the annual data volume of these maps is equivalent to streaming HD movies for over 750 hours non-stop. And that’s only the tip of the iceberg: the volume of input data we need to generate those maps is typically 100 times more. Unlike movie streaming services, we have to independently process each input pixel to locate it accurately, screen for clouds (and even the shadows of clouds), and then transform it into a vegetation parameter value like canopy cover using ML algorithms.

The volume of high-resolution data and the amount of compute we need are challenging and ever-increasing. Rather than dedicating servers 24/7 for constant monitoring, we rely on cloud computing and ML. Cloud computing allows us to manage all this data in a useful and accessible way. We have also been able to successfully use the Google Artificial Intelligence (AI) Platform to calibrate new ML models for third-party datasets.

How did you start working with Google Cloud?

I started using EE together with open source APIs for integration with Google Drive about four years ago. With the pandemic my research group has transitioned to using EE, Google Cloud, and Google Drive for both our Canada-wide mapping as well as for R&D activities. In January we developed and released the LEAF toolbox, which builds on EE satellite data to create customizable maps of foliage density in near real-time. We do all of the pixel processing in EE and leverage the ability to integrate our own functions using their APIs. We combine EE with Google Cloud to handle and manage output datasets. Fortunately, EE has most of the input data already at hand so we don’t need to deal with that 100x larger volume.

What impact do you expect LEAF will have now–and down the road?

Both the Canadian federal government and provinces already use our data products as inputs to permafrost, crop status, and water resource models. Agriculture Canada had been using a conventional Geographic Information System (GIS) and approached us to ask how we manage our data. They want to use LEAF to assess how crops are progressing, which impacts local economies and the global food supply. We’re in a pilot with them to apply EE to their crop condition assessments.

Also exciting is that the Province of Alberta is integrating the LEAF toolbox within a system for monitoring the reclamation status of oil and gas wells and mines. They want to know how to rehabilitate sites that were developed for mineral and gas deposits. This is another great use case but many others are possible. The technology itself is cool, but that’s not even the point. ML algorithms are constantly evolving and improving. New ML algorithms use active learning that detects mistakes and makes updates on the fly. The technology will be different in five years — or five months! — so our priority is making these tools accessible and useful now.

Why is state-of-the-art technology for climate research so important?

Scientific research must be validated, transparent, and rigorous to drive the best solutions to our complex and changing ecosystem. Climate action needs greater public awareness, which requires more knowledge, which demands the best data. By democratizing information and decision-making, we can create an ecosystem of openness and public-private partnerships. We can lower the barrier to entry for advanced research and help scientists validate and reproduce their results. Also, scientists don’t want to have to become software engineers for these custom highly specific solutions. They want user-friendly tools that let them focus on their analyses and share their insights with the world. That’s one of the major appeals of the Google Research Innovators Program for me: I want to share LEAF with colleagues and collaborate with other researchers who are using new tools in new ways.

Do you have any parting words about your mission?

We’re so fortunate to have Canadian taxpayers and the Government of Canada funding our work. We have worked with many scientists over the past two decades to develop and validate the algorithms we use, especially Fred Baret and Marie Weiss at INRA France, who have championed the idea of free and open access to algorithms and knowledge, and data.

I really believe in democratic access to information. I like the fact that the terms of service of Google products allows us to offer not only maps but the actual processing system in a free and open manner to everyone. I also like that EE provides a simple-to-use user interface that works on mobile devices. We designed LEAF not just for experts, but for individuals. My mom was able to make maps of a nearby park in real time on her tablet. It is my hope that expanding access to critical environmental information will increase our collective awareness of how our actions impact both near and far places — and make us active in the cause of a sustainable future.

That’s an inspiring note to end on. Thank you for your time!

Thanks for having me!

Click here to learn more about Google’s commitment to renewable energy. Or try the LEAF toolbox for yourself!