About Tencent Africa

Tencent Africa is dedicated to making every day easier for customers and brands. Opening its doors in 2013 as WeChat Africa, the company is responsible for WeChat operations on the continent.

Industries: Technology

Location: South Africa

About Siatik Systems

Founded in 2008, South Africa-based Siatik Systems is a leading Google Cloud Platform solutions provider with a focus on cloud integration services, information technology consulting services, and bespoke software engineering solutions.

Industries:

Location:

As one of Africa’s leading technology companies, Tencent Africa is responsible for WeChat operations on the continent. WeChat Africa has successfully launched a number of features including the WeChat Wallet, a mobile payment service for smartphones, which enables seamless and secure transactions for friends to send money to one another, cash money out at Standard Bank ATMs, use at accredited retailers, and make purchases from any SnapScan merchant in South Africa. Under the new name, Tencent Africa recently launched two new mobile products, JOOX, a mobile music streaming application, and VOOV, a social live streaming application.

When China’s Tencent brought its hugely popular instant messaging service WeChat to South Africa, it faced a near saturated market. To gain traction, Tencent Africa had to showcase WeChat’s versatility as a platform that goes far beyond just instant messaging. Engaging with local financial partners helped WeChat launch a WeChat Wallet to facilitate electronic payments, while providing exclusive content from South African cultural figures helped bring the platform extra publicity.

As WeChat started to grow its customer base, its on-premises infrastructure was stretched beyond its limits and the company looked for a cloud-based solution. After evaluating all its options, Tencent Africa found Google Cloud Platform best suited its needs.

“We’re a growing business and constantly upgrading our infrastructure. Getting new servers was becoming quite expensive so we wanted a cloud-based solution to reduce costs,” says Christoff Albertyn, Head of Technology at Tencent Africa. “When we did some latency testing, Google came out on top, so it made sense to go with them. Everybody loves Google, it’s a great brand. Just knowing a company like Google is behind it gives you peace of mind.”

Building a platform to scale

Instant messaging services need a stable infrastructure, the ability to scale at speed, and lightning fast performance. When Tencent Africa built an Official Accounts API to run a number of services on WeChat, to help companies engage with users on the platform, latency was its main concern. The Official Accounts services had to deliver replies over WeChat in under five seconds or the connection would drop.

Between the API on-premises servers in South Africa and WeChat’s main servers in China, Tencent Africa’s services regularly hit the limits of the permissible response time. In addition, by 2015, WeChat was starting to garner more customers across South Africa and Tencent Africa had to provision more servers. The on-premises servers were simply being asked to do too much, and expansion was proving expensive, so the company looked for a cloud-based solution.

Before committing to anything, Tencent Services did a thorough latency test of several cloud-based solutions. Google Cloud Platform came top. Tencent also engaged the help of Siatik, a Google Partner, to gauge what solutions best fit the company’s needs and help implement them. Siatik customers span both the public and private sectors and its consultants are regularly engaged in the health care, education, financial and technology industries. In order to provide leading technology solutions, it partners with industry leaders, such as Google, to provide best-of-breed infrastructure and cloud platform solutions to its customers.

“Siatik was a great partner to work with,” says Christoff. “Any questions I had, I could just pick up the phone and whatever it was they’d help me out. They were also able to get some of the team from Google to come to our offices and see how we worked, which was really good.”

With a simple, clean interface in Google Compute Engine, the company was able to spin up virtual instances very quickly and begin migrating its stack to Google. When it became more comfortable with Google Cloud Platform features, Tencent used Google Cloud SQL and Google BigQuery to manage its data streams, and Google Cloud Storage to store them safely.

From day one, it was very smooth. The interface was amazing—Google must have spent a long time perfecting it because it was so easy to use says Christoff. “I set up the first virtual machine in no time without any struggles.”

In addition to powerful hardware from Google Compute Engine, the fast, global network from Google helped to speed up performance. With the infrastructure ported over to Google, Tencent also brought in G Suite as an office productivity solution, which eased the pressure on its servers even more. Throughout the process, Siatik were on hand to advise and provide technical guidance whenever any issues came up.

What I really like is how quickly you can spin servers up. There’s no messing about and the interface is really easy to use, says Christoff. “You can have a virtual machine running in just a few minutes. It’s very impressive.”

Cutting costs, raising expectations

Google Cloud Platform helped Tencent Africa to slash its latency. Before the migration, pings to the Chinese servers and back would regularly take 180 milliseconds. Afterwards, this fell to around 35 milliseconds, helping to ensure smooth service to Tencent’s South African customers. Google offers innovative pricing, such as per-minute billing and sustained use discounts, which helped significantly cut server costs compared to an equivalent on-premises solution.

As its customer base grew, and Tencent Africa began to release new products, the ease of use of Google Compute Engine meant that spinning up new servers took minutes instead of hours and allowed the company to scale up and down at speed. Exploring products such as Kubernetes for rapid deployment or TensorFlow for machine learning, Tencent Africa continues to evolve and experiment with new products to keep WeChat a growing success in South Africa.

“Initially, we just used per-minute billing, but when I saw how much work we were doing with Google BigQuery and Google Cloud SQL, we started to generate sustained-use discounts, which helped save us a lot of money,” says Christoff. “The other great thing is, Google Cloud Platform monitors how you’re using it and tells you what else you can do to reduce costs.”

Organizations are pressed with the need to engage, retain, and upskill employees with many positions staying fully or partially remote as the pandemic continues. The Center for Digital Government (CDG) reports that 74% of state and local governments believe they will continue hybrid operations for the long-term. This transition is the latest in a long list of reasons agencies are looking for solutions to help train their workforce digitally. Some have found Google Classroom to be the perfect solution.

The State of Iowa knew about the advantages of digital classrooms even before the pandemic. They were already successfully using Google Classroom for in-person training, making the transition to digital seamless when the state closed down. Jessica Van Heuveln, a Google support specialist for the State of Iowa CIO’s Office, says the switch to digital doesn’t deprive workers of hands-on training. “You can use Google Classroom for many different environments, so if you’re doing self-taught, in-person or virtual training, it’s going to work with all those approaches.”

The logistics of a digital classroom

Iowa uses Google Classroom to onboard and upskill employees as well as train volunteers. When the pandemic shifted the state’s workforce to 70% working from home, they needed a  platform that could not only handle a vast range of topics and teaching styles to support remote workers but also scale to meet their demand. Van Heuveln notes that Google Classroom was particularly helpful for handling compliance training, especially with its integration with Google Meet. These trainings often require an entire sector of the workforce to attend a single class with more than 100 attendees, according to Van Heuveln. The logistics of these larger sessions cuts down on the number of total sessions the state needs to host. Iowa also found that employees were less apprehensive about learning when they could do it from the comfort of their own homes.

Iowa follows a train-the-trainer model to train employees expected to instruct using Google Classroom. Trainers working in departments across the state learn to use Google Classroom to create and host virtual learning experiences specific to their own departments. 

Collaborative classrooms made simple

Google Classroom meets many needs for Iowa, such as setting up training, managing attendance, reviewing uploaded coursework, and communicating with attendees. Both course attendees and trainers have everything they need in one place, and robust engagement tools help attendees stay focused while giving trainers the feedback they need from their lessons. Built-in survey options let trainers gauge attendee knowledge, and the chat function ensures no one ever misses a question. The reporting dashboard gives trainers access to analytics to track engagement and attendance, providing insights that can improve future training sessions.

Google Classroom also makes it easier to ensure attendees stay on board throughout the training program. Coursework can be uploaded directly into Google Classroom, and training sessions can be recorded and archived. This feature allows employees to work at their own pace or lets them catch up when a scheduling conflict means they might miss a session.

The enterprise version of Google Classroom Iowa uses interfaces with other Google Workspace tools such as Google Meet, Google Calendar, Google Drive, and Gmail, which streamlines training and leads to more collaboration between employees and trainers. It can also interface with applications from other providers, further boosting its utility as an effective virtual classroom. Google Drive stores training materials securely. Lastly, Google Classroom enterprise comes with 24/7 support to make sure agencies always have everything they need for success.

Helping employees be better prepared to serve

Google Classroom can be a powerful part of any training program, whether it be digital or in-person. Building virtual connections empowers organizations to deliver better experiences. Google Classroom is helping the State of Iowa accomplish this for their constituents. Designed to scale and be accessible from anywhere, the platform helps public employees better serve the public. Virtual training strategies are essential for agencies that are looking to grow their workforce and build employee skill levels. The most important thing, however, is that training programs enable employees to further your agency’s mission. To learn more about Google Classroom and see more solutions designed with government in mind, check out the Google Cloud for government page.

Patients about to undergo a healthcare procedure understandably want the best medical professionals they can get. But how can they know which doctors have had the most experience and the best outcomes with that particular procedure? How can they make an informed decision about which doctor to select when the information they have is limited to the doctor’s practice area and subjective reviews from other patients?

MD Insider is working to solve that problem using machine learning (ML) to objectively analyze doctor performance. By analyzing data from thousands of institutions and millions of doctor-patient interactions and medical events, MD Insider identifies physician performance insights based on their experience and outcomes. Insights are then integrated into a triage engine, that enables consumers to search for and schedule appointments with providers who meet their clinical criteria and convenience preferences, such as insurances accepted, office hours, locations, and language.

MD Insider also offers robust data APIs to help health systems, health plans, and employers reduce costs and improve quality of care. Payers use the APIs to curate high-quality provider networks and manage provider directories. Examples of MD Insider’s data APIs include Provider Experience and Share of Practice metrics, Provider Quality and Outcomes, Network Modeling, Expert Clinical Search Taxonomy, Find a Provider, Acute-Care Hospital Quality, and Provider and Facility Metadata.

“We use Google Cloud Platform the way the cloud was supposed to be used. By comparison, other cloud providers feel like you’re renting somebody else’s data center.”

—Ed Holsinger, Lead Data Engineer and Head of Data Science, MD Insider

MD Insider is continuously ingesting the latest performance data about physicians and analyzing billions of rows of data. Requiring constant scalability, the company was born in the cloud; however, it had difficulty configuring server instances for the optimal balance of memory and CPU, and its Hadoop cluster had to be kept running 24/7. Network performance was often slow for no apparent reason. As a result, failure rates from node timeouts increased, and costs grew along with the data. MD Insider had to estimate its usage and pay up front, and received little financial benefit from sustained use commitments.

Knowing that data would continue to grow, MD Insider decided to move its data services — the most demanding and complex portion of its infrastructure — to Google Cloud Platform (GCP), and took advantage of GCP managed services for container management and big data analytics.

“One of the reasons we decided to move to Google Cloud Platform is because it feels like a unified, well-designed cloud architecture and pricing model,” says Ed Holsinger, Lead Data Engineer and Head of Data Science at MD Insider. “We use Google Cloud Platform the way the cloud was supposed to be used. By comparison, other cloud providers feel like you’re renting somebody else’s data center.”

“Moving to Google Cloud Platform and using Kubernetes Engine gave us 5x the processing power for analyzing physician performance at 75% less cost. Our data scientists have more power than ever before to generate insights for our customers.”

—Ed Holsinger, Lead Data Engineer and Head of Data Science, MD Insider

5x the performance, 75% less cost

MD Insider now uses Kubernetes Engine to automate container management and deploy clusters in minutes with just a few clicks. When hundreds of machines are required to analyze a large dataset, automation in Kubernetes Engine deploys ML models as containers, each of which manages the full lifecycle of its task, including scaling up resources, deploying results, and scaling back down when the task is finished. It’s easy for MD Insider to specify exactly how much CPU and memory each container needs, helping maximize performance while reducing costs.

“Moving to Google Cloud Platform and Kubernetes Engine gave us 5x the processing power for analyzing physician performance at 75% less cost,” says Ed. “Our data scientists have more power than ever before to generate insights for our customers. Data scoring jobs that used to take three business days now take four hours.”

Adds Galen Meurer, Senior Software Engineer at MD Insider: “Even if all Google Cloud Platform had to offer was Kubernetes Engine, I would still want to use it. Previously we spent up to 30% of our time managing our container infrastructure, which we can now use for product development.”

A foundation for data science

MD Insider was happy to find that GCP offers a wide variety of managed services. For example, the company is supplementing its Kubernetes Engine clusters with BigQuery for its big data masters and selection jobs, enabling scientists to analyze new and different types of data as well as analyze larger datasets in less time. MD Insider also uses Cloud Storage for big data staging and Cloud Dataproc to run managed Apache Spark clusters for data processing.

“Google Cloud Platform gives us an incredibly powerful cloud architecture for data engineering and data science,” says Eric Wilson, CEO of MD Insider. “That gives our scientists independence, they can do what they need to do without waiting and with no contention between them.”

A developer-friendly platform

Migrating its data services was such a success that MD Insider decided to move the rest of its infrastructure to GCP, including the front end for its web application. Since the migration, MD Insider has experienced no unplanned downtime on GCP, allowing it to easily meet the 99.5% uptime SLA it promises to customers. It’s also taking advantage of Build Triggers in Kubernetes Engine to automate container builds and reduce build times by more than 40%. Production code can be updated in seconds, with no impact to end users other than making new features available.

“GCP has simplified our workflow in so many ways, from intelligent load balancing to content delivery and automating builds,” says Matthew Frey, Software Engineer. “Everything on GCP is cohesive and developer friendly, with a superior UI and better network performance than other cloud providers.”

Ryan Beaini, Senior Software Engineer at MD Insider, agrees: “Since we moved to GCP, our developers are definitely happier. The pain and the headaches we experienced because of the limitations of our previous toolset all went away.”

“We’re a small company, but what we’re doing is incredibly important. We’re helping people make decisions about healthcare providers that could impact their lives and even be life-saving. Google Cloud Platform is helping us make our mission bigger, better, and brighter.”

—Eric Wilson, CEO, MD Insider

Securing billions of rows of clinical and non-clinical healthcare data

As a healthcare technology company, MD Insider processes billions of rows of clinical and non-clinical healthcare data. To control user access to GCP, it uses Identity & Access Management (IAM) along with Yubico YubiKeys for hardware-based two-factor authentication when logging into Google Workspace. MD Insider takes comfort that GCP encrypts data at rest by default, and encrypts and authenticates data in transit when data moves outside physical boundaries not controlled by Google or on behalf of Google.

“On GCP, everything that we need to be encrypted for compliance purposes is encrypted, which is fantastic,” says Eric. “When I tell our potential clients and partners about the resources that Google has dedicated to security, it gives them the confidence that their data will be protected.”

Transforming how teams work

As a growing company, MD Insider must collaborate seamlessly between offices in California, Colorado, and Illinois. It relies on Google Workspace for communication and productivity, using Docs, Sheets, and Slides to drive the business. Employee and team files are stored in Drive, and meetings are conducted via Google Meet with Chromebox for Meetings videoconferencing hardware kits. Google Workspace also helps MD Insider maintain information security by authenticating email domains with digital signatures in Gmail and scanning outgoing email using Gmail Data Loss Prevention (DLP).

“I use Google Workspace every day, and everyone else here does too,” says Eric. “Team Drives are a big time saver for us. We’ve let our previous office software expire, because there’s no need to pay for those licenses anymore.”

Promoting healthcare transparency

With GCP helping MD Insider increase velocity and momentum, the company is making exciting progress. For example, it has entered into a strategic partnership with Zelis Healthcare, which will use MD Insider’s API to provide insights for a next-gen analytics platform that will give health plans unprecedented transparency around physician performance.

“We’re a small company, but what we’re doing is incredibly important,” says Eric. “We’re helping people make decisions about healthcare providers that could impact their lives and even be life-saving. Google Cloud Platform is helping us make our mission bigger, better, and brighter.”

*Google Workspace was formerly known as G Suite prior to Oct. 6, 2020.

I’m excited to share our first Cloud CISO Perspectives post of 2022. It’s already shaping up to be an eventful year for our industry and we’re only in month one. There’s a lot to recap in this post, including the U.S. government’s recent efforts to address critical security issues, like open source software security and zero trust architectures. We’ve also released new resources from our Google Cybersecurity Action Team like the Cloud Security Megatrends and the Boards of Directors whitepaper on cloud risk governance. 

Cloud Security Megatrends 

We’re often asked if the cloud is more secure than on-prem (and why) so we shared our answer in a recent blog post. At Google Cloud, security by design is our priority. We’ve long adopted zero-trust principles for our baseline security architectures and built a global network that relies on defense in depth layers to protect against configuration errors and attacks. But security is always evolving and that is why we also take advantage of the following megatrends:

1. Economy of scale: Decreasing the marginal cost of security raises the baseline level of security. 
2. Shared fate: A flywheel of increasing trust drives more transition to the cloud, which compels even higher security and even more skin-in-the-game from the cloud provider.
3. Healthy competition: The race by deep-pocketed cloud providers to create and implement leading security technologies is the tip of the spear of innovation. 
4. Cloud as the digital immune system: Every security update the cloud gives the customer is informed by some threat, vulnerability, or new attack technique often identified by someone else’s experience. Enterprise IT leaders use this accelerating feedback loop to get better protection.
5. Software-defined infrastructure: Cloud is software defined, so it can be dynamically configured without customers having to manage hardware placement or cope with administrative toil. From a security standpoint, that means specifying security policies as code, and continuously monitoring their effectiveness.
6. Increasing deployment velocity: Because of cloud’s vast scale, providers have had to automate software deployments and updates, usually with automated continuous integration/continuous deployment (CI/CD) systems. That same automation delivers security enhancements, resulting in more frequent security updates.
7. Simplicity: Cloud becomes an abstraction-generating machine for identifying, creating and deploying simpler default modes of operating securely and autonomically. 
8. Sovereignty meets sustainability: The cloud’s global scale and ability to operate in localized and distributed ways creates three pillars of sovereignty. This global scale can also be leveraged to improve energy efficiency.

If you’re an IT decision maker, pay attention to these megatrends that will continue to drive and reinforce cloud security and will outpace the security of on-prem infrastructure well into the future. 

U.S. Federal government cybersecurity momentum 

• Open source software security: Earlier this month, Google participated in the White House Summit on open source software security. The meeting came at a critical time for the industry following December’s Log4j vulnerabilities and was both a recognition of the challenge and an important first step towards addressing it. The open source software ecosystem is not homogenous, despite the fact that the industry often thinks of or treats it this way. Some of it, like Linux, is highly curated, while other critical software is supported through diffuse communities including technology companies and other stakeholders. There is also a long tail of many other critical projects driven by a dedicated community of maintainers around the world, including Googlers. In light of this reality, we welcomed the chance to share our recommendations to advance the future of open source software security. Some work we’ve done includes founding the Open Source Security Foundation, which has been instrumental already in making security improvements. We’ve also helped drive a number of key security initiatives within the open source community including security scorecards, the SLSA framework to improve the security and integrity of open source packages, and Secure Open Source Rewards to financially incentivize improvements to critical open source security projects.  
• OMB’s Federal zero trust strategy: The publication of the Office of Management and Budget’s zero trust architecture strategy marks an important step for the U.S. federal government’s efforts to modernize under Executive Order 14028. Google Cloud supports this approach, which recognizes the immense security benefits offered by modern computing architectures. For the past decade, Google has successfully applied zero trust principles through our BeyondCorp and BeyondProd frameworks for providing end-user access and securing our cloud workloads. And we’ve brought these best practices from our own journey to global governments and businesses of any size through solutions like BeyondCorp Enterprise and capabilities like Binary Authorization and Anthos Service Mesh, which are embedded in Anthos, our managed application platform. For Federal agencies embarking on this zero trust journey, the Google Cybersecurity Action Team will offer our expertise by conducting Zero Trust Foundations strategy workshops, which can help organizations in the public and private sectors develop actionable and achievable strategies and plans for zero trust implementation. 

Google Cybersecurity Action Team Highlights 

Here are the latest updates, products, services and resources across our security teams this month: 

Security

• Democratizing security operations: We recently announced that Siemplify, a leading security orchestration, automation and response (SOAR) provider, is joining Google Cloud to help companies better manage their threat response. Providing a proven SOAR capability with Chronicle’s approach to security analytics is an important step forward in our vision to advance invisible security and democratize security operations for every organization.
• Security by design: The Highmark Health security team is using “secure-by-design” techniques to address the security, privacy, and compliance aspects of its Living Health solution with Google Cloud’s Professional Services Organization (PSO). Google has long advocated for and followed security by design principles, which is why we’re continuously building enhanced security, controls, resiliency and more into our cloud products and services. 
• Secure collaboration for hybrid work environments: The Google Workspace team shared its recommendations for businesses as they prepare for the future of work,  where the hybrid/flexible work model is becoming standard practice and a new approach to security is essential.
• Anthos Policy Controller CIS Benchmark enforcement: A big part of our shared fate philosophy is to build secure products and not just security products. A recent example of this in action is embedding CIS benchmark policy conformance in the Anthos Policy Controller. We believe the more we embed approaches like this into our products, the more application and infrastructure teams can intrinsically embed security at the start and reduce toil for the security team.
• DevOps for technology-driven organizations and startups: A key success factor for many security programs is the partnership and integration with development teams, and there are some great resources and lessons in our DORA research.
• Security by design with Chrome OS: ABN AMRO’s Asia-Pacific region team recently shared how they are using Chrome OS and CloudReady to work securely in the cloud, reduce total cost of ownership, and add flexibility for employees. This is a great example of secure by design principles in the use of Chromium.

Risk & Compliance

• Boards of Directors summary guide to cloud risk governance: The latest whitepaper from the Google Cybersecurity Action Team outlines how boards of directors can prioritize safe, secure, and compliant adoption processes for cloud technologies within their organizations.  
• TruSight Risk Assessment of Google Cloud: TruSight recently released a comprehensive
  risk assessment report on Google Cloud. Our Enterprise Trust team collaborated on this robust assessment of Google Cloud services to validate the design and implementation of controls. TruSight’s risk assessment of our security controls will help customers accelerate and complete their risk management due diligence.
• Data governance: Check out this new blog series on data governance where our teams explain the role of data governance, its importance, and the necessary processes to run an effective data governance program. Implementing data governance will help maximize value derived from business data, build user trust, and ensure compliance with required security measures.

Controls and Products

• Encrypting Data Fusion: To help meet the security, privacy and compliance requirements of customers in regulated industries like finance or public sector, we announced the general availability of Customer Managed Encryption Keys (CMEK) integration for Cloud Data Fusion, which enables encryption of both user data and metadata at rest with a key that customers can control through our Cloud Key Management Service (KMS). 

Don’t forget to sign-up for our newsletter if you’d like to have our Cloud CISO Perspectives post delivered every month to your inbox. We’ll be back next month with more updates and security-related news.

Editor’s note: Today we are hearing from Jono MacDougall , Principal Software Engineer at Ravelin. Ravelin delivers market-leading online fraud detection and payment acceptance solutions for online retailers. To help us meet the scaling, throughput, and latency demands of our growing roster of large-scale clients, we migrated to Google Cloud and its suite of managed services, including Cloud Bigtable, the scalable NoSQL database for large workloads.

As a fraud detection company for online retailers, each new client brings new data that must be kept in a secure manner and new financial transactions to analyze. This means our data infrastructure must be highly scalable and constantly maintain low latency. Our goal is to bring these new organizations on quickly without interrupting their business. We help our clients with checkout flows, so we need latencies that won’t interrupt that process—a critical concern in the booming online retail sector. 

We like Cloud Bigtable because it can quickly and securely ingest and process a high volume of data. Our software accesses data in Bigtable every time it makes a fraud decision. When a client’s customer places an order, we need to process their full history and as much data as possible about that customer in order to detect fraud, all while keeping their data secure. Bigtable excels at accessing and processing that data in a short time window. With a customer key, we can quickly access data, bring it into our feature extraction process, and generate features for our models and rules. The data stays encrypted at rest in Bigtable, which keeps us and our customers safe.

Bigtable also lets us present customer profiles in our dashboard to our client, so that if we make a fraud decision, our clients can confirm the fraud using the same data source we use.

We have configured our bigtable clusters to only be accessible within our private network and have restricted our pods access to it using targeted service accounts. This way the majority of our code does not have access to bigtable and only the bits that do the reading and writing have those privileges.

We also use Bigtable for debugging, logging, and tracing, because we have spare capacity and it’s a fast, convenient location. 

We conduct load testings against Bigtable.  We started at a low rate of ~10 Bigtable requests per second and we peaked at ~167000 mixed read and write requests per second  at absolute peak. The only intervention that was done to achieve this was pressing a single button to increase the number of nodes in the database. No other changes were made.

In terms of real traffic to our production system, we have seen ~22,000 req/s (combined read/write) on Bigtable in our live environment as a peak within the last 6 weeks.

Migrating seamlessly to Google Cloud 

Like many startups, we started with Postgres, since it was easy and it was what we knew, but we quickly realized that scaling would be a challenge, and we didn’t want to manage enormous Postgres instances. We looked for a kind of key value store, because we weren’t doing crazy JOINS or complex WHERE clauses. We wanted to provide a customer ID and get everything we knew about it, and that’s where key value really shines.  

I used Cassandra at a previous company, but we had to hire several people just for that chore. At Ravelin we wanted to move to managed services and save ourselves that headache. We were already heavy users and fans of BigQuery, Google Cloud’s serverless, scalable data warehouse, and we also wanted to start using Kubernetes. This was five years ago, and though quite a few providers offer Kubernetes services now, we still see Google Cloud at the top of that stack with Google Kubernetes Engine (GKE). We also like Bigtable’s versioning capability that helped with a use case involving upserts. All of these features helped us choose Bigtable.

Migrations can be intimidating, especially in retail where downtime isn’t an option. We were migrating not just from Postgres to Bigtable, but also from AWS to Google Cloud. To prepare, we ran in AWS like always, but at the same time we set up a queue at our API level to mirror every request over to Google Cloud. We looked at those requests to see if any were failing, and confirmed if the results and response times were the same as in AWS. We did that for a month, fine tuning along the way. 

Then we took the big step and flipped a config flag and it was 100% over to Google Cloud. At the exact same time, we flipped the queue over to AWS so that we could still send traffic into our legacy environment. That way, if anything went wrong, we could fail back without missing data. We ran like that for about a month, and we never had to fail back. In the end, we pulled off a seamless, issue-free online migration to Google Cloud.

Flexing Bigtable’s features

For our database structure, we originally had everything spread across rows, and we’d use a hash of a customer ID as a prefix. Then we could scan each record of history, such as orders or transactions. But eventually we got customers that were too big, where the scanning wasn’t fast enough. So we switched and put all of the customer data into one row and the history into columns. Then each cell was a different record, order, payment method, or transaction. Now, we can quickly look up the one row and get all the necessary details of that customer. Some of our clients send us test customers who place an order, say, every minute, and that quickly becomes problematic if you want to pull out enormous amounts of data without any limits on your row size. The garbage collection feature makes it easy to clean up big customers.  

We also use Bigtable replication to increase reliability, atomicity, and consistency. We need strong consistency guarantees within the context of a single request to our API since we make multiple bigtable requests within that scope. So within a request we always hit the same replica of Bigtable and if we have a failure, we retry the whole request. That allows us to make use of the replica and some of the consistency guarantees, a nice little trade-off where we can choose where we want our consistency to live.https://www.youtube.com/embed/0-eH5u7rrQQ?enablejsapi=1&

We also use BigQuery with Bigtable for training on customer records or queries with complicated WHERE clauses. We put the data in Bigtable, and also asynchronously in BigQuery using streaming inserts, which allows our data scientists to query it in every way you can imagine, build models, and investigate patterns and not worry about query engine limitations. Since our Bigtable production cluster is completely separate, doing a query on BigQuery has no impact on our response times. When we were on Postgres many years ago, it was used for both analysis and real time traffic and it was not the optimal solution for us. We also use Elasticsearch for powering text searches for our dashboard.

If you’re using Bigtable, we recommend three features:

• Key visualizer. If we get latency or errors coming back from Bigtable, we look at the key visualizer first. We may have a hotkey or a wide row, and the visualizer will alert us and provide the exact key range where the key lives, or the row in question. Then we can go in and fix it at that level. It’s useful to know how your data is hitting Bigtable and if you’re using any anti-patterns or if your clients have changed their traffic pattern that exacerbated some issue.
• Garbage collection. We can prevent big row issues by putting size limits in place with the garbage collection policies.  
• Cell versioning. Bigtable has a 3d array, with rows, columns, and cells, which are all the different versions. You can make use of the versioning to get history of a particular value or to build a time series within one row. Getting a single row is very fast in Bigtable so as long as you can keep the data volume in check for that row, making use of cell versions is a very powerful and fast option. There are patterns in the docs that are quite useful and not immediately obvious. For example, one trick is to reverse your timestamps (MAXINT64 – now) so instead of the latest version, you can get the oldest version effectively reversing the cell version sorting if you need it.

Google Cloud and Bigtable help us meet the low-latency demands of the growing online retail sector, with speed and easy integration with other Google Cloud services like BigQuery. With their managed services, we freed up time to focus on innovations and meet the needs of bigger and bigger customers. 

Learn more about Ravelin and Bigtable, and check out our recent blog, How BIG is Cloud Bigtable?