Editor’s note: Today’s guest post comes from Indian social media platform ShareChat. Here’s the story of how they improved performance, app development, and analytics for serving regional content to millions of users using Google Cloud. 

How do you create a social network when your country has 22 major official languages and countless active regional dialects? At ShareChat, we serve more than 160 million monthly active users who share and view videos, images, GIFs, songs, and more in 15 different Indian languages. We also launched a short video platform in 2020, Moj, which already supports over 80 million monthly active users. 

Connecting with people in the language they understand

As mobile data and smartphones have become more affordable in India, we noticed a large new segment of people, many in rural areas, being welcomed onto the internet. However, many of them didn’t speak English, and when it comes to accessing content and information—language plays a significant role. Instead of joining other social media sites where English reigned supreme, new internet users chose to join language or dialect-specific Whatsapp groups where they felt more comfortable instead.

So, we set out to build a platform where people can share their opinions, document their lives, and make new friends, all in their native language. ShareChat simplifies content and people discovery by using a personalized content newsfeed to deliver language-specific content to the right audience.

Given the high-intensity data and high volume of content and traffic, we rely heavily on IT infrastructure. On top of that, a large number of our users rely on 2G networks to post, like, view, or follow each other. Our platform needs to deliver great experiences to people who are spread out across the country and different networks without any reduction in performance.

The right cloud partner to support future growth

ShareChat was born in the cloud—we already knew how to scale systems to serve a large customer base with our existing cloud provider. But like many companies, we struggled with over-provisioning compute and storage to accommodate unpredictable traffic and avoid running out of storage. With demand rising for local language content and an increase in online interactions in response to the COVID-19 crisis, we realized that we would need a more efficient way to scale dynamically and allocate resources as needed.

Google Cloud was a natural choice for us. We wanted to partner with a technology-first company that would make it easy (and cost-effective) to manage a strong technology portfolio that would allow us to build whatever we wanted. Google is at the forefront of technology innovation and provided everything we needed to build, run, and manage our applications (including creating an efficient DevOps pipeline to fix and release new features quickly). 

We had a few issues in mind at the start of discussions with the Google Cloud team, but over time, as we got information and support from them, we realized that these were the partners we wanted in our corner when it came time to tackle our most challenging problems. In the end, we decided to take our entire infrastructure to Google Cloud.

To support millions of users, we deploy and scale using Google Kubernetes Engine. While we analyze our data using a combination of managed data cloud services, such as Pub/Sub for data pipelines, BigQuery for analytics, Cloud Spanner for real-time app serving workloads, and Cloud Bigtable for less-indexed databases. We also rely on Cloud CDN to help us distribute high-quality and reliable content delivery at low latency to our users. 

We now use just half the total core consumption of our legacy environment to run ShareChat’s existing workloads.

Google Cloud delivers better outcomes at every level 

By moving to Google Cloud, we saw major benefits in several key areas: 

Zero-downtime migration for users

At the time of migration, we had over 70 terabytes of data, consisting of 220 tables—some of which were up to 14 terabytes with nearly 50 billion rows. Due to our data’s interdependencies, moving services over one at a time wasn’t an option for us. 

Even though we were migrating such large volumes of data, we didn’t want to impact any of our customers. Latency spikes for out-of-sync data might affect message delivery. For instance, if a message or notification was delayed, we didn’t want to risk a bad user experience causing someone to abandon ShareChat. 

To prepare for the move, we ran a proof-of-concept cluster for over four months to test database performance in a real-world scenario for handling more than a million queries per second. Using an open-source API gateway, we replicated our legacy data environment into Google Cloud for performance testing and capacity analysis. As soon as we were confident Google Cloud could handle the same traffic as our previous cloud environment, we were ready to execute.

Using wrappers, we were able to migrate without having to change anything in our existing application code. The entire migration of 60 million users to Google Cloud took five hours—without any data loss or downtime. Today, ShareChat has grown to 160 million users, and Google Cloud continues to give us the support we need.

Scaling globally to meet unexpected demand

We rely on real-time data to drive everything on ShareChat by tracking everything that goes on in our app—from messages and new groups to content people like or who they follow. Our users create more than a million posts per day, so it’s critical that our systems can process massive amounts of data efficiently. 

We chose to migrate to Spanner for its global consistency and secondary index. Unlike our legacy NoSQL database, we could scale without having to rethink existing tables or schema definitions and keep our data systems in sync across multiple locations. It’s also cost-effective for us—moving over 120 tables with 17 indexes into Cloud Spanner reduced our costs by 30%.

Spanner also replicates data seamlessly in multiple locations in real time, enabling us to retrieve documents if one region fails. For instance, when our traffic unexpectedly grew by 500% over just a few days, we were able to scale horizontally with zero lines of code change. We were also launching our Moj video app simultaneously, and we were able to move it to another region without a single issue. 

Simplifying development and deployment

On average, we experience about 80,000 requests per second (RPS) –nearly 7 billion RPS per day. That means daily push notifications sent out to the entire user base about daily trending topics can often result in a spike of 130,000 RPS in just a few seconds. 

Instead of over-provisioning, Google Kubernetes Engine (GKE) enables us to pre-scale for traffic spikes around scheduled events, such as holidays like Diwali, when millions of Indians send each other greetings. 

Migrating to GKE has also enabled us to adopt more agile ways of work, such as automating deployment and saving time with writing scripts. Even though we were already using container-based solutions, they lacked transparency and coverage across the entire deployment funnel. 

Kubernetes features, such as sidecar proxy, allows us to attach peripheral tasks like logging into the application without requiring us to make code changes. Kubernetes upgrades are managed by default, so we don’t have to worry about maintenance and stay focused on more valuable work. Clusters and nodes automatically upgrade to run the latest version, minimizing security risks and ensuring we always have access to the latest features.

Low latency and real-time ML predictions

Even though many of our users may be accessing ShareChat outside of metropolitan areas, it doesn’t mean they’re more patient if the app loads slowly or their messages are delayed. We strive to deliver a high-performance experience, regardless of where our users are. 

We use Cloud CDN to cache data in five Google Cloud Point of Presence (PoP) locations at the edge in India, allowing us to bring content as close as possible to people and speeding up load time. Since moving to Cloud CDN, our cache hit ratio has improved from 90% to 98.5%—meaning our cache can handle 98.5% of content requests. 

As we expand globally, we’d like to use machine learning to reach new people with content in different languages. We want to build new algorithms to process real-time datasets in regional languages and accurately predict what people want to see. Google Cloud gives us an infrastructure optimized to handle compute-intensive workloads that will be useful to us both now—and in the future.  

The confidence to build the best platform

Our current system now performs better than before we migrated, but we are continuously building new features on top of it. Google’s data cloud has provided us with an elegant ecosystem of services that allows us to build whatever we want, more easily and faster than ever before. 

Perhaps the biggest advantage of partnering with Google Cloud has been the connection we have with the engineers at Google. If we’re working to solve a specific problem statement and find a specific solution in a library or a piece of code, we have the ability to immediately connect with the team responsible for it. 

As a result, we have experienced a massive boost in our confidence. We know that we can build a really good system because we not only have a good process in place to solve problems—we have the right support behind us.

Vinayak Bhavnani, Co-Founder and CTO of India-based bus transport technology company Chalo, shares how Google Maps Platform is used to improve visibility for commuters and bus operators across India by visualizing geospatial data.

Effective public transport networks contribute to the local economy and help make cities safe, pleasant, and sustainable. In India, buses make up around 90% of the public transport offering, but when you talk to the people who ride them every day, you find that there’s a lot of room for improvement. Heavy traffic means there are rarely any fixed schedules and it’s impossible to know exactly when your bus is coming. We’ve found that people tend to wait at a bus stop for up to 30 minutes a day, which creates a lot of frustration and wasted time.

When we founded Chalo, our aim was to make the daily city commute a more positive experience. Reliability is synonymous with visibility: when you know exactly when the bus is coming, you can plan your day better. If you’re in your office, for example, and see the next bus is in 10 minutes, you can be at the stop at the exact time it arrives, instead of waiting around. To enable this, we base our solutions on richly-detailed geospatial data provided by Google Maps Platform.

Eliminating wait times and increasing revenue with geospatial data

In India, bus passengers tend to have fewer resources. The Chalo App, which can be downloaded for free, allows them to see exactly where their bus is on its route and when it will arrive at their nearest stop. They also tend to be late adopters of mobile technology, meaning we had to create an interface that was user-friendly, reassuring and adapted to all age groups and backgrounds. One of the main reasons we opted for Google Maps Platform is that it’s very present in India and other emerging markets and is familiar to our users, which inspires trust. At the same time, we like the fact that Google Maps Platform provides rich geospatial data while being simple to implement and work with. We use the Geocoding API, Reverse Geocoding, and the Directions API to enable location search and provide directions.

We also worked with MediaAgility to identify the Google Maps Platform products most suited to our needs and the best practices to be followed. This helped to ensure that our business objectives could be met efficiently.

The Chalo App also enables digital ticketing, alongside the Chalo Card, a payment card for those who don’t own a smartphone. India, like the rest of the world, is gradually moving away from cash payments, but the public transport system is proving slow to catch up, meaning people still must have cash in hand when they board the bus. Digital ticketing not only makes commuting more convenient, it also helps protect passengers, drivers and conductors during the COVID-19 health crisis by limiting physical contact. 

Chalo also offers solutions aimed at bus operators that help them improve their services and their bottom line. In major Indian cities, buses are run by a combination of public and private agencies and small private individual bus operators, with the majority of the latter only operating one or two buses. The market is very fragmented and there’s not much incentive for bus operators to invest in infrastructure or customer experience – especially when they have little to no visibility on where their fleet is at a given time, how many kilometers it travels in a day, or how much money it takes. 

The Chalo dashboard provides operators with a map-based real-time overview of bus locations, alongside scheduling features and route, ticketing, and passenger statistics. Geospatial intelligence and insight into passenger demand enable operators to explore new avenues of revenue and adapt routes and services to passenger needs. Operators who’ve partnered with Chalo report an average improvement of 10% to 30% of their bus fleet operations.

Chalo is currently powering about 100 million rides a month on 15,000 buses in 37 cities. We’d like to see the number of rides increase tenfold over the next few years. To do that, we’re looking to broaden our offer and expand to other parts of the country and across international borders. A pilot is underway in Bangkok, and we’re considering expansion into South-East Asia, Africa, and the Middle East. Having access to detailed geospatial data anywhere in the world via Google Maps Platform, without having to make any major investments or changes to our technology stack, will make this considerably easier. 

At the same time, we’re exploring artificial intelligence and machine learning to improve the accuracy of our scheduling features and we’re introducing video-based solutions for people-counting on buses. Our aim is to continually improve our offering and optimize our services. We’re looking forward to working closely with Google to make that happen.  

For more information on Google Maps Platform, visit our website.

Mailjet allows companies to create, send, and track millions of emails to their customers and prospects around the world each month. The company’s offerings range from routing these emails to tailor-made tools that allow clients to design and send marketing campaigns with messages related to sales, user clubs, coupons and discounts, and promotions, as well as transactional emails such as purchase confirmations, shipping confirmations, e-tickets, or password resets.

“All of our services are accessible through programming interfaces. We use the same APIs for ourselves and for our customers. This is part of our unique selling point, which is to make life easier for developers and marketers,” explains François Fanuel, IT Operations Manager at Mailjet.

The company scrupulously monitors its service quality to help ensure that emails sent on behalf of its customers don’t trigger spam filters, and that they obtain the best possible open and click rates.

From hosting to the cloud

In its early years, Mailjet used hosted servers. This model had several disadvantages. It was impossible to instantly activate or deactivate computing power; the company could not be charged for actual, to-the-minute usage; and it lacked the flexibility to add disk space or scale servers in real time.

At the beginning of 2016, Mailjet decided to switch to the public cloud in order to improve technical and economic flexibility. “This two-year evolution required us to review our developments. If we use a virtual machine even for just one hour, our computer code must be able to support it in order to preserve the consistency and integrity of the data and processes,” says Fanuel.

“We needed a way to port our IP addresses to Google Compute Engine. In less than a month, the feature had been developed by Google engineers. This reactivity really impressed us and was definitely a deciding factor.”

—François Fanuel, IT Operations Manager, Mailjet

Mailjet and Google were already partners. The marketing and transactional email specialist is one of only three providers in the world, and the only one in Europe, accredited by Google to send bulk emails through Google Cloud Platform. The choice—based on three main factors—was obvious.

Financial and technical considerations

Factor 1: the right price proportionality. “We are able to activate Google Compute Engine virtual machines on demand, without needing to reserve these instances. While the machine remains active, a sliding-scale price is applied, which is particularly attractive to us,” explains Fanuel.

Factor 2: the quality and level of technical communication. Mailjet has direct access to Google Cloud Platform engineers when required. The company has its own IP addressing infrastructure, which is the foundation of its service quality. “We needed a way to port our IP addresses to Google Compute Engine. In less than a month, the feature had been developed by Google engineers. This reactivity really impressed us and was definitely a deciding factor.” Since then, via specific tunnels, Mailjet IP addresses have been ported directly into Google Compute Engine.

Factor 3: conformity to international data regulations. Mailjet is the only email service provider to be ISO 27001 certified and GDPR-ready, enabling the company to offer its clients the highest level of data security and privacy. Google Cloud Platform is also ISO 27001 certified, which was an important factor for Mailjet.

Power through minimalism

In mid-2016, Mailjet replaced its 30 or so email sending servers installed at a hosting company with smaller Google Compute Engine virtual machines, using the automatic load-balancing function. “Using smaller virtual machines—but more of them—optimized our computing power.” This precision mechanism accompanied an increase in activity for the company, with the number of emails sent doubling every year. As for the company’s some 300 TB of data, this is stored in Google Cloud Storage.

The flexibility and power of Google Cloud Platform helps ensure that Mailjet can adapt to peaks in activity that can reach up to 20 times the normal flow—our customers usually send their messages at the same times and during the same peak periods. “Previously, we had to reserve a lot of computing power with our hosting company, equivalent to 150 servers. Moving to an on-demand infrastructure means we can avoid paying a disproportionate fixed annual cost.”

Google Cloud Platform also provides Mailjet with the infrastructure needed to process events centrally, which facilitates technical support. A new analytics dashboard in the making will provide customers access to their campaigns’ key metrics in real time—a circulation report, open rates, and click-through rates, as well as invalid email addresses— regardless of the volume of emails at stake.

From distribution to development

Following routing operations, the development environment is currently being ported to Google Cloud Platform, and will eventually represent 100 virtual servers. “The addition of any new code is iteratively tested in Google Cloud Platform. We benefit from machines that are custom made for our needs, from machines with one core and 3 gigabytes of memory, up to machines with 64 cores.”

Mailjet uses the Google Cloud Interconnect virtual private network to tailor sending flows by geographical location. “For customers who only use our routing service, we will soon have worldwide load balancers. This will help ensure their API requests are received and their emails are sent to Europe and America from one single IP point, thus avoiding transatlantic latency issues.”

Mailjet also uses the Google Identity Aware Proxy authentication tool. All its employees around the world benefit from unique, more secure access, whether they use Google Sites intranets, or the development, or production environment. “Google Cloud Platform delivers on all the promises of a public cloud: service customization, real-time computing power adjustment during periods of high and low demand, and instant activation of computing resources. For us, it’s the best possible ratio between price, performance, availability, and quality,” concludes Fanuel.

As Google’s productivity adviser, I’ve been closely following the evolving discussion around the future of work. Long before the pandemic, employees, business leaders, and analysts were already thinking about how work needed to evolve, but the crisis brought the future cascading  into the present. Virtually overnight, millions of companies and workers went remote and stayed that way for over a year, pushing their collaboration and productivity tools to the test across living rooms and time zones. While some organizations were able to launch new solutions or adapt existing ones to keep their people connected and productive, many struggled. 

The organizations that struggled quickly discovered their tools weren’t complete, scalable, secure, or built for the cloud era. Meanwhile, across all industries and business types, employees reported spikes in burnout, feelings of disconnection, and frustration with finding the latest information or using unfamiliar tools.

With many businesses headed into a hybrid model, the world of work has been transformed, possibly forever. There’s a fresh and fast-moving conversation about how organizations can succeed within the evolving future of work. 

But at the center of that discussion are two familiar topics: collaboration and productivity. How will they evolve in an era of distributed teams and surging employee demand for  flexibility? And how will businesses meet expectations to innovate quickly and deliver on rising customer expectations while navigating the new future of work? 

Although 2020 was a major inflection point, a closer look reveals that many of the technologies, trends, and cultural norms shaping the future of work have been around for some time. For years, forward-thinking organizations have been wrestling with how to maximize collaboration, productivity, and wellbeing among their employees, and they’ve been developing the tools to make it happen at scale. By that measure, the future of work has been here for some time; it just hasn’t been evenly distributed or easily visible.

With this context in mind, we’re sharing our new guide Create what’s next: The future of collaboration and productivity, which highlights three areas of focus for organizations wanting to catch up with those leading the charge to empower the future of work.

1. Making work-from-anywhere a reality with flexible solutions
2. Giving people helpful tools to maximize their impact
3. Enabling knowledge sharing and human connection

Armed with these three strategies, businesses can improve productivity and encourage innovation while better meeting the needs of their customers and their employees—now and in the years ahead.

We share some highlights from the guide in the infographic below (click to enlarge). You can also download the full guide now.

Let’s face it: in the globalized world, which is now more than ever a digital demand world, you need to scale and reach your customers right where they’re at. Translation is a critical piece of that, whether you’re translating a website in multiple languages or releasing a document, a piece of software, or training materials.

Manual translation does not scale, which is why machine translation, powered by machine learning (ML), is becoming more important to our customers.  Machine translation has historically been challenging because of the sheer volume and breadth of content that can add value when translated into multiple languages. Companies acquire and share content in many languages and formats, and scaling translation to meet needs is a tall order due to multiple document formats, integrations with optical character recognition (OCR), and the need to correct for domain-specific terminology.

Our goal is to simplify translation services, while enabling flexibility and control for our customers’ unique needs across industries. Read on to learn more about recent features and updates. 

Formatting matters: Document Translation is now GA 

In many cases, the layout of a document dictates how it should be interpreted—e.g., readers navigate text and discern meaning based on formatting, like bold or italicized text, or markups for headers, paragraphs, and columns. Previously, to automate translation of documents, text needed to be separated from these layout attributes, meaning the document’s structure was either lost or needed to be recreated later in the developer pipeline, after the text had been translated. This required translation teams to do a lot of extra work and maintain a lot of additional code. But now, those steps are unnecessary. Formatting can be retained throughout the translation process, handled directly by the Translation API Advanced. 

This feature lets customers translate documents in 100+ languages and supports document types such as Docx, PPTx, XLSx, and PDF while preserving document formatting.

And if your needs go beyond Document Translation, we can help you translate audio as well. For real-time streaming translation, check out the Media Translation API, and for offline transcription translation, combine the Translation API with the Video Intelligence API.   

Real-Time translation when you need it, Batch when you don’t

One of the biggest differentiators for Translation API Advanced’s document translation capabilities is the ability to do real-time, synchronous processing for a single file. 

For example, if you are translating a business document such as HR documentation, online translation provides flexibility for smaller files and provides faster results. You can easily integrate with our APIs via REST or gRPC with mobile or browser applications, with instant access to 100+ language pairs so that content can be understandable in any supported language. 

Meanwhile, batch translation allows customers to translate multiple files into multiple languages in a single request. For each request, customers can send up to 100 files with a total content size of up to 1 GB or 100 million Unicode codepoints, whichever limit is hit first.

State of the Art (SOTA) accuracy, with flexibility for customization

In order to achieve the highest level of accuracy for your translation, we now support multiple options:

• Use Google’s SOTA translation models: Each year, Google heavily invests to improve the quality of our translations across Apps, Cloud APIs, and Chrome, as well to enable multilanguage answers in Search. A popular metric for automatic quality evaluation of Machine translation systems is the BLEU score, which is based on the similarity between machine translation and the reference translations that were generated by people. While we push out incremental improvements for individual models on a monthly cadence, there are also times where we make significant leaps. In the releases since 2019, we have improved our average BLEU score by 5pts on average across 100+ languages and 7pts on low resource languages.
• Leverage glossaries for specific terms and phrases: Glossary is our terminology control feature. It allows you to import source content to define preferred translations, such as product names or department names. Then, when calling the glossary in the API request, your preferred translations will be enforced. This will work for words as well as phrase translation.
• Pick a pre-trained model with model selection: If you create custom models for machine translation, we don’t think you should have multiple client libraries and multiple APIs to maintain in order for you to use the best model for your needs. Translation API Advanced now supports Model Selection. Pick your pretrained model or pick your custom ML model built on AutoML for any language pair you’ve created and use the same API and the same client library. 
• Build custom translation models with AutoML: AutoML Translation is a suite of ML products that enable you to build high quality models for your own use case or data, with limited-to-no ML expertise or coding required. Bring your past human-validated translations to improve translation specificity for your domain.

Keep localization local with Regional Endpoints

If you are a customer operating in the EU, we recently launched an endpoint specifically for EU regionalization. This is a configurable endpoint for customers to store and perform machine translation processing of customer data only in the EU multi region. For now, this only supports our pretrained translation models and glossary, but batch translations will be coming soon.

How Eli Lilly uses Cloud Translation to translate content globally

Historically, translations at Eli Lilly have been complicated: numerous translation vendors have been needed for different languages and organizations, all with their own processes and expectations. On top of that, translations have been costly and slow. 

To solve this, Eli Lilly took a codified approach to enable users and systems to spend less time and resources to safely generate quality translations. 

Learn more, and even catch a demo, from Thomas Griffin, Translation Tech Lead & Global Regulatory Architect for Eli Lilly.

Learn more 

• To get started using Cloud Translation – Advanced, complete the setup and then try the Translate text (Advanced edition) quickstart.
• Document Translation is priced per page. For more information, see pricing.