Vinayak Bhavnani, Co-Founder and CTO of India-based bus transport technology company Chalo, shares how Google Maps Platform is used to improve visibility for commuters and bus operators across India by visualizing geospatial data.

Effective public transport networks contribute to the local economy and help make cities safe, pleasant, and sustainable. In India, buses make up around 90% of the public transport offering, but when you talk to the people who ride them every day, you find that there’s a lot of room for improvement. Heavy traffic means there are rarely any fixed schedules and it’s impossible to know exactly when your bus is coming. We’ve found that people tend to wait at a bus stop for up to 30 minutes a day, which creates a lot of frustration and wasted time.

When we founded Chalo, our aim was to make the daily city commute a more positive experience. Reliability is synonymous with visibility: when you know exactly when the bus is coming, you can plan your day better. If you’re in your office, for example, and see the next bus is in 10 minutes, you can be at the stop at the exact time it arrives, instead of waiting around. To enable this, we base our solutions on richly-detailed geospatial data provided by Google Maps Platform.

Eliminating wait times and increasing revenue with geospatial data

In India, bus passengers tend to have fewer resources. The Chalo App, which can be downloaded for free, allows them to see exactly where their bus is on its route and when it will arrive at their nearest stop. They also tend to be late adopters of mobile technology, meaning we had to create an interface that was user-friendly, reassuring and adapted to all age groups and backgrounds. One of the main reasons we opted for Google Maps Platform is that it’s very present in India and other emerging markets and is familiar to our users, which inspires trust. At the same time, we like the fact that Google Maps Platform provides rich geospatial data while being simple to implement and work with. We use the Geocoding API, Reverse Geocoding, and the Directions API to enable location search and provide directions.

We also worked with MediaAgility to identify the Google Maps Platform products most suited to our needs and the best practices to be followed. This helped to ensure that our business objectives could be met efficiently.

The Chalo App also enables digital ticketing, alongside the Chalo Card, a payment card for those who don’t own a smartphone. India, like the rest of the world, is gradually moving away from cash payments, but the public transport system is proving slow to catch up, meaning people still must have cash in hand when they board the bus. Digital ticketing not only makes commuting more convenient, it also helps protect passengers, drivers and conductors during the COVID-19 health crisis by limiting physical contact. 

Chalo also offers solutions aimed at bus operators that help them improve their services and their bottom line. In major Indian cities, buses are run by a combination of public and private agencies and small private individual bus operators, with the majority of the latter only operating one or two buses. The market is very fragmented and there’s not much incentive for bus operators to invest in infrastructure or customer experience – especially when they have little to no visibility on where their fleet is at a given time, how many kilometers it travels in a day, or how much money it takes. 

The Chalo dashboard provides operators with a map-based real-time overview of bus locations, alongside scheduling features and route, ticketing, and passenger statistics. Geospatial intelligence and insight into passenger demand enable operators to explore new avenues of revenue and adapt routes and services to passenger needs. Operators who’ve partnered with Chalo report an average improvement of 10% to 30% of their bus fleet operations.

Chalo is currently powering about 100 million rides a month on 15,000 buses in 37 cities. We’d like to see the number of rides increase tenfold over the next few years. To do that, we’re looking to broaden our offer and expand to other parts of the country and across international borders. A pilot is underway in Bangkok, and we’re considering expansion into South-East Asia, Africa, and the Middle East. Having access to detailed geospatial data anywhere in the world via Google Maps Platform, without having to make any major investments or changes to our technology stack, will make this considerably easier. 

At the same time, we’re exploring artificial intelligence and machine learning to improve the accuracy of our scheduling features and we’re introducing video-based solutions for people-counting on buses. Our aim is to continually improve our offering and optimize our services. We’re looking forward to working closely with Google to make that happen.  

For more information on Google Maps Platform, visit our website.

Businesses around the globe are realizing the benefits of replacing legacy data silos with cloud-based enterprise data warehouses, including easier collaboration across business units and access to insights within their data that were previously unseen. However, bringing data from numerous disparate data sources into a single data warehouse requires you to develop pipelines that ingest data from these various sources into your enterprise data warehouse. Historically, this has meant that data engineering teams across the organization procure and implement various tools to do so. But this adds significant complexity to managing and maintaining all these pipelines and makes it much harder to effectively scale these efforts across the organization. Developing enterprise-grade, cloud-native pipelines to bring data into your data warehouse can alleviate many of these challenges. But, if done incorrectly, these pipelines can present new challenges that your teams will have to spend their time and energy addressing. 

Developing cloud-based data ingestion pipelines that replicate data from various sources into your cloud data warehouse can be a massive undertaking that requires significant investment of staffing resources. Such a large project can seem overwhelming and it can be difficult to identify where to begin planning such a project. We have defined the following principles for data pipeline planning to begin the process. These principles are intended to help you answer key business questions about your effort and begin to build data pipelines that address your business and technical needs. Each section below details a principle of data pipelines and certain factors your teams should consider as they begin developing their pipelines.

Principle 1: Clarify your objectives

The first principle to consider for pipeline development is clarify your objectives. This can be broadly defined as taking a holistic approach to pipeline development that encompasses requirements from several perspectives: technical teams, regulatory or policy requirements, desired outcomes, business goals, key timelines, available teams and their skill sets, and downstream data users. Clarifying your objectives clearly identifies and defines requirements from each key stakeholder at the beginning of the process and continually checks development against these requirements to ensure the pipelines built will meet these requirements.This is done by first clearly defining the desired end state for each project in a way that addresses a demonstrated business need of downstream data users. Remember that data pipelines are almost always the means to accomplish your end state, rather than the end state itself. An example of an effectively defined end-state is “enabling teams to gain a better understanding of our customers by providing access to our CRM data within our cloud data warehouse” rather than “move data from our CRM to our cloud data warehouse”. This may seem like a merely semantic difference, but framing the problem in terms of business needs helps your teams make technical decisions that will best meet these needs. 

After clearly defining the business problem you are trying to solve, you should facilitate requirement gathering from each stakeholder and use these requirements to guide the technical development and implementation of your ingestion pipelines. We recommend gathering stakeholders from each team, including downstream data users, prior to development to gather requirements for the technical implementation of the data pipeline. These will include critical timelines, uptime requirements, data update frequency, data transformation, DevOps needs, and security, policy, or regulatory requirements by which a data pipeline must meet.

Principle 2: Build your team

The second principle to consider for pipeline development is build your team. This means ensuring you have the right people with the right skills available in the right places to develop, deploy, and maintain your data pipelines. After you have gathered your pipeline requirements, you can begin to develop a summary architecture that will be used to build and deploy your data pipelines. This will help you identify the human talent you will need to successfully build, deploy, and manage these data pipelines and identify any potential shortfalls that would require additional support from either third-party partners or new team members.

Not only do you need to ensure you have the right people and skill sets available in aggregate, but these individuals need to be effectively structured to empower them to maximize their abilities. This means developing team structures that are optimized for each team’s responsibilities and their ability to support adjacent teams as needed.

This also means developing processes that prevent blockers to technical development whenever possible, such as ensuring that teams have all of the appropriate permissions they need to move data from the original source to your cloud data warehouse without violating the concept of least privilege. Developers need access to the original data source (depending on your requirements and architecture) in addition to the destination data warehouse. Examples of this are ensuring that developers have access to develop and/or connect to a Salesforce Connected App or read access to specific Search Ads 360 data fields.

Principle 3: Minimize time to value

The third principle to consider for pipeline development is minimize time to value. This means considering the long-term maintenance burden of a data pipeline prior to developing and deploying it in addition to being able to deploy a minimum viable pipeline as quickly as possible. Generally speaking, we recommend the following approach to building data pipelines to minimize their maintenance burden: Write as little code as possible. Functionally, this can be implemented by:

1. Leveraging interface-based data ingestion products whenever possible. These products minimize the amount of code that requires ongoing maintenance and empower users who aren’t software developers to build data pipelines. They can also reduce development time for data pipelines, allowing them to be deployed and updated more quickly. 

• Products like Google Data Transfer Service and Fivetran allow for managed data ingestion pipelines by any user to centralize data from SaaS applications, databases, file systems, and other tooling. With little to no code required, these managed services enable you to connect your data warehouse to your sources quickly and easily.
• For workloads managed by ETL developers and data engineers, tools like Google Cloud’s Data Fusion provide an easy-to-use visual interface for designing, managing and monitoring advanced pipelines with complex transformations.

2. Whenever interface-based products or data connectors are insufficient, use pre-existing code templates. Examples of this include templates available for Dataflow that allow users to define variables and run pipelines for common data ingestion use cases, and the Public Datasets pipeline architecture that our Datasets team uses for onboarding.

3. If neither of these options are sufficient, utilize managed services to deploy code for your pipelines. Managed services, such as Dataflow or Dataproc, eliminate the operational overhead of managing pipeline configuration by automatically scaling pipeline instances within predefined parameters.

Principle 4: Increase data trust and transparency

The fourth principle to consider for pipeline development is increase data trust and transparency. For the purposes of this document, we define this as the process of overseeing and managing data pipelines across all tools. Numerous data ingestion pipelines that each leverage different tools or are not developed under a coordinated management plan can result in “tech sprawl”, which significantly increases the management overhead of data ingestion pipelines as the quantity of data pipelines increases. This becomes especially cumbersome if you are subject to service-level agreements, or legal, regulatory, or policy requirements for overseeing data pipelines. Preventing tech sprawl is, by far, the best strategy for dealing with it by developing streamlined pipeline management processes that automate reporting. Although this can theoretically be achieved by building all of your data pipelines using a single cloud-based product, we do not recommend doing so because it prevents you from taking advantage of features and cost optimizations that come with choosing the best product for your use case. 

A monitoring service such as Google Cloud Monitoring Service or Splunk that automates metrics, events, and metadata collection from various products, including those hosted in on-premise and hybrid computing environments, can help you centralize reporting and monitoring of your data pipelines. A metadata management tool such as Google Cloud’s Data Catalog or Informatica’s Enterprise Data Catalog can help you better communicate the nuances of your data so users better understand which data resources are best fit for a given use case. This significantly reduces your pipeline’s governance burden by eliminating manual reporting processes that often result in inaccuracies or lagging updates.

Principle 5: Manage costs

The fifth principle to consider for pipeline development is manage costs. This encompasses both the cost of cloud resources and the staffing costs necessary to design, develop, deploy, and maintain your cloud resources. We believe that your goal should not necessarily be to minimize cost, but rather maximizing the value of your investment. This means maximizing the impact of every dollar spent by minimizing waste in cloud resource utilization and human time. There are several factors to consider when it comes to managing costs:

• Use the right tool for the job – Different data ingestion pipelines will have different requirements for latency, uptime, transformations, etc. Similarly, different data pipeline tools have different strengths and weaknesses. Choosing the right tool for each data pipeline can help your pipelines operate significantly more efficiently. This can reduce your overall cost, free up staffing time to focus on the most impactful projects, and make your pipelines much more efficient.
• Standardize resource labeling –  Implement and utilize a consistent labeling schema across all tools and platforms to have the most comprehensive view of your organization’s spending. One example is requiring all resources to be labeled by the cost center or team at time of creation. Consistent labeling allows you to monitor your spend across different teams and calculate the overall value of your cloud spending.
• Implement cost controls – If available, leverage cost controls to prevent errors that result in unexpectedly large bills. 
• Capture cloud spend – Capture your spend on all cloud resource utilization for internal analysis using a cloud data warehouse and a data visualization tool. Without it, you won’t understand the context of changes in cloud spend and how they correlate with changes in business.
• Make cost management everyone’s job – Managing costs should be part of the responsibilities of everyone who can create or utilize cloud resources. To do this well, we recommend making cloud spend reporting more transparent internally and/or implementing chargebacks to internal cost centers based on utilization.

Long-term, the increased granularity in cost reporting available within Google Cloud can help you better measure your key performance indicators. You can shift from cost-based reporting (i.e. – “We spent $X on BigQuery storage last month”) to value-based reporting (i.e. – “It costs $X to serve customers who bring in $Y revenue”). 

To learn more about managing costs, check out Google Cloud’s “Understanding the principles of cost optimization” white paper.

Principle 6: Leverage continually improving services

The sixth principle is leverage continually improving services. Cloud services are consistently improving their performance and stability, even if some of these improvements are not obvious to users. These improvements can help your pipelines run faster, cheaper, and more consistently over time. You can take advantage of the benefits of these improvements by:

• Automating both your pipelines and pipeline management: Not only should data pipelines be automated, but almost all aspects of managing your pipelines can also be automated. This includes pipeline/data lineage tracking, monitoring, cost management, scheduling, access management and more. This helps reduce long-term operational costs of each data pipeline that can significantly alter your value proposition and prevent any manual configurations from negating the benefits of later product improvements.
• Minimizing pipeline complexity whenever possible: While ingestion pipelines are relatively easy to develop using UI-based or managed services, they also require continued maintenance as long as they are in use. The most easily maintained data ingestion pipelines are typically the ones that minimize complexity and leverage automatic optimization capabilities. Any transformation in a data ingestion pipeline is a manual optimization of the pipeline that may struggle to adapt or scale as the underlying services improve. You can minimize the need for such transformations by building ELT (extract, load, transform) pipelines rather than ETL (extract, transform, load) pipelines. This pushes transformations down to the data warehouse that is use a specifically optimized query engine to transform your data rather than manually configured pipelines.

Next steps

If you’re looking for more information about developing your cloud-based data platform, check out our Build a modern, unified analytics data platform whitepaper. You can also visit our data integration site to learn more and find ways to get started with your data integration journey.

Once you’re ready to begin building your data ingestion pipelines, learn more about how Cloud Data Fusion and Fivetran can help you make sure your pipelines address these principles.

This April, Gmail will be 18 years old. That’s a big milestone, given that email as a medium just turned 50.

Launched in 2004—with its 1 GB free storage, conversation based grouping, and enhanced search capabilities—Gmail quickly became one of the most popular email services in the world.

What started off as Paul Buchheit’s side project is now part of Google Workspace, a solution that’s used by over 3 billion users. 

Yet despite this growth and Gmail’s long history, the product continues to find new ways to delight its users with great experiences, new features, and constant innovation.  

How does the product team do it? 

To help us unpack this question, I interviewed Matthew Izatt, Product Lead for Gmail.

Matthew has been building and growing Gmail for the last 10+ years. He started off in mobile where he built the Android and iOS apps for Gmail, and eventually he became the lead product owner for all of Gmail’s front end. He also led the development of the new Gmail in 2018, which was a complex rebuild of the technical architecture and visual design, along with the addition of multiple enterprise focused features. Since then, he has been leading Google’s charge on redefining how we work as the product lead for the Workspace Platform and Partnerships efforts with a team of over 100.

What follows is an edited version of our interview. 

Mustafa Kapadia: Matt, thank you for joining us. Now, before we jump into how your team does its magic, can you please share with me some of the challenges you face when you build a new feature or experience on Gmail?

Matthew Izatt: Thanks Mustafa. And I think this is a great place to start.

One of the biggest challenges is that Google Workspace is used by over 3 billion users, and while that is great, there are many different ways of using our product. It’s not like a doorbell, where there is only one way of using it. With Gmail, it’s like an infinite Swiss army knife, where everyone has their own favorite way of using the product.

Second, Gmail is a critical product for many of our users. Yes, it’s fun to use, but people use it to get important work done—sending job proposals, emailing with their kids’ teachers, or booking medical appointments. This is why we have to be careful about what we experiment with and what we roll out. We can’t just throw things out there and see what works. We have to do our homework.

Third, we have to constantly innovate and think outside the box. Email is an old product, but it continues to evolve. Every day we have new startups reinventing email and challenging the big players. If we are to remain a market leader, we literally have to rethink the product every few years. Which is why creating the new Gmail was so much fun. 

Mustafa: I don’t envy you. This is not an easy product to build. Going back to something you touched on: Given these hurdles, how do you build these great experiences? Is there an example that you can share with us?

Matt: A specific example I always like to share is the story of how we came up with the right display density on our thread list. The thread list is your inbox; it’s the list of all the messages that you have. The display density is how that information is presented to you.

In Gmail you can choose to have fewer messages with a lot of information about each message. Or you can choose to see a larger number of messages with much less information about each message.

It really comes down to your preferences. And it’s one of the reasons why so many users like using Gmail. In fact, being able to set inbox display preferences is actually one of the top 10 most used settings within Gmail.

Mustafa: So you were really trying to give users choice on how they would like their information presented?

Matt: Exactly. Now, we could have added 100 different levers so that every user could customize their email feed to an infinite degree. While that is possible from a coding perspective, it’s not practical for three reasons.

First, infinite choice sounds good on paper, but in reality, too many options creates a confusing user experience. Second, development would take many months—maybe even a year. And third, it would be a long-term maintenance nightmare. 

Mustafa: So how did you go about doing it? Share with me the inner workings of Gmail in Google Workspace. How is the sausage made? 

Matt: Well, it came down to four key steps, working closely with our amazing UX Research team.

First came the primary research to find out if building this experience was worth doing. After all, what is the point of building it if no one wants it? For this we actually sat down with real users inside and outside of Google to understand their needs.

Second, we brainstormed 100 potential solutions and narrowed them down to 8 options. Then we mocked up each of the 8 options to see how they looked.

While it would have been easy to just pick the option we liked the most and build it, we wanted to create what our users want, not what we like the most. So we took those options and surveyed the users from step 1 again. We specifically asked them to rank their preferences across three dimensions: Love It, Hate It, or Meh!

Based on the user responses, we realized that 3 configurations would meet the needs of most users. These were the three options we developed and deployed to Google Workplace.

Mustafa: I love how you used data to guide your decision of what to build. This really is Google’s secret sauce—first-hand data collected from users.

Matt: Absolutely. Without their feedback, we would have never been able to narrow down to the 3 options that would satisfy most of our users.

Mustafa: Talking about risks: Getting user feedback and validating before you build is one way to de-risk your product, but you even de-risked your product launch strategy, correct?

Matt: Yes. Sometimes we launch features individually. Sometimes we launch them as a group. In this particular case, we had a lot of moving parts. We had a complete UI redesign, 10 or 12 new features, and architecture changes.

So to de-risk this, we put in place—I kid you not—a 26-stage launch process. We started off with the technical architecture changes—stuff that the user would not notice—to make sure that it was working. And we did it in small increments.

Then we rolled out a few features at a time, launching it first to our early access program—where users have opted in to receive updates. We measured their reaction and satisfaction first, and then moved on to the next group.

At every stage, we kept our blast radius small and put in these checks and balances where we could launch, measure reaction, and then reassess our strategy. And that worked out pretty well for us for this massive launch.

Mustafa: Agreed! Thank you for taking the time to talk to me. And good luck with building the next version of Gmail.

Matt: Thanks for having me.