Editor’s note: February is Black History Month—a time for us to come together to celebrate the diverse set of experiences, perspectives and identities that make up the Black experience. Over the next few weeks, we will highlight Black-led startups and how they use Google Cloud to grow their businesses. Today’s feature highlights Acclinate and its founders, Del and Tiffany. 

As patients, as caregivers, and as parents taking our own children to the doctor, we want recommended medications to be safe and effective. It’s a right everyone deserves.

It’s known that certain medications don’t work in the same way in all populations. For example, Albuterol, a medication often prescribed for asthma, is less effective in 67% of all Puerto Ricans and 47% of Black Americans. These problems—which can have deadly consequences—result from historically limited diversity in pharmaceutical clinical trials. 

We founded our startup Acclinate to integrate culture and technology to achieve more inclusive clinical research. Help pharmaceutical companies and healthcare organizations access and engage communities of color so research is more inclusive.

Bridging the health equity gap by building trust

It’s important that health research organizations access and engage communities of color so their efforts reflect all the people they serve. Take a disease like diabetes, which affects a significantly higher proportion of Black Americans. When you look at even recent clinical trials for diabetic drugs, the representation of Black Americans among participants is only in the low single digits, despite comprising 13 percent of the U.S. population and more than 40 percent of diabetes patients in this country. Industry leaders have been aware of the lack of diversity issue, but some have chosen to ignore it or brush it aside. The biggest problem, in our opinion, is that there have been no penalties for not achieving higher diversity figures in clinical trials, and only minor financial repercussions to pharma/biotech companies when their treatments either do not work across all groups once approved, or there is a lack of uptake by all groups due to the lack of testing in those groups. The lack of clinical trial diversity has adversely impacted the reputation of the industry and the ability to recruit diverse populations in the future. 

Acclinate integrates culture and technology to promote diverse patient representation in medical research. Our approach is not transactional. We build trust through our  #NOWINCLUDED community, which is an ongoing, ever-expanding digital platform that educates and engages with communities of color on health issues.

#NOWINCLUDED includes a website app, and social media presence where members can learn information about diseases, particularly those with greater negative impacts on people of color, such as cancer, diabetes, and cardiovascular diseases. Members can share stories and ask questions. By providing access to trusted resources about these health issues and the latest clinical research, we empower Black people to take control of their health and consider  participating in research that is shaping the future of healthcare.  

For healthcare-related organizations, we offer the opportunity to better understand the attitudes, aspirations, and unmet needs of underrepresented minority communities. Data from #NOWINCLUDED feeds our HIPAA-compliant SaaS platform, e-DICT™ (Enhanced Diversity in Clinical Trials), which uses predictive analytics and machine learning to identify individuals matching the requirements and most likely to be receptive to participation in a particular clinical trial.

Acclinate scales its platform with Google Cloud

We rely on Google Cloud services, including Vertex AI, to know whom to ask, when to ask, and how to ask for clinical trial participation. With Vertex AI, we enjoy a unified platform for developing our artificial intelligence models, including tools for preparing and storing our datasets. We can easily train and compare models using AutoML, which requires minimal ML expertise or effort with its intuitive graphical interface. This allows us to leverage more than ten ordinal and categorical data points to determine in real time a community member’s likelihood to enroll, which we call our Participation Probability Index (PPI). Our models evolve in an iterative process the more we interact with, and learn about, our community members.

We follow the pay-per-use Google Cloud Platform architecture model using serverless technology, which helps reduce infrastructure management costs and lets us focus on product development and engaging with communities across the U.S.

CloudSQL, a fully managed relational database service, integrates easily with BigQuery so we can glean insights for our clients in real time, all with Google Cloud’s robust security, governance, and reliability controls. Virtual Private Cloud (VPC) gives us scalable and flexible networking for our cloud-based resources and services. We also use Identity and Access Management (IAM) to simplify oversight of Google Cloud resource permissions for different user groups and roles, with appropriate security protections. 

API Gateway manages our APIs using Cloud Functions, which both use consumption-based pricing, plus give our developers consistent and highly secure access to our services through a well-defined REST API. We use Memorystore for Redis to reduce platform latency. This is done with a fully managed service powered by the Redis in-memory data store, which builds application caches for fast data access. All of this comes together to provide an outstanding experience for our platform’s users and contributors.

• DSC00049.JPG
• DSC00010 (2).JPG
• pitch del tiff.jpg

Expanding influence with the Google for Startups Accelerator for Black Founders

Three months of one-on-one Google support and mentorship as part of the most recent Google for Startups Accelerator : Black Founders cohort not only helped us build our product, but also helped us earn external credibility. People use Google every day, so whether we’re trying to engage in conversations with industry experts or with somebody in a rural community, it is helpful to have the buy-in of a globally-recognized brand as we take on a historically difficult, systemic issue with challenges around trust. Getting access to the products, best practices, and people we need to build and grow through the Accelerator program has been priceless. For example, working with the Google AdWords team helped us generate important traffic from people interested in learning more about #NOWINCLUDED or sharing their story with us. Jason Scott, who leads the Google for Startups Accelerator: Black Founders program, is still connecting us to people in his network and identifying key opportunities for us months after the program wrapped. He continues to demonstrate that he is invested in seeing us succeed. 

Our company has made great progress against our goals, in part thanks to receiving capital from the Google for Startups Black Founders Fund. We received $100K in non-dilutive funding along with Google Cloud credits, Google.org Ads grants, and hands-on support. We used the funds to pay for the transition and development costs associated with moving to Google Cloud. The Google support and accountability has been incredible. After receiving the Google for Startups Black Founders Fund award, we’ve gone on to raise another $1M and moved our cloud from Salesforce to Google Cloud. 

We also had the amazing opportunity to be selected as one of six companies to take part in a face-to-face web conference with Sundar Pichai, Google’s CEO. We were thrilled to hear him explain his vision around health equity and the role Google plays. Ultimately, for us, it’s not just about the funding we get, but we are also gratified to receive support from an entity that truly believes in addressing this issue. We know Google is aligned with our mission of health and racial equity. 

Championing diversity in clinical trials

Our 2022 looks bright. We expanded our presence to Washington, D.C. as part of the Johnson & Johnson Innovation JLABS ecosystem. We were also selected to take part in the BLUE KNIGHT initiative created between Johnson & Johnson Innovation and the Biomedical Advanced Research and Development Authority (BARDA) under the U.S. Department of Health and Human Services. 

Acclinate is also on track to have contracts with five of the top 25 largest biopharmaceutical companies in the U.S this year. They’ve taken note, as has the Food and Drug Administration (FDA), that the lack of diversity in clinical trials represents a significant health concern—to the extent that the FDA has provided strong guidance for pharmaceutical companies to  diversify their clinical trials. At the same time, the industry is also responding to pressure from communities of people of color to make equitable representation a priority.

Today, we are in the fortunate but challenging position to have significant inbound opportunities coming our way. In response, we continue to recruit and hire talented people to join our team. On the technology side, we are happy to be aligned with Google Cloud to have powerful cloud infrastructure that will scale with us, as well as high-caliber champions united in partnership. With people’s lives at stake, we are passionate in our commitment to helping ensure medications do what they are supposed to do: heal and improve the quality of life for everyone who takes them. 

Hear Acclinate cofounders Del Smith and Tiffany Whitlow chat with Google’s Head of Startup Developer Ecosystem Jason Scott and fellow Black Founders Fund recipient Bobby Bryant about building on Google Cloud in a recent Google for Startups Instagram Live. 

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more

Editor’s note: In this blog post we learn how TapClicks migrated to Google Cloud to offer their marketing customers a unified platform for data management, operations, insights, and analysis.

TapClicks is a smart marketing cloud, powered by data, that unifies our customer’s marketing. By choosing to migrate our core applications last year to Google Cloud, we cut costs, solved data-sharing concerns for our customers, and opened our stack up to a new ecosystem of possibilities. 

The core problem that we’re solving for our customers is how to manage their marketing infrastructures data and operations. Life isn’t easy for marketers now. There are 7,000 different vendors servicing this space today – creating much complexity between digital agencies, media, and brands. Marketers face challenges in navigating all of these systems, logging in and out, understanding pacing goals, and managing the flow of marketing data so they can analyze and report internally as well as to their clients at scale.

We unify omnichannel campaign data (250 API connectors and 6000 Smart Connectors ™ ) from a plethora of marketing sources on an automated data warehousing solution, creating simplicity for organizations. Over 4,000 agencies, media companies, and brands use our Marketing Operations and Data Management Platform, which imports data at scale and creates an automatic data warehouse on Google Cloud. Teams can also leverage TapClicks, like our world class Facebook connector, to import data directly into Google Data Studios.  Beyond importing and storing, we also provide data exporting to other Google solutions like Google Data Studio and Google Sheets.  We also create interactive dashboards that let stakeholders and clients analyze their data, as well as automated, multi-channel reports that go out to clients at specified times. So channel comparisons, optimizations, attribution, and calculations are easily performed.  Some of our customers are able to generate hundreds of thousands of individual reports and dashboards for their clients.

Although we may be best known for our reporting and analytics, we also empower teams managing the marketing operations workflow from customers and internal stakeholders, especially at scale. Our user-friendly, configurable system helps manage their orders and campaigns. Through automation of this process, we deliver tremendous amounts of efficiency, time saving, cost savings, and reduction of errors. The combination of these solutions makes up our unified platform, with additional capabilities like marketing intelligence that offers competitive and brand-level analysis. This is a disruptive solution in use by all leading media companies, agencies and many brands.

Partnering for possibilities

We faced a few challenges with our original tech stack, which included a mix of the leader in web services revenue, leaders in high performance data warehousing, as well as vendors on bare metal servers. 

• One challenge was around costs, which were growing. 
• Second, many of our customers work with multiple brands, and are very hesitant to share their data with the leader in web services, who’s often viewed as their competitor. 
• Third, these vendors are more focused on their own revenue rather than a true long term partnership that would enable their customers to enjoy similar success as they have experienced.

When looking at other cloud providers, Google Cloud emerged for us as the front runner. They were competitive on costs, and their native Kubernetes support was superior— a big selling point for our DevOps team. There’s also a movement in the marketing and advertising industry away from AWS toward Google Cloud because of the data-sharing concern. Finally, most of our customers are already using Google Cloud tools, so there’s brand recognition and familiarity there, and easier integrations with their own systems.

Migrating to Google Cloud

Our migration, which took about five months, involved moving a significant chunk of our infrastructure, including our core applications, using Google Kubernetes Engine (GKE). In our legacy architecture, each of our clients was assigned to one of our virtual machines (VMs), and there was a lot of unused capacity because we had to provision for the max usage. We appreciated GKE’s cloud native capabilities, especially autoscaling, a huge benefit for our web application. We have varying usage patterns during the day, and though our application is mostly used during business hours, there are also days in the month of higher usage, and autoscaling saves us time and costs. GKE also makes deployments much easier, and we anticipate a lot of benefits there for our developer environments. We’ve moved some of our microservices into GKE and plan to move more in the future. All in all, we were able to migrate our core products and the bulk of our AWS spend successfully to Google Cloud. 

We also moved from our other vendors Relational Database Service (RDS) to running MySQL on our own VMs on Google Cloud, which gives us more flexibility in terms of settings and fine tuning. We’re still trying to find the best mix as we’re modernizing our infrastructure, and we took this opportunity to migrate from MySQL 5.7 to 8.0.  

Our next stage is exploring more of the capabilities and services of Google Cloud, including BigQuery, which we’re considering for our own data warehouse. The fact that we could also run Snowflake on Google Cloud, if needed, was another selling point for our migration. 

We’re especially interested in BigQuery ML’s machine learning and natural language processing capabilities, which enabled better predictive insights. Our customers want insights from their campaigns— which are working, which are paying off, where should they invest next? Using our platform, they’re looking not only to generate reporting, but also identify opportunities to improve campaign performance. We plan to use AI and ML to improve those capabilities, so that our customers can seamlessly unlock insight and intelligence from their marketing data and campaigns.

Double-clicking on Google Cloud

For us, being able to deeply leverage and partner with Google Cloud to deliver those solutions on a single stack is critical, and we think our customers will love it. We see TapClicks and Google Cloud partnering at a level beyond what you typically see in a cloud provider relationship. Already, fifty percent of our company is working with various Google Cloud solutions, and we envision TapClicks and Google Cloud as extensions of each other, providing a single, powerful platform solution. 

Google Cloud understands the partnership concept, and their team was able to shine a light on their services and what they could bring to the table. Compared to our previous experiences, dealing with the Google Cloud team has been a true pleasure. Now that we’ve migrated, we’re ready to take our next steps into the services available to us in the Google Cloud ecosystem, and the problems we’ll continue to solve for our customers. Learn more about TapClicks and BigQuery ML.

For 117 years, the U.S. Department of Agriculture’s Forest Service has been a steward of America’s forests, grasslands, and waterways. It directly manages 193 million acres and supports sustainable management on a total of 500 million acres of private, state, and tribal lands. Its impact reaches far beyond even that, offering its research and learning freely to the world.

At Google, we’re big admirers of the Forest Service’s mission. So we were thrilled to learn in 2011 that its scientists were using Google Earth Engine, our planetary-scale platform for Earth Science data and analysis, to aid its research, understanding, and effectiveness. In the years since, Google has worked with the Forest Service to meet its unique requirements for visual information about the planet. Using both historical and current data, the Forest Service built new products, workflows, and tools that help more effectively and sustainably manage our natural resources. The Forest Service also uses Earth Engine and Google Cloud to study the effects of climate change, forest fires, insects and disease, helping them create new insights and strategies.

Besides gaining newfound depths of insight, the Forest Service has also sped up its research dramatically, enabling everyone to do more. Using Google Cloud and Earth Engine, the Forest Service reduced the time it took to analyze 10 years worth of land-cover changes from three months to just one hour, using just 100 lines of code. The agency built new models for coping with change, then mapped these changes over time, in its Landscape Change Monitoring System (LCMS) project.

Emergency responders can now work better on new threats that arise after wildfires, hurricanes, and other natural disasters. Forest health specialists can detect and monitor the impacts of invasive insects, diseases, and drought. More Forest Service personnel can use new tools and products within Earth Engine, thanks to numerous training and outreach sessions within the Forest Service.

Researchers elsewhere also benefited when the Forest Service created new toolkits, and posted them to GitHub for public use. For example, there’s geeViz, a repository of Google Earth Engine Python code modules useful for general data processing, analysis, and visualization.

This is only the start. Recently, the Forest Service started using Google Cloud’s processing and analysis tools for projects like California’s Wildfire and Forest Resilience Action Plan. Forest Service researchers also use Google Cloud to better understand ecological conditions across landscapes in projects like Fuelcast, which provides actionable intelligence for rangeland managers, fire specialists, and growers, and the Scenario Investment Planning Platform for modeling local and national land management scenarios.

The Forest Service is a pioneer in building technology to help us better understand and care for our planet. With more frequent imaging, rich satellite data sets, and sophisticated database and computation systems, we can view and model the Earth as a large-scale dynamic system.

We are honored and excited to respond to the unique set of requirements of the scientists, engineers, rangers, and firefighters of the USFS, and look forward to years of learning about — and better caring for — our most precious resources.

*Image 1: The USDA Forest Service (USFS) Geospatial Technology and Applications Center (GTAC) uses science-based remote sensing methods to characterize vegetation and soil condition after wildland fire events. The results are used to facilitate emergency assessments to support hazard mitigation, to inform post-fire restoration planning, and to support the monitoring of national fire policy effectiveness. GTAC currently conducts these mapping efforts using long-established geospatial workflows. However, GTAC has adapted its post-fire mapping and assessment workflows to work within Google Earth Engine (GEE) to accommodate the needs of other users in the USFS. The spatially and temporally comprehensive coverage of moderate resolution multispectral data sources (e.g., Landsat, Sentinel 2) and analytical power provided by GEE allows users to create geospatial burn severity products quickly and easily. Box 1 shows a pre-fire Sentinel-2 false color composite image. Box 2 shows a post-fire Sentinel-2 false color composite image with the fire scar apparent in reddish brown. Box 3 shows a differenced Normalized Burn Ratio (dNBR) image showing the change between the pre- and post-fire images in Boxes 1 and 2. Box 4 shows a thresholded dNBR image of the burned area with four classes of burn severity (unburned to high severity), which is the final output delivered to forest managers.

*Image 2: Leveraging Google Earth Engine (GEE), the USDA Forest Service (USFS) Geospatial Technology and Applications Center (GTAC) and USFS Region 8, developed the Tree Structure Damage Impact Predictive (TreeS-DIP) modeling approach to predict wind damage to trees resulting from large hurricane events and produce spatial products across the landscape. TreeS-DIP results become available within 48 hours following landfall of a large storm event to allow allocation of ground resources to the field for strategic planning and management. Boxes 1 and 3 above show TreeS-DIP modeled outputs with varying data inputs and parameters. Box 2 shows changes in greenness (Normalized Burn Ratio; NBR) that was measured with GEE during the recovery from Hurricane Ida and is shown as a visual comparison to the rapidly available products from TreeS-DIP.

*Image 3: Severe drought conditions across the American West prompted concern about the health and status of pinyon-juniper woodlands, a vast and unique ecosystem. In a cooperative project between the USDA Forest Service (USFS) Geospatial Technology and Applications Center (GTAC) and Forest Health Protection (FHP), Google Earth Engine (GEE) was used to map pinyon pine and juniper mortality across 10 Western US States. The outputs are now being used to plan for future work including on-the-ground efforts, high-resolution imagery acquisitions, aerial surveys, in-depth mortality modeling, and planning for 2022 field season work.

Box 1 contains remote sensing change detection outputs (in white) generated with GEE, showing pinyon-juniper decline across the Southwestern US. Box 2 shows NAIP imagery from 2017 with, with box 3 showing NAIP imagery from 2021. NAIP imagery from these years shows trees changing from healthy and green in 2017 to brown and dying in 2021. In addition, box 2 and box 3 show change detection outputs from Box 1 for a location outside of Flagstaff, AZ converted to polygons (in white). The polygon in box 2 is displayed as a dashed line to serve as a reference, while the solid line in box 3 shows the measured change in 2021. Converting rasters to polygons allows the data to be easily used on tablet computers, as well as the ability to add information and photographs from field visits.

Everyone wants affordable, quality healthcare but not everyone has it. A 2021 report by the Commonwealth Fund ranked the U.S. in last place among 11 high-income countries in healthcare access.1 Carbon Health is working to change that. We are doing so by combining the best of virtual care, in-person visits, and technology to support patients with their everyday physical and mental health needs.

Rethinking how data and analytics are accessed at Carbon Health

Delivering premium healthcare for the masses that’s accessible and affordable is an ambitious undertaking. It requires a commitment to operating the business in an efficient and disciplined way. To meet our goals, our teams across the company require detailed, daily insights into operating results.

In the last year, we realized our existing BI platform was inaccessible to most of our employees outside of R&D. Creating the analytics, dashboards, and reports needed by our clinic leaders and executives required direct help from our data scientists.

However, this has all changed since deploying Looker as our new BI platform. We initially used Looker to build tables, charts, and graphs that improved how people could access and analyze data about our operating efficiency. As we continued to evaluate how our data and analytics should be experienced by our in-clinic staff, we learned about Connected Sheets for Looker, which has unlocked an entirely new way of sharing insights across the company.

A new way to deliver performance reporting and drive results

Connected Sheets for Looker gives Carbon Health employees who work in Google Sheets—practically everyone—a familiar tool for working with Looker data. For instance, one of our first outputs using the Connected Sheets integration has been a daily and weekly performance push-report for the clinic’s operating leaders, including providers.

Essentially a scorecard, the report tracks the most important KPIs for measuring clinics’ successes, including appointment volume, patient satisfaction such as net promoter score (NPS), reviews, phone call answer rates, and even metrics about billing and collections. To provide easy access, we built a workflow through Google App Script that takes our daily performance report and automatically emails a PDF to key clinic leaders each morning.

Within the first 30 days of the report’s creation, clinic leaders were able to drive noticeable improvements in operating results. For instance, actively tracking clinic volume has enabled us to manage our schedules more effectively, which in turn drives more visits and enables us to better communicate expectations with our patients. Other clinics have dramatically improved their call answer rates by tracking inbound call volume, which has also led to better patient satisfaction.

Greater accountability, greater collaboration

As you can imagine, a report that holds people accountable for outcomes in such a visible way can create some anxiety. We’ve eased those concerns by using the information constructively, with the goal to use reporting as a positive feedback mechanism to bolster open collaboration and identify operational processes that need improvement. For example, data about our call answer rates initiated an investigation that led to an operational redesign of how phones are deployed and managed at more than 120 clinics across the U.S.

Looker as a scalable solution with endless applications

We’re now rolling out Connected Sheets for Looker to deliver performance push-reporting across all teams at Carbon Health. Additionally, we continue to find new ways to leverage Connected Sheets for Looker to meet other needs of the business.

For instance, we’ve recently been able to better understand our software costs by analyzing vendor spend from our accounting systems directly in Google Sheets. Going forward, this will allow us to build a basic workflow to monitor subscription spend and employee application usage, which will lead to us saving money on unnecessary licenses and underutilized software.

We’ve come a long way in the last year. Between Looker and its integration with Google Sheets, we can meet the data needs of all our stakeholders at Carbon Health. Connected Sheets for Looker has been an impactful solution that’s going to help us drive measurable results in how we deliver premium healthcare to the masses.

^{1. Mirror, Mirror 2021: Reflecting Poorly
2.  HEALTHCARE EDITORS’ PICK Meet The Immigrant Entrepreneurs Who Raised $350 Million To Rethink U.S. Primary Care}

October 23rd (this past Saturday!) was my 4th Googlevarsery and we are wrapping an incredible Google Next 2021!

When I started in 2017, we had a dream of making BigQuery Intelligent Data Warehouse that would power every organization’s data driven digital transformation. 

This year at Next, It was amazing to see Google Cloud’s CEO, Thomas Kurian, kick off his keynote with CTO of WalMart, Suresh Kumar , talking about how his organization is giving its data the “BigQuery treatment”.

AS  I recap Next 2021 and  reflect on our amazing journey over the past 4 years, I’m so proud of the opportunity I’ve had to work with some of the world’s most innovative companies from Twitter to Walmart to Home Depot, Snap, Paypal and many others.   

So much of what we announced at Next is the result of years of hard work, persistence and commitment to delivering the best analytics experience for customers. 

I believe that one of the reasons why customers choose Google for data is because we have shown a strong alignment between our strategy and theirs and because we’ve been relentlessly delivering innovation at the speed they require. 

Unified Smart Analytics Platform 

Over the past 4 years our focus has been to build industries leading unified smart analytics platforms. BigQuery is at the heart of this vision and seamlessly integrates with all our other services. Customers can use BigQuery to query data in BigQuery Storage, Google Cloud Storage, AWS S3, Azure Blobstore, various databases like BigTable, Spanner, Cloud SQL etc. They can also use  any engine like Spark, Dataflow, Vertex AI with BigQuery. BigQuery automatically syncs all its metadata with Data Catalog and users can then run a Data Loss Prevention service to identify sensitive data and tag it. These tags can then be used to create access policies. 

In addition to Google services, all our partner products also integrate with BigQuery seamlessly. Some of the key partners highlighted at Next 21 included Data Ingestion (Fivetran, Informatica & Confluent), Data preparation (Trifacta, DBT),  Data Governance (Colibra), Data Science (Databricks, Dataiku) and BI (Tableau, PowerBI, Qlik etc).

Planet Scale analytics with BigQuery

BigQuery is an amazing platform and over the past 11 years we have continued to innovate in various aspects. Scalability has always been a huge differentiator for BigQuery. BigQuery has many customers with more than 100 petabytes of data and our largest customer is now approaching  an exabyte of data. Our large customers have run queries over trillions of rows. 

But scale for us is not just about storing or processing a lot of data. Scale is also how we can reach every organization in the world. This is the reason we launched BigQuery Sandbox which enables organizations to get started with BigQuery without a credit card. This has enabled us to reach tens of thousands of customers. Additionally to make it easy to get started with BigQuery we have built integrations with various Google tools like Firebase, Google Ads, Google Analytics 360, etc. 

Finally, to simplify adoption we now provide options for customers to choose whether they would like to pay per query, buy flat rate subscriptions or buy per second capacity. With our autoscaling capabilities we can provide customers best value by mixing flat rate subscription discounts with auto scaling with flex slots.

Intelligent Data Warehouse to empower every data analyst to become a data scientist

BigQuery ML is one of the  biggest innovations that we have brought to market over the past few years. Our vision is to make every data analyst a data scientist by democratizing Machine learning. 80% of time is spent in moving, prepping and transforming data for the ML platform. This also causes a huge data governance problem as now every data scientist has a copy of your most valuable data.  Our approach was very simple.  We asked:”what if we could bring ML to data rather than taking data to an ML engine?” 

That is how BigQuery ML was born. Simply write 2 lines of SQL code and create ML models. 

Over the past 4 years we have launched many models like regression, matrix factorization, anomaly detection, time series, XGboost, DNN etc. These  models are used by customers to solve complex  business problems simply from segmentation, recommendations, time series forecasting, package delivery estimation etc. The service is very popular: 80%+ of our top customers are using BigQueryML today.  When you consider that the average adoption rate of ML/AI is in the low 30%, 80% is a pretty good result!

We announced tighter integration of BQML with Vertex AI. Model explainability will provide the ability to explain the results of predictive ML classification and regression models by understanding how each feature contributes to the predicted result. Also users will be able to manage, compare and deploy BigQuery ML models in Vertex; leverage Vertex Pipelines to train and predict BigQuery ML models.

Real-time streaming analytics with BigQuery 

Customer expectations are changing and everyone wants everything in an instant: according to Gartner, by the end of 2024, 75% of enterprises will shift from piloting to operationalizing AI, driving a 5X increase in streaming data and analytics infrastructures.

The BigQuery’s storage engine is optimized for real-time streaming. BigQuery supports streaming ingestion of 10s of millions of events in real-time and there is no impact on query performance. Additionally customers  can use materialized views and BI Engine (which is now GA) on top of streaming data. We guarantee always fast, always fresh data. Our system automatically updates MVs and BI Engine. 

Many customers also use our PubSub service to collect real-time events and process these through Dataflow prior to ingesting into BigQuery. This is a streaming ETL pattern which is very popular. Last year,we announced PubSub Lite to  provide customers with a 90% lower price point and aTCO that is lower than any DIY Kafka deployment. 

We also announced Dataflow Prime, it is our next generation platform for Dataflow. Big Data processing platforms have only focused on horizontal scaling to optimize workloads. But we have seen new patterns and use cases like streaming AI where you may have a few steps in pipelines that perform data prep and then customers  have to run a GPU based model. Customers  want to use different sizes and shapes of machines to run these pipelines in the most optimum manner. This is exactly what Dataflow Prime does. It delivers vertical auto scaling with the right fitting for your pipelines. We believe this should lower costs for pipelines significantly.

With Datastream as our change data capture service (built on Alooma technology), we have solved the last key problem space for customers. We can automatically detect changes in your operational databases like MySQL, Postgres, Oracle etc and sync them in BigQuery.

Most importantly, all these products work seamlessly with each other through a set of templates. Our goal is to make this even more seamless over next year. 

Open Data Analytics with BigQuery

Google has always been a big believer in Open Source initiatives. Our customers love using various open source offerings like Spark, Flink, Presto, Airflow etc. With Dataproc & Composer our customers have been able to run various of these open source frameworks on GCP and leverage our scale, speed and security. Dataproc is a great service and delivers massive savings to customers moving from on-prem Hadoop environments. But customers want to focus on jobs and not clusters. 

That’s why we launched Dataproc Serverless Spark (GA) offering at Next 2021. This new service adheres to one of our key design principles we started with: make data simple.  

Just like with BigQuery, you can simply RUN QUERY. With Spark on Google Cloud, you simply RUN JOB.  ZDNet did a great piece on this.  I invite you to check it out!

Many of our customers are moving to Kubernetes and wanted to use that as the platform for Spark. Our upcoming Spark on GKE offering will give the ability to deploy spark workloads on existing Kubernetes clusters.  

But for me the most exciting capability we have is, the ability to run Spark directly on BigQuery Storage. BigQuery storage is highly optimized analytical storage. By running Spark directly on it, we again bring compute to data and avoid moving data to compute. 

BigSearch to power Log Analytics

We are bringing the power of Search to BigQuery. Customers already ingest massive amounts of log data into BigQuery and perform analytics on it. Our customers have been asking us for better support for native JSON and Search. At Next 21 we announced the upcoming availability of both these capabilities.

Fast cross column search will provide efficient indexing of structured, semi-structured and unstructured data. User friendly SQL functions let customers rapidly find data points without having to scan all the text in your table or even know which column the data resides in. 

This will be tightly integrated with native JSON, allowing customers to get BigQuery performance and storage optimizations on JSON as well as search on unstructured or constantly changing  data structures. 

Multi & Cross Cloud Analytics

Research on multi cloud adoption is unequivocal — 92% of businesses in 2021 report having a multi cloud strategy. We have always believed in providing customers choice to our customers and meeting them where they are. It was clear that all our customers wanted us to take our gems like BigQuery to other clouds as their data was distributed on different clouds. 

Additionally it was clear that customers wanted cross cloud analytics not multi-cloud solutions that can just run in different clouds. In short, see all their data with a single pane of glass, perform analysis on top of any data without worrying about where it is located, avoid egress costs and finally perform cross cloud analysis across datasets on different clouds.

With BigQuery Omni, we deliver on this vision, with a new way of analyzing data stored in multiple public clouds.  Unlike competitors, BigQuery Omni does not create silos across different clouds. BigQUery provides a single control plane that shows an analyst all data they have access to across all clouds. Analyst just writes the query and we send it to the right cloud across AWS, Azure or GCP to execute it locally. Hence no egress costs are incurred. 

We announced BQ Omni GA for both AWS and Azure at Google Next 21 and I’m really proud of the team for delivering on this vision.  Check out Vidya’s session and learn from Johnson and Johnson how they innovate in a multi-cloud world.

Geospatial Analytics with BigQuery and Earth Engine

We have partnered with our Google Geospatial team to deliver GIS functionality inside BigQuery over the years. At Next we announced that customers will be able to integrate Earth Engine with BigQuery, Google Cloud’s ML technologies, and Google Maps Platform. 

Think about all the scenarios and use-cases your team’s going to be able to enable sustainable sourcing, saving energy or understanding business risks.

We’re integrating the best of Google and Google Cloud together to – again – make it easier to work with data to create a sustainable future for our planet.  

BigQuery as a Data Exchange & Sharing Platform

BigQuery was built to be a sharing platform. Today we have 3000+ organizations sharing more than 250 petabytes of data across organizations. Google also brings more than 150 public datasets to be used across various use cases. In addition to this, we are also bringing some of the most unique datasets like Google Trends to BigQuery. This will enable organizations to understand in real-time trends and apply to their business problems.

I am super excited about the Analytics Hub Preview announcement. Analytics Hub will provide the ability for organizations to build private and public analytics exchanges. This will include data, insights, ML Models and visualizations. This is built on top of the industry leading security capabilities of BigQuery.

Breaking Data Silos

Data is distributed across various systems in the organization and making it easy to break the data silo and make all this data accessible to all is critical. I’m also particularly excited about the Migration Factory we’re building with Informatica and the work we are doing for data movement, intelligent data wrangling with players like Trifacta and FiveTran, with whom we share over 1,000 customers (and growing!).  Additionally we continue to deliver native Google service to help our customers. 

We acquired Cask in 2018 and launched our self service Data Integration service in Data Fusion. Now Fusion allows customers to create complex pipelines with just simple drag and drop. This year we focused on unlocking SAP data for our customers. We have launched various SAP connectors and accelerators to achieve this.

At GCP Next we also announced our BigQuery Migration service in preview. Many of our customers are migrating their legacy data warehouses and data lakes to BigQuery. BigQuery Migration Service provides end-to-end tools to simplify migrations for these customers. 

And today, to make migrations to BigQuery easier for even more customers, I am super excited to announce the acquisition of CompilerWorks. CompilerWorks’ Transpiler is designed from the ground up to facilitate SQL migration in the real world and will help our customers accelerate their migrations. It supports migrations from over 10 legacy enterprises data warehouses and we will be making it available as part of our BigQuery Migration service in the coming months.

Data Democratization with BigQuery

Over the past 4 years we have focused a lot on making it very  easy to derive actionable insights from data in BigQuery. Our priority has been to provide a strong ecosystem of partners that can provide you with great tools to achieve this but also deliver native Google capabilities. 

With our BI engine GA announcement which we introduced in 2019, previewed earlier this year and showcased with tools like Microsoft PowerBI and Tableau, is now available for all to play with.

BigQuery + Data Studio are like peanut butter and Jelly. They just work well together. We launched BI Engine first with Data Studio and scaled it to all the users. More than 40% of our BigQuery customers use Data Studio. Once we knew BI Engine works extremely well we now have made it an integral part of BigQuery API and launched it for all our internal and partner BI tools. 

We announced GA for BI Engine at Next 2021 but we were already GA with Data Studio for the past 2 years. We recently moved the Data Studio team back into Google Cloud making the partnership even stronger. If you have not used Data Studio, I encourage you to take a look and get started for free today here!! 

Connected Sheets for BigQuery is one of my favorite combinations. You can give every business user in your organization the ability to analyze billions of records using standard Google Sheets experience. I personally use it everyday to analyze all our product data. 

We acquired Looker in Feb 2020 with a vision of providing a semantic modeling layer to our customers with a governed BI solution. Looker is tightly integrated with BigQuery including BigQuery ML. Our latest partnership with Tableau where Tableau customers will soon be able to leverage Looker’s semantic model, enabling new levels of data governance while democratizing access to data. 

Finally, I have a dream that one day we will bring Google Assistant to your enterprise data. This is the vision of Data QnA. We are in early innings on this and we will continue to work hard to make this vision a reality. 

Intelligent Data Fabric to unify the platform

Another important trend that shaped our market is the Data Mesh.  Earlier this year, Starburst invited me to talk about this very topic. We have been working for years on this concept, and although we would love for all data to be neatly organized in one place, we know that our customers’ reality is that it is not (If you want to know more about this, read about my debate on this topic with Fivetran’s George Fraser, a16z’s Martin Casado and Databricks’ Ali Ghodsi).

Everything I’ve learned from customers over my years in this field is that they don’t just need a data catalog or a set of data quality and governance tools, they need an intelligent data fabric.  That is why we created Dataplex, whose general availability we announced at Next.

Dataplex enables customers to centrally manage, monitor, and govern data across data lakes, data warehouses, and data marts, while also ensuring data is securely accessible to a variety of analytics and data science tools.  It lets customers organize and manage data in a way that makes sense for their business, without data movement or duplication. It provides logical constructs – lakes, data zones, and assets – which enable customers to abstract away the underlying storage systems to build a foundation for setting policies around data access, security, lifecycle management, and so on.  Check out Prajakta Damle’s session and learn from Deutsche Bank how they are thinking about a unified data mesh across distributed data.

Closing Thoughts

Analysts have recognized our momentum and, as I look back at this year, I couldn’t thank our customers and partners enough for the support they provided my team and I across our large Data Analytics portfolio: in March, Google BigQuery was named a Leader in The Forrester Wave™: Cloud Data Warehouse, Q1 2021.  And in June, Dataflow was named a Leader in The Forrester Wave™: Streaming Analytics, Q2 2021 report.

If you want to get a taste for why customers choose us over other hyperscalers or cloud data warehousing, I suggest you watch the Data Journey series we’ve just launched, which documents the stories of organizations modernizing to the cloud with us.

The Google Cloud Data Analytics portfolio has become a leading force in the industry and I couldn’t be more excited to have been part of it.  I do miss you, my customers and partners, and I’m frankly bummed that we didn’t get to meet in person like we’ve done so many times before (see a photo of my last in-person talk before the pandemic), but this Google Next was extra special, so let’s dive into the product innovation and their themes.

I hope that I will get to see you in person next time we run Google Next!