3087
Of your peers have already watched this video.
20:30 Minutes
The most insightful time you'll spend today!
How Visual Inspection AI Transforms the Manufacturing Industry
The pandemic has created demand volatility and has placed lots of pressure on manufacturers. Decreased demand for new products, the disruption of retail channels, and interruptions to supply chain operations have made it very challenging for manufacturers to operate profitable businesses.
This has left manufacturers keen to decrease costs and improve work efficiency by automating many of their work processes.
And many are discovering that visual inspection utilizing AI can help.
This video, hosted by Ying Fei, Product manager, Google Cloud and Sudhindra K Ghanathe, Industry Solutions Lead of Accenutre Google Business Group, Accenture, showcases how world-leading manufacturing companies use visual inspection AI to transform their business. Customers such as Siemens share how they use Google visual inspection AI to automate quality control process, achieve cost savings, and improve work efficiency.
4 Methods How AI/ML Boosts Innovation and Reduces Costs

2929
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
“Cloud Wisdom Weekly: for tech companies and startups” is a new blog series we’re running this fall to answer common questions our tech and startup customers ask us about how to build apps faster, smarter, and cheaper. In this installment, we explore how to leverage artificial intelligence (AI) and machine learning (ML) for faster innovation and efficient operational growth.
Whether they’re trying to extract insights from data, create faster and more efficient workflows via intelligent automation, or build innovative customer experiences, leaders at today’s tech companies and startups know that proficiency in AI and ML is more important than ever.
AI and ML technologies are often expensive and time-consuming to develop, and the demand for AI and ML experts still largely outpaces the existing talent pool. These factors put pressure on tech companies and startups to allocate resources carefully when considering bringing AI/ML into their business strategy. In this article, we’ll explore four tips to help tech companies and startups accelerate innovation and reduce costs with AI and ML.
4 tips to accelerate innovation and reduces costs with AI and ML
Many of today’s most innovative companies are creating services or products that couldn’t exist without AI—but that doesn’t mean they’re building their AI and ML infrastructure and pipelines from scratch. Even for startups whose businesses don’t directly revolve around AI, injecting AI into operational processes can help manage costs as the company grows. By relying on a cloud provider for AI services, organizations can unlock opportunities to energize development, automate processes, and reduce costs.
1. Leverage pre-trained ML APIs to jumpstart product development
Tech companies and startups want their technical talent focused on proprietary projects that will make a difference to the business. This often involves the development of new applications for an AI technology, but not necessarily the development of the AI technology itself. In such scenarios, pre-trained APIs help organizations quickly and cost-effectively establish a foundation on which higher-value, more differentiated work can be layered.
For example, many companies building conversational AI into their products and services leverage Google Cloud APIs such as Speech-to-Text and Natural Language. With these APIs, developers can easily integrate capabilities like transcription, sentiment analysis, content classification, profanity filtering, speaker diarization, and more. These powerful technologies help organizations focus on creating products rather than having to build the base technologies.
See this article for examples of why tech companies and startups have chosen Google Cloud’s Speech APIs for use cases that range from deriving customer insights to giving robots empathetic personalities. For an even deeper dive, see
- our AI product page to explore other APIs, including Translation, Vision, and more;
- and the Google Cloud Skills Boost for ML APIs.
2. Use managed services to scale ML development and accelerate deployment of models to production
Pre-trained models are extremely useful, but in many cases, tech companies and startups need to create custom models to either derive insights from their own data or to apply new use cases to public data. Regardless of whether they’re building data-driven products or generating forecasting models from customer data, companies need ways to accelerate the building and deployment of models into their production environments.
A data scientist typically starts a new ML project in a notebook, experimenting with data stored on the local machine. Moving these efforts into a production environment requires additional tooling and resources, including more complicated infrastructure management. This is one reason many organizations struggle to bring models into production and burn through time and resources without moving the revenue needle.
Managed cloud platforms can help organizations transition from projects to automated experimentation at scale or the routine deployment and retraining of production models. Strong platforms offer flexible frameworks, fewer lines of code required for model training, unified environments across tools and datasets, and user-friendly infrastructure management and deployment pipelines.
At Google Cloud, we’ve seen customers with these needs embrace Vertex AI, our platform for accelerating ML development, in increasing numbers since it launched last year. Accelerating time to production by up to 80% compared to competing approaches, Vertex AI provides advanced end-to-end ML Ops capabilities so that data scientists, ML engineers, and developers can contribute to ML acceleration. It includes low-code features, like AutoML, that make it possible to train high performing models without ML expertise.
Over the first half of 2022, our performance tests found that the number of customers utilizing AI Workbench increased by 25x. It’s exciting to see the impact and value customers are gaining with Vertex AI Workbench, including seeing it help companies speed up large model training jobs by 10x and helping data science teams improve modeling precision from the 70-80% range to 98%.
If you are new to Vertex AI, check out this video series to learn how to take models from prototype to production. For deeper dives, see
- this article about Vertex AI’s role in an ambitious project to measure climate change with AI;
- BigQuery has built-in Machine Learning (ML) and Analytics that you can use to create no-code predictions using just SQL queries.
- this blog about how Vertex AI and BigQuery work together to make data analysis easier and more powerful;
- and this blog about Example-based explanations, one of our most recent updates to make model iteration more intuitive and efficient.
3. Harness the cloud to match hardware to use cases while minimizing costs and management overhead
ML infrastructure is generally expensive to build, and depending on the use case, specific hardware requirements and software integrations can make projects costly and complicated at scale. To solve for this, many tech companies and startups look to cloud services for compute and storage needs, attracted by the ability to pay only for resources they use while scaling up and down according to changing business needs.
At Google Cloud, customers share that they need the ability to optimize around a variety of infrastructure approaches for diverse ML workloads. Some use Central Processing Units (CPUs) for flexible prototyping. Others leverage our support for NVIDIA Graphics Processing Units (GPUs) for image-oriented projects and larger models, especially those with custom TensorFlow operations that must run partially on CPUs. Some choose to run on the same custom ML processors that power Google applications—Tensor Processing Units (TPUs). And many use different combinations of all of the preceding.
Beyond matching use cases to the right hardware and benefiting from the scale and operational simplicity of a managed service, tech companies and startups should explore configuration features that help further control costs. For example, Google Cloud features like time-sharing and multi-instance capabilities for GPUs — as well as features like Vertex AI Training Reduction Server — are built to optimize GPU costs and usage.
Vertex AI Workbench also integrates with the NVIDIA NGC catalog for deploying frameworks, software development kits and Jupyter Notebooks with a single click—another feature that, like Reduction Server, speaks to the ways organizations can make AI more efficient and less costly via managed services.
4. Implement AI for operations
Besides using pre-trained APIs and ML model development to develop and deliver products, startup and tech companies can improve operational efficiency, especially as they scale, by leveraging AI solutions built for specific business and operational needs, like contract processing or customer service.
Google Cloud’s DocumentAI products, for instance, apply ML to text for use cases ranging from contract lifecycle management to mortgage processing. For businesses whose customer support needs are growing, there’s Contact Center AI, which helps organizations build intelligent virtual agents, facilitate handoffs as appropriate between virtual agents and human agents, and generate insights from call center interactions. By leveraging AI to help manage operational processes, startups and tech companies can allocate more resources to innovation and growth.
Next steps toward an intelligent future
The tips in this article can help any tech company or startup find ways to save money and boost efficiency with AI and ML. You can learn more about these topics by registering for Google Cloud Next, kicking off October 11, where you’ll hear Google Cloud’s latest AI news, discussions, and perspectives—in the meantime, you can also dive into our Vertex AI quickstarts and BigQuery ML tutorials. And for the latest on our work with tech companies and startups, be sure to visit our Startups page.
An AI-Powered Cost Cutting Guide: 8 Strategies for Maximizing Profits

1709
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
We are increasingly seeing one question arise in virtually every customer conversation: How can the organization save costs and drive new revenue streams?
Everyone would love a crystal ball, but what you may not realize is that you already have one. It’s in your data. By leveraging Data Cloud and AI solutions, you can put your data to work to achieve your financial objectives. Combining your data and AI reveals opportunities for your business to reduce expenses and increase profitability, which is especially valuable in an uncertain economy.
Google Cloud customers globally are succeeding in this effort, across industries and geographies. They are improving ROI by saving money and creating new revenue streams. We have distilled the strategies and actions they are implementing—along with customer examples and tips—in our eBook, “Make Data Work for You.” In it, you’ll find ways you can pare costs, increase profitability, and monetize your data.
Find money in your data
Our Google Cloud teams have identified eight strategies that successful organizations are pursuing to trim expenses and uncover new sources of revenue through intelligent use of data and AI. These use cases range from scaling small efficiencies in logistics to accelerating document-based workflows, monetizing data, and optimizing marketing spend.

The results are impressive. They include massive cost savings and additional revenue. On-time deliveries have increased sharply at one company, and procure-to-pay processing costs have fallen by more than half at another. Other organizations have reaped big gains in ecommerce upselling and customer satisfaction.
We’ve found that businesses across every industry and around the globe are able to take action on at least one of these eight strategies. Contrary to common misperceptions, implementation does not require massive technology changes, crippling disruption to your business, or burdensome new investments.
What success looks like
If you worry your business is not ready or you need to gain buy-in from leadership, the success stories of the 15 companies in this report are helpful examples. Learning how organizations big and small, in different industries and parts of the world, have implemented these data and AI strategies makes the opportunities more tangible.
Carrefour
Among the world’s largest retailers, Carrefour operates supermarkets, ecommerce, and other store formats in more than 30 countries. To retain leadership in its markets, the company wanted to strengthen its omnichannel experience.
Carrefour moved to Google Data Cloud and developed a platform that gives its data scientists secure, structured access to a massive volume of data in minutes. This paved the way for smarter models of customer behavior and enabled a personalized recommendation engine for ecommerce services.
The company saw a 60% increase in ecommerce revenue during the pandemic, which it partly attributes to this personalization.
ATB Financial
ATB Financial, a bank in the Canadian province of Alberta, uses its data and AI to provide real-time personalized customer service, generating more than 20,000 AI-assisted conversations monthly. Machine learning models enable agents to offer clients real-time tailored advice and product suggestions.
Moreover, marketing campaigns and month-end processes that used to take five to eight hours now run in seconds, saving over CA$2.24 million a year.
Bank BRI
Bank BRI, which is owned by the Indonesian government, has 75.5 million clients. Through its use of digital technologies, the institution amasses a lot of valuable data about this large customer base.
Using Google Cloud, the bank packages this data through more than 50 monetized open APIs for more than 70 ecosystem partners who use it for credit scoring, risk management, and other applications. Fintechs, insurance companies, and financial institutions don’t have the talent or the financial resources to do quality credit scoring and fraud detection on their own, so they are turning to Bank BRI.
Early in the effort, the project generated an additional $50 million in revenue, showing how data can drive new sources of income.
How to get going now
“Make Data Work for You” will help you launch your financial resiliency initiatives by outlining the steps to get going. The process lays the groundwork for realizing your own cost savings and new revenue streams by leveraging data and AI.
Among these steps include building frameworks to operate cost efficiently, make informed decisions related to spending and optimize your data and AI budgets.

Operate: Billing that’s specific to your use-case
Control your costs by choosing data and analytics vendors who offer industry-leading data storage solutions and flexible pricing options. For example, multiple pricing options such as flat rate and pay-as-you-go allow you to optimize your spend for best price-performance.
Inform: make informed decisions based on usage
Use your cloud vendor’s dashboards or build a billing data report to gain insights on your spending over time. Make use of cost recommendations and other forecasting tools to predict what your future expenses are going to be.
Optimize: Never pay more than you use
While planning data analytics capacity, organizations often overprovision and overpay than what they actually use. Consider migrating your workloads that have unpredictable demand to a data warehousing solution that offers granular level autoscaling features so that you never have to pay for more than what you use.
There are other key moves that will set your initiative up for success including how to shorten time to value in building AI models and measuring impact. You can find details in the report.
A brighter future
The teams at Google Cloud helped the companies in “Make Data Work for You,” along with many more organizations, use their data and AI to achieve meaningful results. Download the full report to see how you can too.
Improved TabNet on Vertex AI: High-performance, scalable Tabular Deep Learning

2512
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Data scientists choose models based on various tradeoffs when solving machine learning (ML) problems that involve tabular (i.e., structured) data, the most common data type within enterprises. Among such models, decision trees are popular because they are easy to interpret, fast to train, and can obtain high accuracy quickly from small-scale datasets. On the other hand, deep neural networks offer superior accuracy on larger datasets, as well as the benefits of end-to-end learning, but are black-box and difficult to interpret. TabNet, an interpretable deep learning architecture developed by Google AI, combines the best of both worlds: it is explainable, like simpler tree-based models, and can achieve the high accuracy of complex black-box models and ensembles.
We’re excited to announce that TabNet is now available in Vertex AI Tabular Workflows! Tabular Workflows provides fully managed, optimized, and scalable pipelines, making it easier to use TabNet without worrying about implementation details, and to deploy TabNet with the MLOps capabilities of Vertex. TabNet on Vertex AI Tabular Workflows is optimized for efficient scaling to massive tabular datasets. Moreover, TabNet on Vertex AI Tabular Workflows come with machine learning improvements on top of the original TabNet, yielding better accuracy for real-world data challenges.
TabNet on Vertex AI is well-suited for a wide range of tabular data tasks where model explainability is just as important as accuracy, such as financial asset price prediction, fraud/cyberattack/crime detection, retail demand forecasting, user modeling, credit/risk scoring, diagnosis from healthcare records, and product recommendations.
Overview of Tabnet
TabNet has a specially-designed architecture (overviewed in Fig. 1), based on sequential attention, that selects which model features to reason from at each step. This mechanism makes it possible to explain how the model arrives at its predictions and the thoughtful design helps with superior accuracy. TabNet not only outperforms alternative models (including neural networks and decision trees) but also provides interpretable feature attributions. More details, including results on academic benchmarks, can be found in our AAAI 2021 paper.

Since its publication, TabNet has received significant traction from various enterprises across different industries and a variety of high-value tabular data applications (most of which include the ones for which deep learning was not even used a priori). It has been used by numerous enterprises like Microsoft, Ludwig, Ravelin, and Determined. Given high customer interest on TabNet, we’ve worked on making it available on Vertex given the real-world deep learning development and productionization needs, as well as improving its performance and efficiency.
Highlights of TabNet on Vertex AI Tabular Workflows
Scaling to Very Large Datasets
Fueled by the advances in cloud technologies like BigQuery, enterprises are increasingly collecting more tabular data, and datasets with billions of samples and hundreds/thousands of features are becoming the norm. In general, deep learning models get better learning from more data samples, and more features, with the optimal methods as they can better learn the complex patterns that drive the predictions. The computational challenges become significant though when model development on massive datasets is considered. This results in high cost or very long model development times, constituting a bottleneck for most customers to fully take advantage of their large datasets. With TabNet on Tabular Workflows, we’re making it more efficient to scale to very large tabular datasets.
Key Implementation Aspects: The TabNet architecture has unique advantages for scaling: it is composed mainly of tensor algebra operations, it utilizes very large batch sizes, and it has high compute intensity (i.e., the architecture employs a high number of operations for each data byte transmitted). These open a path to efficient distributed training on many GPUs, utilized to scale TabNet training in our improved implementation.
In TabNet on Vertex AI Tabular Workflows, we have carefully engineered the data and training pipelines to maximize hardware utilization so that users can get the best return for their Vertex AI spending. The following features enable scale with TabNet on Tabular workflows:
- Parallel data reading with multiple CPUs in a pipeline optimized to maximize GPU utilization for distributed training, reflecting best practices from Tensorflow.
- Training on multiple GPUs that can provide significant speedups on large datasets with high compute requirements. Users can specify any available machine on GCP with multiple GPUs, and the model will automatically run on them with distributed training.
- For efficient data parallelism with distributed learning, we use Tensorflow mirrored distribution strategy to support data parallelism across many GPUs. Our results demonstrate >80% utilization with several GPUs on billion-scale datasets with 100s-1000s of features.
Standard implementations of deep learning models could yield a low GPU utilization, and thus inefficient use of resources. With our implementation, TabNet on Vertex, users can get the maximal return on their compute spend on large-scale datasets.
Examples on real-world customer data: We have benchmarked the training time specifically for enterprise use cases where large datasets are being used and fast training is crucial. In one representative example, we used 1 NVIDIA_TESLA_V100 GPU to achieve state-of-the-art performance in ~1 hour on a dataset with ~5 million samples. In another example, we used 4 NVIDIA_TESLA_V100 GPUs to achieve state-of-the-art performance in ~14 hours on a dataset with ~1.4 billion samples.
Improving Accuracy given Real-World Data Challenges
Compared to its original version, TabNet on Vertex AI Tabular Workflows has improved machine learning capabilities. We have specifically focused on the common real-world tabular data challenges. One common challenge for real-world tabular data is numerical columns having skewed distributions, for which we productionized learnable preprocessing layers (e.g. including parametrized power transform families and quantile transformations) that improve the TabNet learning. Another common challenge is the high number of categories for categorical data, for which we adopted tunable high-dimensional embeddings. Another one is imbalance of label distribution, for which we added various loss function families (e.g. focal loss and differentiable AUC variants). We have observed that such additions can provide a noticeable performance boost in some cases.
Case studies with real-world customer data: We have worked with large customers to replace legacy algorithms with TabNet for a wide range of use cases, including recommendation, rankings, fraud detection, and estimated arrival time predictions. In one representative example, TabNet was stacked against a sophisticated model ensemble for a large customer. It outperformed the ensemble in most cases, leading to a nearly 10% error reduction on some of the key tasks. This is an impressive result, given that each percentage improvement on this model resulted in multi-million savings for the customer!
Out-of-the-box Explainability
In addition to high accuracy, another core benefit of TabNet is that, unlike conventional deep neural network (DNN) models such as multi-layer perceptrons, its architecture includes explainability out of the box. This new launch on Vertex Tabular Workflows makes it very convenient to visualize explanations of the trained TabNet models, so that the users can quickly gain insights on how the TabNet models arrive at its decisions. TabNet provides feature importance output via its learned masks, which indicate whether a feature is selected at a given decision step in the model. Below is the visualization of the local and global feature importance based on the mask values. The higher the value of the mask for a particular sample, the more important the corresponding feature is for that sample. Explainability of TabNet has fundamental benefits over post-hoc methods like Shapley values that are computationally-expensive to estimate, while TabNet’s explanations are readily available from the model’s intermediate layers. Furthermore, post-hoc explanations are based on approximations to nonlinear black-box functions while TabNet’s explanations are based on what the actual decision making is based on.
Explainability example: To illustrate what is achievable with this kind of explainability, Figure 2 below shows the feature importance for the Census dataset. The figure indicates that education, occupation, and number of hours per week are the most important features to predict whether a person can earn more than $50K/year (the color of corresponding columns are lighter). The explainability capability is sample-wise, which means that we can get the feature importance for each sample separately.

Benefits as a Fully-Managed Vertex Pipeline
TabNet on Vertex Tabular Workflows makes the model development and deployments tasks much simpler – without writing any code, one can obtain the trained TabNet model, deploy it in their application, and use the MLOps capabilities enabled by Vertex Managed Pipelines! Some of these benefits are highlighted as:
- Compatibility with Vertex AI ML Ops for implementing automated ML at scale including products like Vertex AI Pipelines and Vertex AI Experiments.
- Deployment convenience: Vertex AI prediction services, both in batch and online mode, are supported out-of-the-box.
- Customizable feature engineering to enable the best utilization of the domain knowledge of users.
- Using Google’s state-of-the-art search algorithms, automatic tuning to identify the best-performing hyperparameters, with automatic selection of the appropriate hyper-parameter search space based on dataset size, prediction type, and training budget.
- Tracking the deployed model and convenient evaluation tools.
- Easiness in comparative benchmarking with other models (such as AutoML and Wide & Deep Networks) as the user journey would be unified.
- Multi-region availability to better address international workloads.
More details
If you’re interested in trying TabNet on Vertex AI on your tabular datasets, please check out Tabular Workflow on Vertex AI and fill out this form.
Acknowledgements: We’d like to thank Nate Yoder, Yihe Dong, Dawei Jia, Alex Martin, Helin Wang, Henry Tappen and Tomas Pfister for their contributions to this blog.
3791
Of your peers have already watched this video.
2:30 Minutes
The most insightful time you'll spend today!
How e-Com Firm Bukalapak Achieved 5X ROAS With Machine Learning
In 2018, Indonesia accounted for 94% of SEA’s $23 billion e-commerce industry. Today, the country’s massive e-commerce sector continues to grow, along with the number of brands looking for innovative ways to compete for a piece of the pie.
As one of the largest e-commerce companies in the region, Bukalapak receives a high volume of website visitors via direct traffic, Shopping ads, Google Display Network ads, and YouTube ads.
But when the brand noticed too many potential customers were browsing its website without converting, it knew it had to reconsider its marketing strategy. In an effort to reach consumers who were more likely to buy its products, Bukalapak turned to Smart Shopping campaigns.
Experimenting with Automation
By combining standard shopping and dynamic remarketing campaigns, Smart Shopping campaigns use automated bidding and ad placement to promote products to users across Search, Display, and YouTube. The automated solution also allows brands to reach high-value users who have already seen its ads or visited its site directly but left without converting.
Always open to trying new strategies, Bukalapak launched a three-month Smart Shopping campaign focused on 5% of its Shopping ads traffic using a maximize conversion value bidding strategy. The rest of the brand’s traffic (95%) was assigned standard shopping and dynamic remarketing campaigns, and the results of these were measured against the automated alternative.
The team was able to launch the Smart Shopping campaign with little manual effort by:
- creating a separate campaign with a determined traffic split and a recommended daily budget.
- uploading the Bukalapak logo and image banner for responsive display ads.
- designating Indonesia as the country of sale.
The campaign combined the brand’s existing product feed with Google’s machine learning algorithm to serve more than 40 million products to potential customers across multiple channels — all while automating ad placement and bidding for maximum conversion value.
Smart Shopping Campaign Saves Time, Boosts ROAS
The Smart Shopping campaign achieved 5X higher ROAS than the standard shopping effort while also driving 4X more conversions and 300% growth in conversion value, leading to 2.5X more new customers.
“The automation not only allowed the team to focus less on manual campaign optimization but also helped them boost relevance among high-value users,” said Tushar Bhatia, associate vice president of growth at Bukalapak.

The impressive results encouraged Bukalapak to increase its investment in Smart Shopping campaigns by 27X over the past year. The brand plans to remain at the forefront of innovation by continually testing new products and further optimizing its campaign strategies.
How Connected-Stories Uses BigQuery and AI/ML to Craft Personalized Ad Experiences

4013
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery
In the field of producing engaging video content such as ads, many marketers ignore the power of data to improve their creative efforts to meet the consumers’ need for personalized messages. The demand for creative tech to efficiently personalize is real as marketers need personalized video Ads to reach their audience with the right message at the right time. Data, Insights and Technology are the main ingredients to deliver this value while ensuring security and privacy requirements are met. The Connected-Stories team partnered with Google Cloud to build a platform for Ad personalization. Google Data Cloud and BigQuery are at the forefront to assimilate data, leverage ML models, create personalized ads, and capitalize on real-time intelligence as the core features of the Connected-Stories NEXT platform.
Connected-Stories NEXT is an end-to-end creative management platform to develop, serve, and optimize interactive video and display ads that scale across any channel. The platform ingests first-party data to create custom ML models, measure numerous third-party data points to help brands develop unique customer journeys and create videos that their data signals can drive. An intelligent feedback loop passes real-time data back, enabling brands to make data-driven and actionable video ads that take the brand’s campaigns to the next level.

The core use case of the NEXT platform revolves around collecting user’s interaction data and optimizing for precision and speed to create an actionable Ad experience that is personalized for each user. The platform processes complex data points to create interactive data visualizations that allow for accurate analysis. The platform uses Vertex AI to access managed tools, workflows, and infrastructure to build, deploy, and scale ML models that have improved the accuracy to identify segments for further analysis.
The platform ingests 200M data events with peaks and valleys of activity. These events are processed to generate dashboards that enable users to visualize metrics based on filters in real-time. These dashboards have high performance requirements in terms of a responsive user interface under constantly changing data dimensions.
Google Cloud’s serverless stack coupled with limitless data cloud infrastructure has been the core to the NEXT platform’s data-driven innovation. The growing volume of data ingested, streamed and processed were scaled uniformly across the compute, storage and analytical layers of solution. A lean development team at Connected-Stories were able to focus all-in on the solution, while the serverless stack scaled, lowered attack service in terms of security and optimized the cost footprint through pay-as-you-go features.
BigQuery has been the backbone to support the vast amounts of data spreading over multiple geos resulting in workloads running at petabyte scale. BigQuery’s fully managed serverless architecture, real-time streaming, built-in machine learning and rich business intelligence capabilities distinguishes itself from a cloud data warehouse. It is the foundation needed to approach data and serve users in an unlimited number of ways. For an application with zero tolerance for failure, given its fully managed nature, BigQuery handles replication, recovery, data distributed optimization and management.
The platform’s requirements include the need for low maintenance, constantly ingesting and refreshing data and smart-tuning of aggregated data. These capabilities can be implemented by BigQuery’s materialized views feature. Materialized views are useful for precomputed views that regularly cache query results for better performance. These views possess the innate feature to read only the delta change from base tables and calculate the up-to-date aggregations. Materialized views impart faster outputs and consume fewer resources while reducing the cost footprint.
Some key considerations in using Google cloud and focusing on the Serverless stack include: quick onboarding to development, prototyping in short sprints and ease of preparing data in a rapidly changing environment. Typical considerations around low code / no code include data transformation, aggregation and reduced deployment time. These considerations are fulfilled through using serverless capabilities within Google Cloud such as PubSub, Cloud Storage, Cloud Run, Cloud Composer, Dataflow and BigQuery as described in the Architecture diagram below. The use of each of these components and services are described below.

- Input/Ingest: At a high-level, microservices hosted in Cloud Run collect and aggregate incoming Ads events.
- Enrichment: The output of this stage is a Pub-Sub message enriched with more attributes based on a pre-configured campaign.
- Store: a Cloud Dataflow streaming job to create text files in Cloud Storage buckets.
- Trigger: Cloud Composer triggers the spark jobs based on text files to process and group them to produce desired output as one record per impression, a logical group of events.
- Deploy: Cloud Build is then used to automate all deployments.
Thus far, all Google cloud managed services work together to ingest, store and trigger the orchestration, all of which are scalable based on configurations including autoscaling capabilities.
- Visualization: A visualization tool reads data from BigQuery to compute pre-aggregations required for each dashboard.
- Data Model Evolution considerations: Though the solution served the purpose of creating pre-aggregations, as the data model evolved by adding a column or creating a new table, it led to recreating pre-aggregations and querying the data again. Alternatively, creating aggregate tables as an extra output of current ETLs seemed like a viable option. However, this would increase the cost and complexity of jobs. A similar situation to reprocess or update aggregated tables would occur as data is updated.
Precomputed views of data that is periodically cached are critical to reach the audience with the right message at the right time.
- Performance: In order to increase the performance of the platform, we need to have regularly precomputed views of the data, cached .
- Materialized Views: Consumers of these views needed faster response times, to consume fewer resources and output only the changes in comparison to a base table. BigQuery Materialized views were used to solve this very requirement. Materialized views have been highly leveraged to optimize the design resulting in lesser maintenance and access to fresh data with high performance with a relatively low technical investment in creating and maintaining SQL code.
- Dashboards: Application dashboards pointing to the Materialized views are highly performant and provide a view into fresh data.
- Custom Reports with Vertex AI Notebooks: Vertex AI notebooks directly read data from BigQuery to produce custom reports for a subset of customers. Vertex AI has been hugely beneficial to data analysts, where an environment with pre-installed libraries simplifies the readiness to use. Vertex AI Workbench notebooks are used to share these reports within the team allowing them to work always on the cloud without having the need to download data at any time. Besides, it increases the velocity to develop and test ML models faster.
The NEXT platform has yielded benefits such as customers having the ability to create unique consumer journeys powered by AI / ML personalization triggers, using first-party data and business intelligence tools to capitalize on real-time creative intelligence, which is a dashboard to measure campaign performance for cross-functional teams to analyze the impact of Ad content experience at a granular level. All of these while ensuring controlled access to data to enrich data without moving across clouds. The NEXT platform can keep up with increased demands for agility, scalability and reliability through the underlying usage of Google Cloud.
Partnering with Google, in the context of the Google Built with BigQuery program has surfaced the differentiated value in areas of creating interactive personalized Ads by using real-time data. In addition, by sharing this data across organizations as assets, ML models have fueled higher levels of innovation. Connected-Stories plan to deepen the penetration into the entire spectrum of services offered in the AI/ML area to enhance core functionality and provide newer capabilities to the platform.
Click here to learn more about Connected-Stories NEXT Platform capabilities.
The Built with BigQuery Advantage for ISVs
Through Built with BigQuery, launched in April ‘22 as part of Google Data Cloud Summit, Google is helping tech companies like Connected-Stories co-innovate in building applications that leverage Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs. Participating companies can:
- Get started fast with a Google-funded, pre-configured sandbox.
- Accelerate product design and architecture through access to designated technical experts from the ISV Center of Excellence who can share insights from key use cases, architectural patterns, and best practices encountered in the field.
- Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.
The Google Data Cloud spectrum of products and specifically BigQuery give ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge and expanding partner ecosystem and support for multi-cloud, open source tools and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in and exercise choice.
We thank the Google Cloud and Connected-Stories team members who co-authored the blog: Connected-Stories: Luna Catini, Marketing Director, Google: Sujit Khasnis, Cloud Partner Engineering
More Relevant Stories for Your Company

Transform Your Marketing Strategy with Tinyclues and Google Cloud CDP
Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery. What are Customer Data Platforms (CDPs) and why do we need them? Today, customers utilize a wide array of devices when interacting with a brand. As an example, think

A Road to Possibilities: Google Maps Platform Website
For more than 15 years, developers have used Google Maps Platform to deliver location-based experiences to their end users and used location intelligence to optimize their businesses. Along this journey, we’ve made a variety of changes to better support our community as needs have changed and new industries and technologies

Google Cloud’s Firebase Realtime Database and BigQuery AllowsCastbox to Ramp Up Customer Experience
Demand for spoken audio content such as podcasts remains robust despite the proliferation of video services and other entertainment options for consumers. Shibin Li, Co-founder of Castbox, credits growth of the global podcast platform to the following: speed and availability, market-leading features, the proliferation of smart devices to deliver audio content,

The Secret to Accelerated ML Model Training
As an infrastructure or a data science professional, it’s more critical than ever to keep abreast of the changes taking place to the infrastructure powering machine learning. If we take a step back, we will realize that there’s been a tremendous amount of progress in machine learning in the last






