Why do we need a BigQuery anti-pattern recognition tool?

BigQuery is a serverless and cost-effective enterprise data warehouse that works across cloud environments and scales with your data. As with any large scale data-intensive platform, following best practices and avoiding inefficient anti-patterns goes a long way in terms of performance and cost savings. 

Usually SQL optimization requires a significant time investment from engineers, who must read high-complexity queries, devise a variety of approaches to improve performance and efficiency, and test several optimization techniques. The best place to start is to fix anti-patterns, since this only requires easily applicable changes and provides significant performance improvements.

To facilitate the task of identifying and fixing said anti-pattern, Google Professional Services Organization (PSO) and Global Services Delivery (GSD) have developed a BigQuery anti-pattern recognition tool. This tool automates the process of scanning SQL queries, identifying antipatterns, and providing optimization recommendations.

What is the BigQuery anti-pattern recognition tool?

The BigQuery anti-pattern recognition tool let you easily identify performance impacting anti-patterns across a large number of SQL queries in a single go. 

It utilizes ZetaSQL to parse BigQuery SQL queries into abstract syntax trees (AST) and then traverses the tree nodes to detect the presence of anti-patterns. 

The tool takes a BigQuery SQL query as an input, such as:

SELECT
 t.dim1,
 t.metric1
FROM
 `dataset.table` t
WHERE
 t.id NOT IN (
 SELECT
   id
 FROM
   `dataset.table2`)

And produces the output as:

Subquery in filter without aggregation at line 8.

It examines potential optimizations, including:

• Selecting only the necessary columns
• Handling multiple WITH-clause references 
• Addressing subqueries in filters with aggregations
• Optimizing ORDER BY queries with LIMIT
• Enhancing string comparisons
• Improving JOIN patterns
• Avoiding subquery aggregation in the WHERE clause

The solution supports reading from various sources, such as:

• Command line 
• Local files
• Cloud Storage files
• Local folders
• Cloud Storage folders
• CSV (with one query per line)
• INFORMATION_SCHEMA

Additionally, the solution provides flexibility in writing output to different destinations, including:

• Printing to the terminal
• Exporting as CSV

Writing to a BigQuery table

Using the BigQuery anti-pattern recognition tool

The BigQuery anti-pattern recognition tool is hosted on GitHub. Below are the Quick Start steps on using the tool via command line for inline queries. You can also leverage Cloud Run to deploy it as a container on cloud. 

Prerequisites

• Linux OS
• JDK 11 or above is installed
• Maven
• Docker
• gcloud CLI

Quick start: – steps

1. Clone the repo into your local machine.

git clone git@github.com:GoogleCloudPlatform/bigquery-antipattern-recognition.git

2. Build the tool image inside the `bigquery-antipattern-recognition` folder.

mvn clean package jib:dockerBuild -DskipTests

3. Run the tool for a simple inline query.

docker run \
  -i bigquery-antipattern-recognition \
  --query "SELECT * FROM \`project.dataset.table1\`"

4. Below is the output result in the command-line interface:

Additionally, the above tool can read queries from Information Schema and load the output recommendations to a BigQuery table.

Below is an example of the BigQuery anti-pattern recognition tool results exported to a BigQuery table.

Getting started

Ready to start optimizing your BigQuery queries and cutting costs? Check out the tool here and contribute to the tool via GitHub.

Have questions or feedback?

We’re actively working on new features to make the tool as useful to our customers. Use it and tell us what you think! For product feedback/technical questions, reach out to us at bq-antipattern-eng@google.com. If you’re already a BigQuery customer and would like a briefing on the tool, please reach out, we’d be happy to talk.

As the volume of data that people and businesses produce continues to grow exponentially, it goes without saying that data-driven approaches are critical for tech companies and startups across all industries. But our conversations with customers, as well as numerous industry commentaries, reiterate that managing data and extracting value from it remains difficult, especially with scale.

Numerous factors underpin the challenges, including access to and storage of data, inconsistent tools, new and evolving data sources and formats, compliance concerns, and security considerations. To help you identify and solve these challenges, we’ve created a new whitepaper, “The future of data will be unified, flexible, and accessible,” which explores many of the most common reasons our customers tell us they’re choosing Google Cloud to get the most out of their data.

For example, you might need to combine data in legacy systems with new technologies. Does this mean moving all your data to the cloud? Should it be in one cloud or distributed across several? How do you extract real value from all of this data without creating more silos?

You might also be limited to analyzing your data in batch instead of processing it in real-time, adding complexity to your architecture and necessitating expensive maintenance to combat latency. Or you might be struggling with unstructured data, with no scalable way to analyze and manage it. Again, the factors are numerous—but many of them accrue to inadequate access to data, often exacerbated by silos, and insufficient ability to process and understand it.

The modern tech stack should be a streaming stack that scales with your data, provides real-time analytics, incorporates and understands different types of data, and lets you use AI/ML to predictively derive insights and operationalize processes. These requirements mean that to effectively leverage your data assets:

• Data should be unified across your entire company, even across suppliers, partners, and platforms., eliminating organizational and technology silos.
• Unstructured data should be unlocked and leveraged in your analytics strategy.
• The technology stack should be unified and flexible enough to support use cases ranging from analysis of offline data to real-time streaming and application of ML without maintaining multiple bespoke tech stacks.
• The technology stack should be accessible on-demand, with support for different platforms, programming languages, tools, and open standards compatible with your employees’ existing skill sets.

With these requirements met, you’ll be equipped to maximize your data, whether that means discerning and adapting to changing customer expectations or understanding and optimizing how your data engineers and data scientists spend their time. In coming weeks, we’ll explore aspects of the whitepaper in additional blog posts—but if you’re ready to dive in now, and to steer your tech company or startup towards success by making your data better work for you, click here to download your copy, free of charge.

Education data can inform strategy, enhance both teaching and learning, and help us better understand customer needs. But that data is only as useful as an organization’s ability to access, analyze, and act upon it. Education Technology (EdTech) companies with a strong data warehouse management policy can help to make the most of educational data at all levels.

Creating a data team

When an EdTech doesn’t have a dedicated data team, databases will likely be poorly maintained and data can be hard to find, it can lack critical security, or fail to meet a customer’s specific requirements. A dedicated data team can prevent this and better support an organization’s data needs.  There’s a correlation between the maturity of a data team and the success of an organization’s data utilization. When considering the three primary components of team, outlook, and results, we can better understand an organization’s maturity. By using this matrix from Looker’s Analytical Maturity Playbook, you can assess the maturity of your organization’s data team. Most would agree, the goal is for your operation to move from data chaos to data-driven. But, how can you do that? Think about your data this way: the intelligence you gain from your data is not only an asset…it’s a product. When your data team views your business intelligence as a product—one that’s just as important as the services you offer—they can set the stage for creating a successful data-driven company. Mature data organizations:

• Report functionality regularly
• Constantly assess potential improvements to existing systems
• Manage content rollout to internal employees

Redivis: Making data accessible, connected & secure

Redivis is a platform that connects academic researchers with data and the tools to understand education intelligence. Their customers, who are researchers, expressed how difficult it is to find new datasets for their studies. Once they found a dataset, they often had to access and work with it before they could tell if the information would be useful for their studies. Redivis Data administrators were also concerned about data security. Redivis built their platform on top of Google Cloud’s security infrastructure, making it a highly scalable data processing environment. 

Redivis developed a transparent, tiered access system for datasets that allows researchers to request separate access to a dataset’s documentation to determine whether they can use the dataset. Detailed audit logs (supported by Google Cloud Logging) and robust application-level security controls tighten data access. To offer more insights for their researchers, they securely connected their private datasets with public datasets hosted in BigQuery allowing researchers to run queries on billions of records in seconds, not hours.

With a data team in place, it’s time to visualize where to store and organize your data. Data is an asset. You want to invest in collecting, operationalizing, and storing it properly. “Inadequate data aggregation, harmonization, and processing impedes teams from making the right data-driven decisions and pivoting when needed,” according to Jesus Trujillo Gomez, a strategic business executive at Google Cloud. You can determine how mature your warehouse is by using this matrix to help you identify where your organization falls on the spectrum (least mature on the left, most mature on the right). A company’s data warehouse can be assessed in four areas: Technology, Pipeline, Quality, and Performance. Here are some suggested steps to bring your EdTech company one step closer to a data powerhouse. 

Classcraft: Using a data-driven approach to make learning fun

Launched in 2014, learning company Classcraft is reimagining the classroom, creating better learning environments with gamification. To support their mission, their team has created a complete data platform that uses data analytics to find new ways to engage students in the classroom. Classcraft became a Google for Education Build Partner to keep up with rising demand. They expanded their footprint to reach 6 million students and educators in 165 countries. 

They also migrated to Google Cloud, building on several integrations with Google Workspace for Education, Google Classroom, and Chrome for Education, allowing them to support customers ranging from individual teachers to entire school districts.  Classcraft migrated its database and analytics infrastructure to BigQuery and Cloud Storage. “We capture about 10 million new data points every month related to school culture and student performance,” says Chief Technology Officer and VP of Product Stéphane Guillemette. “With BigQuery and other Google Cloud tools, we can quickly deliver valuable insights to educators.”

Editor’s note: PedidosYa is the market leader for online food ordering in Latin America, serving 15 markets and over 400 cities. It’s also one of the largest brands within the German multinational company Delivery Hero SE. With over 20 million app downloads, PedidosYa provides the best online delivery experience through 71,000+ online partners, including restaurants, shops, drugstores, and specialized markets. 

Having constant access to fresh customer data is a key requirement for PedidosYa to improve and innovate our customer’s experience. Our internal stakeholders also require faster insights to drive agile business decisions. Back in early 2020, PedidosYa’s leadership tasked the data team to make the impossible possible. Our team’s mission was to democratize data by providing universal and secure access while creating a comprehensive information ecosystem across PedidosYa. We also had to achieve this goal while keeping costs under control— even during the migration stage and removing operational bottlenecks. 

Challenges with legacy cloud infrastructure

PedidosYa first built its data platform on top of AWS. Our data warehouse ran on Redshift, and our data lake was in S3. We used Presto and Hue as the user interfaces for our data analysts. However, maintaining this infrastructure was a daunting task. Our legacy platform couldn’t keep up with the increasing analytics demands. For example, the data stored on S3 complemented by Presto/Hue required high operational overhead. This was because Presto and our IAM (identity access management) didn’t integrate well in our legacy ecosystem. Managing individual users and mapping IAM roles with groups and Kerberos was operationally time-consuming and costly. Further, sharding access on the S3 files was far too complicated to enable seamless ACLs (access control lists).  

There were also challenges with workload management. Our data warehouse had batch data loaded overnight. If one analyst scheduled a query to run during the overnight ETL (extract, transform, load) workload, it would disrupt the current ETL task. This could stop the entire data pipeline. We’d have to wait until data engineers intervened with a manual fix.

It was also difficult to understand whether a query error was due to performance issues or platform resource exhaustion. This lack of clarity affected our data analysts’ ability to autonomously improve querying efficiency. Data team members needed to manually inspect personal queries looking for performance issues. Also,  the current architecture was prone to a ‘tragedy of the commons’ situation; it was seen as an unlimited and free resource. As a result, it was impossible to disentangle the infrastructure from different stakeholder teams, as all had very different needs. 

The decision to modernize our data warehouse

Given the growing challenges from our legacy platform, our tech team decided to transform our analytics environment with a modern data warehouse. They required the following key criteria from their next data platform: 

• Scalability – The ability to grow with elastic infrastructure.
• Cost control – Cost management and transparency. These factors promote efficiency and ownership—both key aspects of data democratization.
• Metadata management – Intuitive data platform focusing on users’ previous SQL knowledge. Plus, being able to enrich the informational ecosystem with metadata,  to diminish data gatekeepers.
• Ease of management – The team needed to reduce operational costs with a serverless solution. Data engineers wanted to focus on their key roles rather than acting as database administrators and infrastructure engineers. The team also wanted much higher availability, and to reduce the impact of maintenance windows and vacuum/analysis.
• Data governance and access rights – With a growing employee base with varying data access requirements, the team needed a simple yet comprehensive solution to understand and track user access to data.

Migrating to Google Cloud

After exploring other alternatives, we concluded Google Cloud had an answer to each of our decision drivers. Google Cloud’s serverless, managed, and integrated data platform, coupled with its seamless integration across open-source solutions, was the perfect answer for our organization. In particular, the natural integration with Airflow as a job orchestrator and Kubernetes for flexible on-demand infrastructure was key.  

We  used Dataflow together with Pub/Sub and Cloud Functions for our data ingestion requirements, which has made our deployment process with Terraform seamless. Because we set up everything in our environment programmatically, operation time has diminished. Google Cloud reduced the deployment process from about 16 hours in our legacy platform to 4 hours.  This is partly due to the friendliness of automating the deployment (such as schema check, load test, table creation, build.) process with Terraform, Cloud Functions, Pub/Sub, Dataflow, and BigQuery on GCP. Input messages processed with Dataflow allow us to abstract and plan the schema changes according to the needs of the functional team. For example, schema changes raise an alarm, and then we can modify the raw layer table schema. By doing this, we ensure that backend modifications that we don’t control do not affect upper layers.

A key reason why we picked Google Cloud was because of its advanced cost and workload management coupled with its transparent log analytics. This information gives us a complete view into any query performance issues to make improvements on the fly. Further, we achieved a significant amount of cost savings by consolidating multiple tools to BigQuery.With BigQuery, we’ve been able to reduce our total cost per query by 5x.

This was due to a number of reasons:

• Automating pipeline deployment made it much simpler to maintain the data processing processes. 
• Analysts are conscious about what queries they’re running, resulting in running better, more optimized queries. 
• Analysts use a Data Studio dashboard to see their queries and all the associated costs. As a result, there’s a lot more transparency for each persona.

 With these changes, we can easily manage and assign costs associated with each workload with their own cost centers using specific Google Cloud projects.

Change management is always challenging. However, BigQuery is intuitive and doesn’t have a steep learning curve from Hue/Hive on SQL basics. BigQuery also allowed the team to expand its capabilities and enabled them to properly work with nested structures, avoiding unnecessary joins and improving query efficiency. Additionally, we now use Data Catalog as our unique point of truth for metadata management. This allows our team to break the data access barriers and enable federation of data across the organization. By using Airflow to orchestrate everything, we keep track of every data stream. With this information, each end user can see their regularly used data entities’ status via the dashboard. This also adds transparency to our everyday data processes.

Finally, with Google Cloud’s IAM rules applied across the different products, data sharing and access is close to a noOps experience. We have programmatically implemented access according to roles and level access within the company. This allows certain pre-validated roles to view more sensitive information. These solutions help drive a more automated data governance experience. 

Up next: Google Cloud AI/ML

The new stack based on BigQuery has created significant productivity gains. Freed from the burden of operational management, PedidosYa’s data team can now focus on adding value through data tools and products.  

• Our data engineers are better equipped to integrate constantly changing transactional and operational data.
• The dataOps team can automate the infrastructure and provide autonomy to the end user.
• Our data quality team can focus on bringing added value to data stakeholders. 
• Data scientists and data analytics can spend more time analyzing data and less time asking data gatekeepers for data access.

PedidosYa can now democratize data access with a well-governed architecture. We are still at the beginning of our journey, but we are closer to achieving our vision of building a data-driven organization. Up next: expanding our artificial intelligence and machine learning capabilities.

Tune in to Google Cloud’s Applied ML Summit on June 10th, 2021, or listen on-demand later, to learn how to apply groundbreaking machine learning technology in your projects.

It’s 2022 and nanosatellites, NFTs, and autonomous cars that deliver your pizza are in full force. In a world where people rely on simple technology to untangle complex problems, companies must deliver simple experiences to be successful in today’s landscape. For many cloud providers this means enabling tightly integrated data offerings that simplify the data delivery process without losing sight of the sophisticated needs of the modern data consumer. 

But while the name of the game is helping companies reach informed decisions from their data simpler and faster, what about the data practitioners – data analysts, data engineers, database administrators, developers, etc – who use these cloud data tools and technologies everyday? To proactively stay ahead of data cloud market trends in 2022 should data practitioners invest their time in specializing their data cloud skill sets (e.g. go deep in, say, data pipelining skills) or instead invest their time generalizing their data cloud skill sets (e.g. growing proficiencies in a mix of data analytics, databases, AI/ML, and more domains)? 

Skill deep or wide with data – that is the question

For Abdul Razack, VP, Solutions Engineering, Technology Solutions and Strategy at Google Cloud, the answer is a bit of both. 

“Data practitioners need to be broad in terms of their technology skills, but specialized with respect to the domain or domains in which they apply them. The reason why is because many things that used to be separate skill sets are now converging – like business analytics, streaming, machine learning, data pipelines, and data warehousing. Data practitioners need to be able to implement end-to-end workflows that solve specific business problems using skills from each category.”

It’s true, thousands of customers are choosing Google’s data cloud because it offers a unified and open approach to cloud that enables their practitioners to break down silos, begin and end projects without leaving the data platform, and innovate faster across their organization. 

The data practitioners who mirror Google data cloud’s frame of mind of being smart and agile across data domains in their skilling and learning will reap the benefits of solving more nuanced problems – building out internet-scale applications, fine tuning smart processes with analytics and AI, constructing data meshes that make product building simple, etc – at a larger scale than they would if they specialized in just one or two areas alone.

“Of course at the end of the day it depends on what tools a data practitioner is using to complete their workflows. There’s only so much you can learn and skills you can develop when you’re using limited tools. Growing data proficiencies across the board is made a lot easier when you’re using a data platform like BigQuery to address all these needs. BigQuery eliminates the choices you have to make – for instance you don’t have to choose between streaming data and data at rest, batch and realtime, or business intelligence and data science. This freedom gives data professionals a huge advantage when they’re building their skill sets and taking on more complex projects.” -Abdul Razack – VP, Solutions Engineering, Technology Solutions and Strategy, Google Cloud

Knowing your value is half the battle when upskilling

While some experts think technology is the limiting factor of whether or not you can even go wide or go deep in the first place, others like Google Cloud’s Head of Data and Analytics Bruno Aziza purport that it also depends on who you are, who you wish to be, and what investments your company is making to ensure you can become that person. 

“If you wish to set yourself up to be a Chief Data Officer, then you’ll want to understand how technologies fit together across your data estate first” said Aziza. “Only after you feel like you’re the go-to ‘data person’ can you then decide which part of the technology stack you want to double-down on.”

But technology isn’t everything. Aziza notes, “Make sure you focus on the  business impact that your data work provides.  You want to spend as much time as you can with your business counterparts to understand their business goals and challenges. The Harvard Business Review provides great guidance on how to succeed as a Chief Data Officer.”

Even if you don’t have your sights set on a C-suite role, both Aziza and Razack contend that the number one skill data practitioners should tackle in 2022 is actually a broad and perhaps abstract one: develop and exercise the curiosity to solve problems with a data-driven strategy. 

That is, today’s data practitioners should always be interested in educating themselves in the industry and continually upskilling in something. And their employers should also be invested in helping practitioners develop those interests, most likely through exposure to learning materials, engaging in career conversations, subsidized courses, or incentives attached to pursuing a new certification or skill.  

“Every industry is going through a digital transformation and the ability to identify what data to collect, how to prepare the data, and how to derive insights from it is critical. Therefore, the ability to find business challenges and formulate a data-driven approach to address those problems is the most important skill to have.” Abdul Razack – VP, Solutions Engineering, Technology Solutions and Strategy, Google Cloud.

Take the example of the “Data Mesh” I just wrote about in VentureBeat.  You’ll find 3 types of attitudes towards this new concept. There are Disciples who encourage continued learning only from the source – like the author of a new book or the creator of a theory. There are Distractors who tell you that new skills, trends, and technologies are fake news. And there are Distorters like vendors who will sell you one easy fix solution. But it’s the data practitioner who needs to proceed with caution when interacting with all three types  and forge their own path to discovering the truth when they’re learning and building skills. And for better or worse, this comes with trial and error, experimentation, and an eagerness to grow relative to where they began.”

Ready to start data upskilling? Start here.

For those interested in keeping up their data curiosities, check out our Data Journeys video series. Each week Bruno Aziza investigates a new authentic customer’s data journey – from migrating to cloud or  building a data platform to carrying out new data for good initiatives. Learn how they did it, their data dos and don’ts, and what’s next for them on their journey. These videos include a flavor of both specializing your data competencies and broadening your data competencies. 

For those interested in deepskilling, connect with Google’s data community at our upcoming virtual event: Latest Google Cloud data analytics innovations. Register and save your spot now to get your data questions answered live by GCP’s top data leaders and watch demos from our latest products and features including BigQuery, Dataproc, Dataplex, Dataflow, and more.

If you have any questions or need support along your learning journey – we’re here for you! Sign up to be a Google Cloud Innovator, and join the Google Cloud Data Analytics Community.