What is BigQuery? - Build What's Next

6805

Of your peers have already watched this video.

2:30 Minutes

The most insightful time you'll spend today!

Explainer

What is BigQuery?

BigQuery is Google Cloud’s enterprise data warehouse designed to help you ingest, store, analyze, and visualize big data with ease.

Organizations rely on data warehouses to aggregate data from disparate sources, process it, and make it readily available for data analysis that supports their strategic decision-making.

You can ingest data into BigQuery either through batch uploading or streaming data directly to deliver real-time insights.

As a fully-managed data warehouse, Google takes care of the infrastructure so you can focus on analyzing your data up to petabyte scale.

BigQuery supports the same Structured Query Language, or SQL, for analyzing your data, which you may be familiar with if you’ve worked with ANSI-compliant relational databases in the past.

If you’re looking to create machine learning models using your enterprise data, you can do so with BigQuery ML.

With only a few lines of SQL, you can train and execute models on your BigQuery data without needing to move it around.

When it comes time to visualize your data, BigQuery integrates with Looker, as well as several other business intelligence tools across our partner ecosystem.

Now, how do you use BigQuery?

Luckily, it’s straightforward to get up and running with BigQuery.

After creating a GCP project, you can immediately start querying public data sets, which Google Cloud hosts and makes available to all BigQuery users, or you can load your own data into BigQuery to analyze.

Interacting with BigQuery to load data, run queries, or even create ML models can be done in three different ways.

First is by using the UI and the Cloud Console. Second is by using the BigQuery command line tool. And third is by making calls to the BigQuery API, using client libraries available in several languages.

BigQuery is integrated with Google Cloud’s Identity and Access Management Service so you can securely share your data and analytical insights across the organization.

What does it cost to use BigQuery?

With BigQuery, you pay for storing and querying data and streaming inserts.

Loading and exporting data are free of charge.

Storage costs are based on the amount of data stored and have two rates based on how often the data is changing.

Query costs can be either on demand, meaning you are charged per query by the amount of data processed, or flat rate for customers who want to purchase dedicated resources.

Blog

Fairygoodboss and Google Cloud Tied to Advancing Diversity and Female Leadership at Workplaces

5123

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

Fairygodboss, the largest career community for women, relies on Google Cloud to process, store, and analyse data based on the user-driven events on their site. Learn how this collab allows for better job recommendations and opportunities for women.

May is Asian American Pacific Islander Heritage Month — a time for us to come together to celebrate and remember the important people and history of Asian and Pacific Island heritage. This feature highlights founder Georgene Huang and how her startup Fairygodboss uses easy to use Google Cloud tools to grow and innovate. 

Coming from a male-dominated industry like Wall Street, I never thought much about my gender. I was so focused on work that I didn’t feel like I could focus on my identity as a woman of color. That all changed when I found myself thrust into the job market while two months pregnant.

Traditionally, job-seeking sites are built by – and geared towards – men and their professional needs. There was no recruiting site that broke down what candidates who identify as women might expect from a company, let alone women of color like myself. I founded Fairygodboss to create a space where women can crowdsource information about opportunities and employers in order to make informed career decisions. 

Fairygodboss 

Fair•y•god•boss (noun): A person who elevates women at work.

Fairygodboss is the largest career community for women. In researching potential employers, I had many questions surrounding company culture, maternity leave, and women in leadership positions. While sites like Glassdoor provide high-level details for job seekers, I wanted more concrete examples and data than general salary information and anecdotes about work-life balance. 

Fairygodboss began as a job review site geared towards women, but has expanded significantly since our initial launch. In addition to providing free resources like career connections, job listings, virtual recruiting events, community advice, and real research on how companies treat women, we also proactively help companies improve gender diversity. Fairygodboss partners with hundreds of top employers to create a 360-degree recruiting and employer branding program designed to drive female applicants to job postings by showcasing a company’s track record of women in leadership within a company, and their diversity investments. 

FGB

Google Cloud + Fairygodboss: Data defined 

At our core, Fairygodboss is a data-driven company, and we rely on Google Cloud to help us process, store, and analyze user-driven events occurring on the site. As a highly scalable, serverless, and cost-effective data warehousing solution, BigQuery collect millions of events each day, helping us identify where traffic is coming from (and where it goes next), and it produces actionable insights to provide better recommendations, information, and opportunities for women visiting our platform. 

After seamlessly connecting our BigQuery data to Data Studio, we have been able to visualize and make sense of all the data we receive on our platform. Unlike some other data visualization tools, Data Studio allows us to analyze data in real time. With tools like BigQuery, we are now able to identify trends and gain a deep understanding for how our users interact with our site. 

Google for Startups Accelerator: Women Founders 

In order to provide more accurate job recommendations for our users, the Fairygodboss team wanted to build a machine learning prototype to classify our users and jobs into categories to facilitate more effective matching. So we applied for the Google for Startups Accelerator: Women Founders, a three-month digital accelerator program for high-potential Seed to Series A tech startups based in the U.S. and Canada. Along with tailored mentorship and product support from Googlers and industry experts, the Google for Startups Accelerator provided $100K in Google Cloud credits to scale our businesses. The mentorship we received from the Google Cloud technical experts as part of the Accelerator – special shout out to Peter Novig! – empowered us to integrate Cloud AutoML into our systems and ultimately curate more accurate job suggestions for our users.  

Not only did the Google for Startups Accelerator program help to achieve our business goals for the year, it was also extremely beneficial to be connected with a cohort of other women founders of color. While we are all building different kinds of businesses across different industries, the guidance around fundraising, scaling teams, and coping with the struggles of being a founder rang true for all of us. 

Advancing the community

It can be daunting to launch your own business – and even more so as an AAPI woman. Throughout my career, I am certain I have experienced unconscious bias around my race. Specifically, there have likely been “model minority” stereotypes about my demeanor and math abilities, assumptions that I may be mild-mannered or agreeable. While frustrating, these unfair tropes actually inspired me to see myself beyond others’ perceptions. Why be either analytical or creative, meek or brash, inspiring or agreeable? It’s the ors of life that prevent us from seeing ourselves as truly multi-dimensional individuals. I aspired to be more instead of or, and want my company to be as well. 

As an inclusion-focused business, it is extremely important that Fairygodboss mindfully engage with all underrepresented groups. Diversity can be sliced in many different ways, and these intersections lead to great opportunities for change. We have a team of volunteer ‘culture wizards’ at work that provide educational resources and videos for those wanting to learn more about different cultures and identities from members of under-represented groups. We also practice what we preach at Fairygodboss by prioritizing diverse slates and diversity in our own hiring process. We aim for the talent we hire to be representative of the diverse communities we support. It is imperative to take an intentional approach to hiring diverse talent, as it falls on all employers to promote equity and inclusion—there are always ways in which we can all improve. Fairygodboss looks forward to evolving with Google Cloud in 2021 and beyond as we work together to foster inclusive teams around the world. 

If you want to learn more about how Google Cloud can help your startup, visit our Startup Program application page here and sign up for our monthly startup newsletter to get a peek at our community activities, digital events, special offers, and more.

3010

Of your peers have already watched this video.

25:00 Minutes

The most insightful time you'll spend today!

Explainer

Data Catalog for Data Discovery and Metadata Management

Data discovery and metadata management is a common pain point for most enterprises. Data Catalog is a managed service that addresses this pain point with a scalable and performant solution.

“Data catalog, simply put, enables users to discover, understand, and manage all their data assets,” says Shekhar Bapat, Product Manager, Google Cloud.

Data Catalog also offers integration with DLP for auto detection and tagging of sensitive data and facilitates better data governance. Data Catalog provides support for non-Google Cloud data assets and enables customers to discover, understand, and manage ALL their data.

The session is co-presented with Shruti Thaker, Head of Alt Vendor Data and Alpha Capture, BlackRock, which has invested in Data Catalog. Thaker will share how BlackRock is using Data Catalog to create an effective metadata solution for BlackRock data assets.

Blog

An Expert’s Opinion on What Early-stage Startups Must Know

6452

Of your peers have already read this article.

3:00 Minutes

The most insightful time you'll spend today!

Many start-ups and businesses are launching on Google Cloud. To scale business and leverage Google Cloud's technology, our analytics and AI expert shares data points across selecting tech stack, customer interactions, product launches and more.

As lead for analytics and AI solutions at Google Cloud, my team works with startups building on Google Cloud. This puts us in the fortunate position to learn from founders and engineers about how early-stage startups’ investments can either constrain them or position them for success, even at the seed level. In this post, I want to share a few of the best practices to keep in mind as you’re building. 

Understand your value proposition before diving into a technology stack

If you’re launching a startup in the cloud, you’re no doubt thinking about a technology stack, but it’s important to step back a bit and think carefully about the major value proposition that your startup offers to your customers. That value proposition is going to fundamentally drive the kind of technology that you should pick.

For example, does your system need processing in real time, or can it be done in a batch mode? Can you rely on once-a-day insights or do the insights have to come in as events happen?

Additionally, what kind of latency will your customers face? That latency makes your value proposition either usable or unusable. Early on in Google’s development, leaders realized that no one was going to wait more than a few hundred milliseconds for a web page to show them their results, and that realization drove the technology decisions that have allowed Google to scale from being a startup in a garage to being a trillion dollar company. Your startup needs to define its value to customers with this level of specificity before it can build a technology stack suited to its needs. 

Focus on customer interactions

A few companies have gracefully pulled off big IT pivots that reshaped their value proposition. Netflix, for example, moved from mostly sending DVDs through the mail to becoming a streaming service and major content producer. That’s a huge shift in the user experience and the technology stack necessary to support it, even if the underlying value proposition (i.e., get content to customers) was broadly the same. But it’s also an outlier. If you’re planning for potential changes of this magnitude, rather than focused on getting your value proposition to users, you probably need to sharpen what that value proposition is.

Specifically, you need a clear vision of how customers will access and interact with your business. Typically, they’ll do so over a website or a mobile app, but there are still so many variables. 

Are customers going to transmit documents? If so, in what format? Is handwriting supported or is input limited to typing? Can they use images for optical character recognition? Will it mostly be forms? Will the data be structured or unstructured? If all that sounds  a little overwhelming, don’t worry, it’ll seem simpler by the end of this article—but also be aware: we’re just getting warmed up.

Imagine that most of your customers will access your business via voice, so you know you’ll want to prioritize conversational workflows. That’s a start—but dig deeper.  Even if we suppose you’re usingDialogflow, a Google Cloud conversational AI platform that lets you build and deploy virtual agents, we’re still not really seeing the value proposition.  How will all this work, from the beginning of a typical full customer interaction to the resolution? How many interactions will have to be facilitated over low-bandwidth connections, for example? When it comes to user interactions, make sure you can see an end-to-end use case.

Another example: you’re building a retail website, and one of your end-to-end use cases involves the customer asking if a certain amount of a given product is in stock, whether it’s one unit of the product, ten or hundreds. If the product is not sufficiently stocked, you want your app to offer similar items that are. Will your technology stack support this end-to-end use case?

These considerations are not an argument for premature optimization. There’s value in moving fast, getting minimum viable products to users, and then iterating. But in the early stages, you only get one chance to start on the right foot—and how you navigate that chance will influence a lot of dollars and effort down the road. You need to make sure you have business use cases, not just an idea, before you can start designing a technology stack.  

Here’s how to get in the right frame of mind. Pick three use cases: two that are “bread and butter” and one that is technologically complex.  Make sure your proposed technology stack can support all three, end to end. 

Default toward higher levels of abstraction

Now that we’re in the right frame of mind, we’re ready to think about the technology stack more directly. 

As a startup, you’ll need to conserve resources, and to do that, you’ll want to build at the highest level of abstraction possible for your value proposition. For example, you probably don’t want your people setting up clusters. You don’t want them configuring things if they can use a fully managed service. You want them focused on building your prototype, not managing infrastructure.

1 Canonical Data Stack on Google Cloud.jpg
Canonical Data Stack on Google Cloud

This focus has definitely informed how we create products at Google Cloud, as our canonical data stack—Pub/Sub, Dataflow, BigQuery, and Vertex AI—consists of auto-scaling and serverless products.

But management of infrastructure is not the only place where you should err toward a less-is-more philosophy. 

When it comes to architecture, choose no-code over low-code and low-code over writing custom code. For example, rather than writing ETL pipelines to transform the data you need before you land it into BigQuery, you could use pre-built connectors to directly land the raw data into BigQuery. That’s no code right there. Then, transform the data into the form you need using SQL views directly in the data warehouse. This is called ELT, and it is low code. You will be a lot more agile if you choose an ELT approach over an ETL approach. 

Another place is when you choose your ML modeling framework. Don’t start with custom TensorFlow models. Start with AutoML. That’s no-code. You can invoke AutoML directly from BigQuery, avoiding the need to build complex data and ML pipelines. If necessary, move on to pre-built models from TensorFlow Hub, HuggingFace, etc. That’s low-code. Build your own custom ML models only as a last resort.

2 No-code, low-code Data Stack on Google Cloud.jpg
No-code, low-code Data Stack on Google Cloud

Focus on getting your vision to market, not chasing technology hype  

The goal is to pick the right technology stack for bringing your vision to market, generating value for customers, conserving resources, and maintaining flexibility for growth. Early IT investments should usually gravitate toward things that preserve flexibility, such as managed services built on standard protocols or open APIs, but they needn’t always rush to the flashiest technologies.  The answer isn’t always ML, for example. The answer might be heuristics to start, with a path to ML once you have collected enough data. You want to make sure that your intelligence layer has enough abstraction so you can mark it up with simple rules at first, but then replace it with a more robust system as you go along. 

Launch and iterate fast with these principles 

The preceding discussion is a reminder that your most expensive resource is your people—and that you really want them to be focused on building your prototype, minimum viable product or production app  You want to launch fast and iterate fast, and the only way you can do that is by focusing on the things that differentiate you. 

But regardless of the technologies you use, the bottom line is the same: follow these four principles. 

  • Figure out your major value proposition and design your tech stack around it. 
  • Be very careful about user interactions. User experience is super important; you need to make sure you deliver the kind of experience that your customers have grown to expect.
  • When you’re building, pick the highest possible level of abstraction possible—the most fully managed tools and no-code/low-code frameworks that give you the functionality that you need. 
  • Instead of choosing new or flashy technologies, consider if you can build a “good enough” minimum viable product quickly and come back to a better implementation later. 

To learn more about why startups are choosing Google Cloud, click here.

Case Study

Your DW Need Scaling Up? Try What This Company Did: It Can Run 25,000 Events a Second

5064

Of your peers have already read this article.

7:30 Minutes

The most insightful time you'll spend today!

“We were close to the limits of our internal data warehouse, scalability-wise. We didn’t want to get to the point where we’d have to delete data or cancel projects,” says Levente Otti, Head of Data, Emarsys, a digital marketing platform.

With access to more data than ever before, companies have never been better positioned to adopt precision marketing methods and target the right customers at the right time. Emarsys, a digital marketing platform, enables its clients to collect, analyze, and act on a wide variety of data. From websites to mobile apps to emails, Emarsys’ customers can handle data from all its digital channels on a single, easy-to-use platform. Emarsys also makes sure that customers receive the highest quality data possible, making for smarter decisions and better business practices.

“We were close to the limits of our internal data warehouse, scalability-wise. We didn’t want to get to the point where we’d have to delete data or cancel projects. In Google Cloud we saw a platform that could scale with our ambitions and be optimized for AI and real-time solutions.”

Levente Otti, Head of Data, Emarsys

Since launching as an email solutions provider in 2000, Emarsys has grown into the world’s largest independent digital marketing platform, with more than 2,500 clients worldwide and reaching more than 1.4 billion people. By 2016, the company felt that its existing data warehouse platform was close to its limits, affecting not just day-to-day operations but also important strategic goals.

“We were close to the limits of our internal data warehouse, scalability-wise. We didn’t want to get to the point where we’d have to delete data or cancel projects,” says Levente Otti, Head of Data at Emarsys. “In Google Cloud, we saw a platform that could scale with our ambitions and be optimized for AI and real-time solutions.”

Minimal maintenance, unlimited scale with Google Cloud

Digital marketing is a highly competitive environment. Emarsys works alongside big players with a huge market share on the one hand and smaller, specialist companies on the other. It has thrived by successfully combining the all-inclusive offerings of the former with the agility of the latter, constantly looking for ways to innovate and improve. In recent years, the company had started to feel that the ability to handle large quantities of data was no longer enough. The next challenge was speed. “We truly believe that in the future, everything will be done in real time, including data processing, analytics, and AI predictive models,” Levente says.

At the start of 2016, Emarsys’ existing data warehouse was a software-as-a-service solution running on-premises, which required hardware and software maintenance in order to keep up with the company’s growing appetite for data-heavy use cases such as prediction and analytics. The existing platform had proven its worth processing large amounts of data in batches, but its real-time capabilities were limited. Moreover, Emarsys had begun to experiment with AI technology, but found that its data warehouse couldn’t scale to accommodate some of the more resource-intensive processes, such as training the predictive models. The company decided that it needed a new, cloud-based data platform.

After evaluating some of the leading cloud providers, Emarsys chose Google Cloud for its mature AI capabilities and its ease of use. “With the other solutions, we still had to rent virtual machines and hardware and be responsible for maintenance. At the time, Google Cloud was the only provider that could take that management overhead away from us, while keeping customers accounted for on every query level,” Levente says.

To implement its new data platform, Emarsys teamed up with Google Cloud Partner Aliz. Over a series of meetings, workshops, and architecture reviews, Aliz helped Emarsys navigate the Google Cloud ecosystem to find the right products for the solution it was looking for. “Aliz really helped us set off in the right direction,” explains Levente.

With Google BigQuery, we can run queries which process terabytes of data, in seconds. We can also develop our own user-defined functions, incorporating Bayesian statistics into our predictive algorithms. That means we can take into account historical data, resulting in much more accurate predictions in a scalable way within seconds.”

Levente Otti, Head of Data, Emarsys

Emarsys’ new data platform would actually be two: one platform for batch processing data and one for real-time analysis and interactions. Firstly, a proprietary publishing component gathered all the data points from Emarsys’ various channels including the website, mobile, emails, and custom events. With Cloud Pub/Sub and Cloud Dataflow, Emarsys transported and processed the data into BigQuery, which allows for further work and reviews that take into account errors or delayed events. After this, the data was exported to the main batch processing platform, which ran on BigQuery. For the real-time analytics, Emarsys used Cloud Bigtable to access data and Cloud Dataflow to pipeline it into the real-time platform, which could communicate with AI components or interaction components via an API to deliver real-time interactions with customers.

On top of the overall data infrastructure, Emarsys built a new AI platform with Google Cloud components. Training the predictive models had been an issue in the past due to the large number of resources required, so Emarsys chose to use Google Kubernetes Engine clusters, which can scale up and down on demand, without the need for hardware configuration or management. The trained models were held securely in Cloud Storage. From here, they were integrated with BigQuery for power and flexibility, allowing Emarsys to improve not just the speed of its AI predictions but also the quality.

“With Google BigQuery, we can run queries which process terabytes of data, in seconds,” shares Levente. “We can also develop our own user-defined functions incorporating Bayesian statistics into our predictive algorithms. That means we can take into account historical data, resulting in much more accurate predictions in a scalable way within seconds.”

Real-time insight, long-term satisfaction

Google Cloud enabled Emarsys to build a scalable data and AI platform that delivers powerful, actionable insights in real time. According to Levente, the company wanted to spend less time managing overload and more time considering how it should handle data. An immediate result of the new platform has been that data is now available in a scalable way, without hardware additions and management.

“With Google Cloud, we’ve been able to build a truly real-time data platform. The norm used to be daily batch processing of data. Now, if an event happens, marketing actions can be executed within seconds, and customers can react immediately. That makes us very competitive in our market.”

Levente Otti, Head of Data, Emarsys

The clear and innovative pricing schemes of Google Cloud have also brought a new level of accountability to Emarsys’ costs in a way that wasn’t possible with its on-premises infrastructure. “Now that we only pay for what we use, we can assign costs to specific customers or queries, which has a huge impact on our pricing and product development strategies,” says Levente.

Thanks to the power of BigQuery and the scale at which it can handle data, Emarsys can now apply its analytics and AI tools to their full potential. “It’s very important to enable our clients to create the best possible experience for customers,” says Levente. At the same time, the company has cut its AI platform costs by 70% with Kubernetes while increasing scalability compared to the previous solution. The whole data platform was built to be scalable, and its first big test came during the retail peak of Black Friday, when it comfortably handled 250,000 events per second. “Perhaps the biggest impact on the business came with the real-time nature of the new platform,” says Levente.

“With Google Cloud, we’ve been able to build a truly real-time data platform,” he explains. “The norm used to be daily batch processing of data. Now, if an event happens, marketing actions can be executed within seconds, and customers can react immediately. That makes us very competitive in our market.”

Since implementing the new platform, Emarsys has continued to innovate with it and is about to release a new Real-Time Decision Framework, which will provide customers with even more real-time products and tools. The company continues to work with Aliz and Google Cloud, exploring other products such as Google BigQuery ML and TensorFlow to improve its AI processes. “We had a problem that we wanted to tackle now, and for us, Google Cloud was the best way of doing that,” says Levente. “But it was also about looking ahead. We felt that Google Cloud offered us the best way of future-proofing our platform.”

Emarsys data and AI platform
E-book

A Pre-ML Checklist That Will Save You Hours of Work

DOWNLOAD E-BOOK

4977

Of your peers have already downloaded this article

2:35 Minutes

The most insightful time you'll spend today!

One of the toughest challenges for data scientists, and big data engineers is gathering, preparing, and transforming data, and creating data pipelines.

Gathering data for a machine learning or big data initiative can be hard work. The mere act of getting it all together can leave data teams so excited that they can overlook critical safeguards. The result? You could have data that’s skewed, or biased, or data sets that are too small to generate accurate models.

This is why it’s important to have a pre-ML data checklist. A pre-machine-learning data checklist will ensure you have the right data sets for your models, thereby improving your chances of success.

Here are some other benefits:

You waste less time: A checklist allows you to save the time you would normally spend trying to work through your own mental checklist.

Fewer errors: A pre-ML data checklist ensures you have don’t overlook obvious mistakes, and have relevant, unbiased, and representative data.

Lower cognitive load: By using a checklist, you remove the burden of unnecessary cognitive load. This enables you to free up the brain power required for more productive tasks such as model selection and tuning.

More Relevant Stories for Your Company

Blog

NVIDIA and Google Cloud Pave the Way for Single-cell Genomic Analysis

In the past decade, the Healthcare and Life Sciences industry has enjoyed a boon in technological and scientific advancement. New insights and possibilities are revealed almost daily. At Google Cloud, driving innovation in cloud computing is in our DNA. Our team is dedicated to sharing ways Google Cloud can be

Blog

Data Cloud Skills to Pick Up in 2022: Google Experts Recommended

It’s 2022 and nanosatellites, NFTs, and autonomous cars that deliver your pizza are in full force. In a world where people rely on simple technology to untangle complex problems, companies must deliver simple experiences to be successful in today’s landscape. For many cloud providers this means enabling tightly integrated data

Blog

How Vertex AI Helps Coca-Cola Bottlers Japan Analyze Billions of Data Records

Japan is home to millions of vending machines installed on streets and in buildings, sports stadiums and other facilities. Vending machine owners and operators, including beverage manufacturers, stock these machines with different product combinations depending on location and demand. For example, they primarily display coffee and energy drinks in machines

Case Study

Bit Capital Rolls Out a Digital Financial Solution in Under 3 Months and at 2/3 the Cost

In the past few years, a series of new technologies and regulatory changes has been transforming Brazil’s financial industry. The concept of blockchain added security and agility to financial transactions. The open banking system being deployed by the country’s Central Bank (BC) allows for platform integration and data sharing -with

SHOW MORE STORIES