What is BigQuery? - Build What's Next

6802

Of your peers have already watched this video.

2:30 Minutes

The most insightful time you'll spend today!

Explainer

What is BigQuery?

BigQuery is Google Cloud’s enterprise data warehouse designed to help you ingest, store, analyze, and visualize big data with ease.

Organizations rely on data warehouses to aggregate data from disparate sources, process it, and make it readily available for data analysis that supports their strategic decision-making.

You can ingest data into BigQuery either through batch uploading or streaming data directly to deliver real-time insights.

As a fully-managed data warehouse, Google takes care of the infrastructure so you can focus on analyzing your data up to petabyte scale.

BigQuery supports the same Structured Query Language, or SQL, for analyzing your data, which you may be familiar with if you’ve worked with ANSI-compliant relational databases in the past.

If you’re looking to create machine learning models using your enterprise data, you can do so with BigQuery ML.

With only a few lines of SQL, you can train and execute models on your BigQuery data without needing to move it around.

When it comes time to visualize your data, BigQuery integrates with Looker, as well as several other business intelligence tools across our partner ecosystem.

Now, how do you use BigQuery?

Luckily, it’s straightforward to get up and running with BigQuery.

After creating a GCP project, you can immediately start querying public data sets, which Google Cloud hosts and makes available to all BigQuery users, or you can load your own data into BigQuery to analyze.

Interacting with BigQuery to load data, run queries, or even create ML models can be done in three different ways.

First is by using the UI and the Cloud Console. Second is by using the BigQuery command line tool. And third is by making calls to the BigQuery API, using client libraries available in several languages.

BigQuery is integrated with Google Cloud’s Identity and Access Management Service so you can securely share your data and analytical insights across the organization.

What does it cost to use BigQuery?

With BigQuery, you pay for storing and querying data and streaming inserts.

Loading and exporting data are free of charge.

Storage costs are based on the amount of data stored and have two rates based on how often the data is changing.

Query costs can be either on demand, meaning you are charged per query by the amount of data processed, or flat rate for customers who want to purchase dedicated resources.

2855

Of your peers have already watched this video.

16:00 Minutes

The most insightful time you'll spend today!

How-to

Conversational AI in Search, Maps and Online Shopping!

Did you know about 77 percent of customers are likely to make a purchase from a brand they can message with? Direct interaction with the brand to gather product information shortens buyers’ journey and personalizes it with appropriate messages. To helps businesses add speed, simplicity and convenience in brand-customers interaction, Google’s Business Messages helps add chat feature in Search, Maps or other mediums.

Watch the video to learn to integrate conversational AI based on Google’s superior AI and ML features and build interactive and connected chat automation using Google Cloud’s suite of tools and products!

Case Study

Google Cloud Modernized Wisconsin Department of Workforce Development to Process 150K+ Unemployment Claims Per Week

2989

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

Wisconsin's Department of Workforce Development had archaic unemployment insurance systems that couldn't support a deluge of claims during the COVID-19 pandemic. With Google Cloud, the agency navigated issues while serving the community better.

When confronted with challenges, resilient organizations adapt and innovate to address changing customer needs. Through its partnership with Google Cloud, the Wisconsin Department of Workforce Development exemplifies this evolutionary approach. 

 Just a few months ago, challenges created by the antiquated unemployment insurance (UI) system threatened the agency’s ability to serve constituents who lost their jobs during the pandemic. Now, and in the months ahead, that continuing evolution will transform how the DWD serves its constituents with the modernization of their UI system, which includes customer experience workflows, predictive analytics, and streamlining and expediting paper applications.

Like many workforce development agencies nationwide, last year, DWD experienced a deluge of claims filed in response to the COVID-19 economic fallout. Unfortunately, DWD’s legacy UI infrastructure — largely written in the COBOL computing language and hosted on a mainframe server – was no match for the volume and complexity of cases. Wisconsin’s multiple technology systems required a large amount of manual processing, with staff using spreadsheets to manually calculate benefit adjustments. As a result, the state just couldn’t keep up with the surge of claims, and it needed to pivot quickly to a new solution to keep up with demand. 

DWD initially responded by staffing up call centers, hiring UI application adjudicators, and deploying other personnel–all told, hiring, contracting with, or reassigning more than 1,300 individuals. However, while this enabled DWD to respond to approximately 7 million calls per month, the massive number of incoming claims surged to create a  backlog of more than 750,000 claims. Legacy technology issues continued to significantly slow the query-response time. 

DWD leaders recognized that staffing alone was not the solution and that innovation was needed to overcome past, inadequate IT investment. To address the inherited issues, Wisconsin turned to Google Cloud. Working together, we were able to scale the state’s response to claims and speed up overall response time. We were also successful in screening out fraudulent claims so that the UI program could be administered–with integrity–to Wisconsinites who needed financial assistance.   

Year-to-date, Wisconsin has now disbursed $2 billion in unemployment benefits, in addition to successfully clearing its entire 2020 UI backlog. As a result of our partnership, the state is now processing an average of 157,000 claims each week and releasing most payments to citizens within two to three business days. Before the new system was in place, the response time could be weeks or even months.

Here is how Wisconsin and Google Cloud are modernizing the state’s current legacy system through a modular approach:

  • Artificial intelligence (AI)/ machine learning (ML) for predictive analytics: Through the use of Google AI/ML, the state is creating predictive analytics based on historical data to shorten adjudication decision-making for UI claims, enabling it to release payments to eligible claimants faster. Comprehensive data models and confidence scores analyze the backlog data to determine the shortest route to approval and payments, with a high level of confidence and accuracy. DWD is using Google Cloud technology to identify where in the process the claimant was getting stuck in one of the “hold buckets” for processing a UI claim. Using Google Cloud’s data and AI/ML tools, Wisconsin is able to identify problem areas and quickly resolve those issues. This informed DWD’s rewrite of the UI claim application process. And it also helped DWD identify fraudulent claims.
  • Document AI (DocAI) for streamlining paper applications: DWD is also partnering with Google Cloud to streamline paper applications and fax documents as part of UI claims processing–enabling documents to be submitted online instead of by fax or hard-copy mail. Our DocAI solution helps Wisconsin staff make faster decisions by rapidly extracting critical data from documents, saving time, and removing manual processes, which allows employees to focus on high-priority activities. 

These modernization steps in Wisconsin are resulting in a bold new vision for state unemployment systems across the United States. Through a combination of design thinking, deep partnership with state officials, and modern technology, DWD’s solutions are tailored to maximize benefits to the constituents they’re designed to serve. By joining forces with Google Cloud and utilizing modernizing technology to make informed, data-driven decisions, Wisconsin DWD is helping residents have a better experience that is easier to understand and navigate–all while better serving the community overall.

Blog

Incorporating Custom Holidays into Your Time-Series Models with BigQuery ML

1117

Of your peers have already read this article.

4:30 Minutes

The most insightful time you'll spend today!

Explore how JCB disrupted conventional organizational structures, embraced SRE, and leveraged Google Cloud products to achieve a transformative digital journey, manifesting the spirit of its 'Dejima' concept. Know more...

About three years ago, JCB, one of the biggest Japanese payment companies, launched a project to develop new high-value services with agility. We set up a policy of starting small from scratch without using the existing system, which we call the concept of “Dejima”, where we focused on improving various aspects such as team structure, risk management, and application and platform development process.

Until now, large Japanese enterprises have built decision-making systems focused on eliminating unnecessary business processes and efficiently increasing quarterly profits. As a result, we are seeing more organizational structures that make it difficult to take on new challenges or experiments with trial and error. We wanted to breathe a new life into this situation, and that is how the concept of Dejima came up. In the Edo period, Japan closed its national border to other countries under its national isolation policy. At the time, Dejima was the only area where special rules were applied and allowed people from different cultures to come and go, and trade. This special rule generated the culture of inclusion and led to Dejima’s prosperity. Like Dejima, we believe that creating an organization that is independent from other business practices can be effective in enabling digital transformation for the organization. 

We have been able to make this transformation with the direct help of the Google Cloud and its products such as Google Kubernetes Engine (GKE), Cloud Spanner and Anthos Service Mesh, applying domain-driven design and microservice architecture. We named this the “JCB Digital Enablement Platform (JDEP),” which now hosts multiple business critical production services.

A key benefit of GKE is that the team can easily add resources and release them when they are finished, allowing them to be flexible to accommodate busy periods and off-seasons. Meanwhile, Anthos Service Mesh helps us manage complex environments easily. With containerization and managed services, we are prepared for the future for when more services go into production, as it would be easy to maintain and provide version upgrade support. At the same time, Cloud Spanner ensures that we maintain a 99.99% availability at all times.

Our initial motivation for introducing SRE practices was to break proverbial walls between business, development and operations, which was a success. Now we are focused on ensuring its reliability and maintaining customer satisfaction with our SRE practices.

To ensure the success of SRE practices that we implemented, there were a few categories we needed to address, from defining the organizational culture and practices to ensuring the policies attached to the new models created were practical enough to be implemented on the ground level. This is so that the Dejima concept remains sustainable for the long run.  

Instilling a culture of measurement 

Here, “appropriate” reliability is the key. According to the conventional way of thinking at JCB, “service failure must not occur” and “SLA should be maintained as high as possible.” We started by discussing what was the “appropriate reliability” that our customers really needed, but it was not as easy as we thought because the level of reliability for user satisfaction differed from application to application. 

Eventually, the business, development and operations teams formulated specific SLIs and SLOs together, something we would never have been able to do if we discussed separately. This is because the business is required to compromise on lower service levels, since our reliability standard used to be too high. The collaboration of development and operations teams is necessary to understand how our system works upon our users’ interactions.

After Google Cloud helped us run a series of workshops where all teams participated, we saw change within the organization. The business team started evangelizing SRE to other members in the business department, and the development and operations teams started collaborating autonomously. We felt like we were working at Google speed, accomplishing so much in a short amount of time. 

Understanding SRE as an entire company is necessary to progress. We are now working on creating internal training materials to spread the SRE concept throughout the company.

Eliminating ambiguity

With the cooperation of Google Cloud, we have created a Team Charter that defines the team’s mission, values and engagement models. We also created policy documents that include Incident Response Policy, Postmortem Policy, On-call Policy, Toil Policy and Error Budget Policy, to eliminate ambiguity in day-to-day operations.

For example, when an incident occurs, we can identify exactly the level of importance, the roles that are assigned to each person, and in what order they need to follow. When to do a postmortem, who owns it? What to do if the error budget is exhausted? How do other teams reach out to SRE when they have problems? The written policy documents will dramatically improve efficiency and motivate teams to adopt a culture of learning from failures.

The format for such policies are written in Google’s SRE book, but when we adopt it, it needs to take into account the circumstances specific to our company. Simply copying an existing policy won’t work, which is why it’s important to formulate a policy that fits the situation each team is in.

Reformalizing teams

Based on these policies, JCB’s SRE team has two sub-teams. One is called Sheriff which works as the platform SRE, and provides infrastructure services for the application team. The other is called the Diplomat which works as the embedded SRE, and participates in the application team to lead productionisation. There is also a team called Architecture that is separate from the SRE teams whose role is to consult SRE on system design and review architecture.

https://storage.googleapis.com/gweb-cloudblog-publish/images/JCB.max-2000x2000.jpg

The SRE team was a single role when it was first launched, but now has two sub-teams. This is because as the number of application teams increases, the number of support tasks for the teams also increases, which can result in a shortage of resources to work on overall improvement. Securing people who are not interrupted from day-to-day support tasks and focusing on the main task improves efficiency.

Whereas both sub-teams share the on-call duty, some engineers are not allowed to do it by contract as they are not allowed to get paged. For those who cannot participate in on-call duties, we created what’s called a Toil Shift, which allows them to focus on resolving tickets in our backlogs instead.

This works well so far, but we will keep evolving as our business grows.

3553

Of your peers have already watched this video.

44:30 Minutes

The most insightful time you'll spend today!

Case Study

Bra Fit or Brad Pitt? Fun Moments of Using AI for Contact Centers

For many enterprises, the wish to tap into the power of AI is negated only by a lack of know-how: How AI works, what sort and how much data they will need to gather and structure correctly, and how to use the platforms that make AI adoption easier.

Google understands that. Which is why the AI solutions they have built “are basically plug and play, ready to go with our partners, so that you (businesses) can have immediate business value for your specific use case, and for your specific workflow without a deep investment of any kind into machine learning,” says Levent Besik, Group Product Manager, Google Cloud.

Among the more interesting use cases that’s seeing adoption is contact center AI.

“So one thing that we see quite often, especially from our B2C customers, is that they often have this growing pain in their call centers. They face a trade-off between operational efficiency and great customer service. With the advances in language and conversational AI, that doesn’t have to be the case anymore,” says Besik.

That’s exactly the problem in front of Akash Parmar, Enterprise Architect,. “One of the big challenge we had was: how do we effectively manage 14 million calls, which come into our contact center, stores and head office? In the past, these calls were managed by completely different platforms with their own IVRs, with their own routing and reporting solutions. It was expensive, plus the experience across them was very inconsistent.”

Digging deeper, Parmar and team figured that the challenge was in the fact that an IVR couldn’t really capture the hundreds of reasons customers call.

To get around the problem Parmar and team decided to let customer tell them why they were calling. They then used AI to decipher what the customer was saying, extract the customer’s intent from that, and then help them directly, or re-route them to the best department.

Overall, it proved to be a big success, says Parmer. Although there were funny moments.

“In our early days, we were getting some transcriptions from the Speech API, which were not what we expected. It was not word by word and we had to kind of train the model to make sense of it. For example, we started to get calls about Brad Pitt. We were like, we’ve really won the Oscar here. Brad Pitt is calling us. It was not actually call for Brad Pitt. These were calls about bra fit, which it’s a big business for M&S,” remembers Parmer.

Find out how three companies, including Marks and Spencer, are using AI today.

Case Study

The Power of Personalization: Ocado Retail’s Strategy to Boost Revenue and Lower Churn

1716

Of your peers have already read this article.

3:30 Minutes

The most insightful time you'll spend today!

Discover how Ocado Retail leveraged the power of customer personalization to significantly grow revenue and reduce churn, creating lasting loyalty and improved customer experience. Learn more!

Retailers are becoming more skilled at making individual customers feel heard and valued. This is a necessity given the fact that 66% of respondents to a McKinsey survey stated that they expect email marketing messages to be tailored to their needs. While marketing personalization expertise is growing, it’s still difficult to manage, especially at scale.

Ocado Retail, one of the world’s largest dedicated online grocery retailers, delivering to over 645,000 customers in the U.K., has made personalization integral to its success. 

Let’s look at how Ocado retail worked with Google Cloud and partner Cognizant to develop a new data platform to power its personalization efforts from the ground up. 

Unifying data for more holistic, powerful personalization

To achieve its goals of personalization at scale, Ocado Retail needed a central data warehouse that could turn all forms of merchandising, advertising, business, and customer intelligence data into actionable insights. It wanted a means to accelerate customer segment identification, as well as the ideation and launch of relevant campaigns. 

“We standardized on Google Cloud, including BigQuery, as the foundation for our data platform because we knew it was the right solution for now and the future,” says Kieren Johnson, Head of IT at Ocado Retail. “We have an incredibly lean team and we needed a partner with exceptional expertise to help build an ambitious enterprise data warehouse to provide powerful insights. Cognizant was also the clear choice to help us get there.”

Cognizant worked closely with Ocado Retail to make sure its expertise in Google Cloud and other technologies aligned with Ocado Retail’s vision to drive more advanced personalization at higher scales using machine learning. The partner helped build the foundation on BigQuery, and then incorporated other Google Cloud tools such as Cloud RunVertex AI, and Vertex AI Natural Language to provide no-ops, all-code warehousing, and analytics capabilities.

Cognizant also took advantage of the Google Cloud Partner Success Services (PSS) program to ensure best practices were being followed throughout the project. PSS provided advisory services that guided Cognizant through the highly complex process of building the new data warehouse for Ocado Retail on Google Cloud.

Building the enterprise data platform in this way allows Ocado Retail to leverage the full power of cloud-based analytics while maintaining a lean team. It also allows the company to greatly scale up its personalization efforts.

Making customers feel valued at every touch

The work Ocado Retail has done with Google Cloud and Cognizant has positioned it to make its growing customer base feel valued, understood, and supported in every interaction. Before launching the project with Cognizant and Google Cloud, Ocado Retail was only able to run a couple of campaigns per week and knew it lacked optimal insight into each campaign’s efficacy. 

“We now run 10 times the number of campaigns we used to with the help of the data platform Cognizant built on Google Cloud,” says Kieren. “We run multiple campaigns every day for different customer segments, and all of that increased activity is entirely driven by data insights. The positive impacts on our marketing and customer service performance have been clear. We’re now working to expand what we do.”

Ocado Retail has enjoyed solid growth since the new data platform went live, including a 13% rise in active customers during fiscal year 2022, and has also seen a reduction in churn. It attributed these improvements to being better able to tailor products and communications to specific customer preferences.

Throughout the project, Cognizant supported data clean up while maximizing the scalable, flexible, and future-proofed data analytics infrastructure offered by Google Cloud.

Increasing data-driven actions

Ocado Retail plans to provide more data-driven insights to its commercial suppliers through a product called Beet Insights. So far, Beet Insights offers suppliers with intelligence about how their products are performing on the shelves. The result has been improving the role data plays throughout the supply chain, from production to purchase and beyond.

“By putting real-time insights about costs, marketing spend, and supply-funded activities into the hands of our commercial team and buyers, we are better positioned to improve our profits,” says Kieren. “At the same time, building data analytics into every part of our business will allow us to build on our personalization efforts.”

From the project’s inception, Ocado Retail ensured that the platform would be scalable and dynamic. Now, it is working to feed more data sources and streams into the warehouse to accelerate time to insights. Ocado Retail believes this next step in the evolution of the platform will unlock even more opportunities to initiate high-impact programs that transform personalization and every customer interaction.

Learn more about what Google Cloud and partners like Cognizant can do for your customer intelligence and personalization.

More Relevant Stories for Your Company

Case Study

How TapClicks’ Google Cloud Migration Makes Life Easy for Marketers

Editor’s note: In this blog post we learn how TapClicks migrated to Google Cloud to offer their marketing customers a unified platform for data management, operations, insights, and analysis. TapClicks is a smart marketing cloud, powered by data, that unifies our customer’s marketing. By choosing to migrate our core applications last

Research Reports

Looking for a Cloud Data Warehouse? Find out Why Forrester Thinks Google BigQuery is a Leader

We are thrilled to announce that Google has been named a Leader in The Forrester Wave™: Cloud Data Warehouse, Q1 2021 report. For more than a decade, BigQuery, our petabyte-scale cloud data warehouse, has been in a class of its own. We're excited to share this recognition and we want to thank

Case Study

IKEA’s AI-driven Personalized and Real-time Recommendations Up its Conversion Rates and Average Order Value

Background At IKEA we have multiple places in our customer journey in various channels where different kinds of personalization can deliver a superior customer experience. Product recommendations in the shopping basket, content recommendations in editorial sections, inspirational recommendations on product pages and more. After a while in the broader “recommendations”

Case Study

Vodafone Leverages Google Cloud to Aid COVID-19 Frontline with Anonymized Insights on Population Mobility

Editor’s note: When Europe’s largest mobile communications company, Vodafone, was asked by the European Commission to help understand population movement across the European Union and the UK to help fight COVID-19, it was able to provide anonymized mobile network-based insights to answer the call. Here’s how Vodafone, with the support

SHOW MORE STORIES