Data Cloud Skills to Pick Up in 2022: Google Experts Recommended
4959
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
It’s 2022 and nanosatellites, NFTs, and autonomous cars that deliver your pizza are in full force. In a world where people rely on simple technology to untangle complex problems, companies must deliver simple experiences to be successful in today’s landscape. For many cloud providers this means enabling tightly integrated data offerings that simplify the data delivery process without losing sight of the sophisticated needs of the modern data consumer.
But while the name of the game is helping companies reach informed decisions from their data simpler and faster, what about the data practitioners – data analysts, data engineers, database administrators, developers, etc – who use these cloud data tools and technologies everyday? To proactively stay ahead of data cloud market trends in 2022 should data practitioners invest their time in specializing their data cloud skill sets (e.g. go deep in, say, data pipelining skills) or instead invest their time generalizing their data cloud skill sets (e.g. growing proficiencies in a mix of data analytics, databases, AI/ML, and more domains)?
Skill deep or wide with data – that is the question
For Abdul Razack, VP, Solutions Engineering, Technology Solutions and Strategy at Google Cloud, the answer is a bit of both.
“Data practitioners need to be broad in terms of their technology skills, but specialized with respect to the domain or domains in which they apply them. The reason why is because many things that used to be separate skill sets are now converging – like business analytics, streaming, machine learning, data pipelines, and data warehousing. Data practitioners need to be able to implement end-to-end workflows that solve specific business problems using skills from each category.”
It’s true, thousands of customers are choosing Google’s data cloud because it offers a unified and open approach to cloud that enables their practitioners to break down silos, begin and end projects without leaving the data platform, and innovate faster across their organization.
The data practitioners who mirror Google data cloud’s frame of mind of being smart and agile across data domains in their skilling and learning will reap the benefits of solving more nuanced problems – building out internet-scale applications, fine tuning smart processes with analytics and AI, constructing data meshes that make product building simple, etc – at a larger scale than they would if they specialized in just one or two areas alone.
“Of course at the end of the day it depends on what tools a data practitioner is using to complete their workflows. There’s only so much you can learn and skills you can develop when you’re using limited tools. Growing data proficiencies across the board is made a lot easier when you’re using a data platform like BigQuery to address all these needs. BigQuery eliminates the choices you have to make – for instance you don’t have to choose between streaming data and data at rest, batch and realtime, or business intelligence and data science. This freedom gives data professionals a huge advantage when they’re building their skill sets and taking on more complex projects.” -Abdul Razack – VP, Solutions Engineering, Technology Solutions and Strategy, Google Cloud
Knowing your value is half the battle when upskilling
While some experts think technology is the limiting factor of whether or not you can even go wide or go deep in the first place, others like Google Cloud’s Head of Data and Analytics Bruno Aziza purport that it also depends on who you are, who you wish to be, and what investments your company is making to ensure you can become that person.
“If you wish to set yourself up to be a Chief Data Officer, then you’ll want to understand how technologies fit together across your data estate first” said Aziza. “Only after you feel like you’re the go-to ‘data person’ can you then decide which part of the technology stack you want to double-down on.”
But technology isn’t everything. Aziza notes, “Make sure you focus on the business impact that your data work provides. You want to spend as much time as you can with your business counterparts to understand their business goals and challenges. The Harvard Business Review provides great guidance on how to succeed as a Chief Data Officer.”
Even if you don’t have your sights set on a C-suite role, both Aziza and Razack contend that the number one skill data practitioners should tackle in 2022 is actually a broad and perhaps abstract one: develop and exercise the curiosity to solve problems with a data-driven strategy.
That is, today’s data practitioners should always be interested in educating themselves in the industry and continually upskilling in something. And their employers should also be invested in helping practitioners develop those interests, most likely through exposure to learning materials, engaging in career conversations, subsidized courses, or incentives attached to pursuing a new certification or skill.
“Every industry is going through a digital transformation and the ability to identify what data to collect, how to prepare the data, and how to derive insights from it is critical. Therefore, the ability to find business challenges and formulate a data-driven approach to address those problems is the most important skill to have.” Abdul Razack – VP, Solutions Engineering, Technology Solutions and Strategy, Google Cloud.
Take the example of the “Data Mesh” I just wrote about in VentureBeat. You’ll find 3 types of attitudes towards this new concept. There are Disciples who encourage continued learning only from the source – like the author of a new book or the creator of a theory. There are Distractors who tell you that new skills, trends, and technologies are fake news. And there are Distorters like vendors who will sell you one easy fix solution. But it’s the data practitioner who needs to proceed with caution when interacting with all three types and forge their own path to discovering the truth when they’re learning and building skills. And for better or worse, this comes with trial and error, experimentation, and an eagerness to grow relative to where they began.”
Ready to start data upskilling? Start here.
For those interested in keeping up their data curiosities, check out our Data Journeys video series. Each week Bruno Aziza investigates a new authentic customer’s data journey – from migrating to cloud or building a data platform to carrying out new data for good initiatives. Learn how they did it, their data dos and don’ts, and what’s next for them on their journey. These videos include a flavor of both specializing your data competencies and broadening your data competencies.
For those interested in deepskilling, connect with Google’s data community at our upcoming virtual event: Latest Google Cloud data analytics innovations. Register and save your spot now to get your data questions answered live by GCP’s top data leaders and watch demos from our latest products and features including BigQuery, Dataproc, Dataplex, Dataflow, and more.
If you have any questions or need support along your learning journey – we’re here for you! Sign up to be a Google Cloud Innovator, and join the Google Cloud Data Analytics Community.
Cloud Databases: An Essential Building Block for Transforming Customer Experiences
DOWNLOAD RESEARCH REPORTS2755
Of your peers have already downloaded this article
11:30 Minutes
The most insightful time you'll spend today!
Around the globe, companies realize that iterating and innovating the customer experience (CX) is the key to their business transformation and success goals. Customers expect exceptional and speedy service from organizations through personalized experiences and easy-to-use apps.
However, while most companies want data-driven innovation, many of them overlook a key mechanism to do so: operational databases. These databases run a company’s day-to-day operations and power applications around the globe.
“Operational databases are often overlooked because they are behind the scenes,” says Kumar Menon, senior vice president of data fabric and decision science technology at Atlanta-based Equifax, the multinational consumer credit reporting agency. “Companies often focus on providing strong customer-facing applications, but the application will only achieve the desired results with a well-architected operational data capability that helps drive the real-time analytics needed to make the customer experience effective and memorable. It is the backbone of anything that you will build on top of it.”
Better customer experience is a business outcome of using cloud databases to build applications. This dictum stands to reason because moving from a traditional on-premises database to a modern database in the cloud, or simply building new applications in the cloud, allows companies to address operational overhead, unlock new possibilities, implement new features quickly, increase application reliability, and serve users across more regions—all the things that enhance CX and make it possible. Google Cloud sponsored this research by Harvard Business Review Analytic Services to understand how digital innovators are delighting their customers by using modern cloud databases. The report analyzes why operational databases are an essential building block for transformative experiences and highlights some of the critical capabilities leaders should prioritize. Through interviews with technology executives, data leaders, and industry experts, it provides real-world examples of benefits companies are realizing with cloud databases. We hope you find the findings and real-world examples in this report insightful and inspiring.
Harnessing the Power of Data and AI to Transform Life Science Supply Chains
3966
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Global life science supply chains are lengthy and complex with many moving parts. One small disruption can create serious delays and affect your ability to deliver therapeutics for patients.
Supply chain disruptors
Over the last few years, healthcare organizations have encountered a range of obstacles, from both internal and external factors, that have resulted in supply networks failing to get drugs and medical devices to where they need to be on time. These obstacles include:
- Labor and supply shortages
- Rising material costs
- Raw material constraints
- Geo-political events
- Unpredictable weather
How do you overcome supply chain disruptors that are out of your control?
The intelligent healthcare supply chain
While many organizations have already implemented data-driven supply chains, organizations are still faced with the challenges of static, siloed, and different functional supply chain applications; limited data exchange with key trading partners across upstream and downstream operations; and the inability to effectively leverage relevant external data.
At Google Cloud, we believe the key to meaningful and effective change is a data-driven supply chain that allows you to achieve visibility, flexibility, and innovation.
Our solutions help you prepare for the unpredictable and enhance the value of your data. By unlocking AI-driven insights, you can strengthen distribution networks and optimize your workflows and supply chains to become more reliable, intelligent, and sustainable. Some of the business challenges we address include:
- Predicting demand with Vertex AI Forecast
- Visual inspection for quality and predictive maintenance with pre-built ML models
- Automating and optimizing pickup and delivery operations with Cloud Fleet Routing API
- Real time and holistic inventory visibility with Supply Chain Twin
Make sure you’re prepared for the unpredictable with real-time visibility over your distribution networks. Learn how you can harness the power of AI and analytics and gain actionable insights that enhance your supply chain.
3747
Of your peers have already watched this video.
20:30 Minutes
The most insightful time you'll spend today!
Business Modernization with Oracle Workloads in Google Cloud
With today’s ever-changing business dynamics, business continuity is a key focus for all of us, and enterprise databases like Oracle play a critical role. We’ll explore how running Oracle workloads in Google Cloud creates a variety of solutions, from a Bare Metal Solution to open source and cloud-native databases, to help reduce your operational overhead and keep your business running smoothly.
This session is led by Paul Miller, Head of Solution Management for DB, Google Cloud. Paul is an accomplished data management leader with over 20 years experience in application development, data management, and solution engineering. He is an emphatic team builder who loves to figure things out. Paul has deep experience on databases from leading enterprise software companies such as Google Cloud and Oracle.
Held Back by Database Scalability, This Financial Services Company Switches to Google Cloud and Cloud Spanner
11562
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
Azimut Group operates an international network of companies handling investment and asset management, mutual funds, hedge funds, and insurance. Founded in Milan, Italy in 1988, Azimut Group today has branches in fifteen countries, including Brazil, China, and the USA.
“We have subsidiaries and manage funds all over the world,” explains Simone Bertolotti, IT Manager at Azimut Holding S.p.a. “That means that any technology that we put in place has to cover needs from many different countries.”
“When complicated analysis has to be executed, we have to increase our table space in a couple of minutes so that the AI can drill down into the data and deliver the information we need.”
—Simone Bertolotti, IT Manager, Azimut Holding S.p.a.
Azimut manages its funds with investment advisors who use information sourced from Bloomberg, Reuters and others. “They use a huge amount of data,” says Simone. “They work with spreadsheets, algorithms, formulae and they analyse data in minutes.” In finance, every second is crucial, which is why Azimut decided to develop a risk management dashboard that can process information even more quickly, then distribute it worldwide.
“When an advisor manages data, that data is used to make immediate decisions on funds, capital movements or whether to sell stock,” says Simone. “They have to be ready to make recommendations for any amount of data that comes to them. For our dashboard, that means that when additional information arrives or complicated analysis has to be executed, we have to increase our table space in a couple of minutes so that the AI can drill down into the data and deliver the information we need.”
Generating insights at speed
Investors and investment managers make decisions based on the most accurate, up-to-date information possible. For Azimut Group, information sourced through financial data vendors such as Bloomberg and Reuters provided only part of the data that the group required.
“We looked to collect information from a range of different providers,” explains Simone, “then analyse it to develop a predictive algorithm that could work faster than an advisor stationed at the terminal. We set ourselves the challenge to try to manipulate that data to add new insights into our matrix, so that every one of our branches across the world can see risk information about the funds in real-time.”
“We compared Google Cloud Platform’s performance with our previous cloud provider, and saw huge benefits of switching to Google. For me, the key performance issue is scaling. With Google Cloud Platform I know that I can increase and decrease my infrastructure quickly, when I need it.
—Simone Bertolotti, IT Manager, Azimut Holding S.p.a.
The first cloud provider Azimut used to build its system struggled to scale quickly to meet different kinds of data challenges. “If we wanted to add more cores, that was fine,” says Simone. “But the previous cloud provider made it complicated to raise the amount of space in a database infrastructure and scale up to demand. Scaling up for more in-depth analysis would take a day, and our need was immediate.”
That’s why Azimut switched one year ago to Google Cloud Platform to run the 150 VMs on its risk analysis platform. “We compared Google Cloud Platform’s performance with our previous cloud provider, and saw huge benefits of switching to Google. For me, the key performance issue is scaling,” says Simone. “With Google Cloud Platform I know that I can increase and decrease my infrastructure quickly, when I need it. Instead of waiting a day to scale up infrastructure, we can request and add space to our database in a couple of minutes.”
The infrastructure of Azimut’s solution handles around 800TB of data per month, and Google’s global network of servers and high-speed connections ensure that it gets to where it’s most needed by the most direct route. Impressed by the speed, security and availability of Google Cloud Platform, Azimut has moved its intranet on to Google Cloud Platform, too, eliminating the need for staff to login with VPNs.
“Instead of waiting a day to scale up infrastructure, we can request and add space to our database in a couple of minutes.”
—Simone Bertolotti, IT Manager, Azimut Holding S.p.a.
Driving ahead with Noovle
For Azimut, migrating the risk management dashboard is the latest of many Google product collaborations with cloud consultancy Noovle. “Everything started five years ago,” says Simone, “when Noovle assisted us in migrating to Gmail from our on-premise email solution. From G Suite to Google Cloud Platform, we’ve had a great relationship. Noovle provides consultancy services, support for mobility, and external advisors who work on our premises, such as when they trained us how to broadcast our meetings on Google Hangouts. As an independent company, we know we can trust them for transparent advice. All they care about is the best way to get a job done and to help us reach our goals.”
New app, new customers
In a business case comparison, Google Cloud Platform cost Azimut 35% less to run than the previous cloud provider. Now the group is building a major new mobile application on Google App Engine to be released in 2018.
“The new mobile application will allow customers to trade directly, without human advisors, by proposing different investment solutions depending on targets the customers set,” says Simone. “So if a customer aims to make money with investments, they enter their relevant personal information and we carry out the necessary regulatory checks and suggest what they could buy. The entire project will be based on Google Cloud Platform, so customers can control their investments through the app while we manage the fund, using Google Cloud Spanner on the backend.”
Meesho’s Zero Downtime Success: Cloud CDN Migration Made Easy
1981
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
Meesho is an Indian online marketplace that serves millions of customers every day. Recently, the company decided to adopt a multi-cloud strategy, leveraging Google Cloud’s scalable and reliable infrastructure to drive operational efficiency, modernize and scale for growth. To do so, they needed to migrate billions of static files and images to Google Cloud, to render the static content that serves their web and mobile applications. But with over a petabyte of data in their object storage system, and 10 billion requests per day, Meesho needed to perform this gigantic migration gradually, with zero downtime — a huge challenge.
In this blog post, we look at how Meesho did this using Storage Transfer Service, Cloud Storage and Cloud CDN. We also look at how it saved on storage capacity by resizing static images as needed on the fly, using Cloud Run.
CDN migration requirements
Migrating from one cloud to another isn’t easy. To pull it off, Meesho identified the following requirements:
- Petabyte-scale data transfer: Meesho needed to migrate billions of image files from their existing object storage server to Cloud Storage.
- Dynamic image resizing: To save on storage costs, Meesho wanted the ability to dynamically resize the images based on the end user platform and store the smaller images in the Cloud CDN cache.
- High-throughput data transfer: To support consumer demand, Meesho needed images to be served at a throughput of thousands of requests per second.
- Zero downtime: Since any downtime involves potential loss of revenue, Meesho needed to perform the migration without taking any systems offline.
Migration architecture
The above figure depicts the CDN migration architecture implemented in Meesho. The existing DNS server points to both the source load balancer as well as Google External HTTP Load Balancer with weighted distribution. The source load balancer points to the source object storage. Images were transferred from the source object storage to Google Cloud Storage.
The Google External HTTP Load Balancer was deployed with Cloud CDN to serve static images that are stored in the CDN cache to users. The Google Load Balancer public IP is configured as an end point on their existing DNS server. The Load Balancer is connected to Cloud Run, which talks to the Cloud Storage bucket. When a request reaches the Load Balancer in the edge, it first checks if the content is available in Cloud CDN, and returns the object from the closest edge network. If the image is not available in the Cloud CDN cache, the request is sent to Cloud Run which obtains the image from the Cloud Storage bucket and performs dynamic resizing of the image if necessary.
Data transfer
Meesho used Google Cloud’s Storage Transfer Service to transfer data from their current object storage to Cloud storage bucket over the internet. Since the number of files and total size of the data to be transferred was huge, Meesho executed multiple parallel transfers by specifying folders and subfolders as prefixes in a Storage Transfer Service job.
Dynamic image resizing
Meesho delivers static images to multiple end user platforms — mobile, laptop — at multiple resolutions. Rather than store each image at multiple image resolutions, Meesho opted to store a single high-resolution mezzanine image. It then attached Cloud Run as a serverless network endpoint group to a Cloud Load Balancer. Application requests for images specify the name of the object, the format of the image, and its resolution (for example, abc.jpeg with 750*450 resolution). If the specific image exists for the requested resolution, then it is returned from the Cloud Storage bucket to the end user and stored in the Cloud CDN cache. If an image for a specified resolution and/or format is not found, the mezzanine image (in our example, abc.jpeg) is resized to the specified resolution and format, then stored in Cloud Storage bucket and returned to the end user. The dynamic resizing and formatting is only performed the first time for a specific resolution.
In this architecture, it is important to configure Cloud Run to scale appropriately as it handles a bulk of “CDN cache-miss” requests. Meesho performed the following configuration steps:
- Configured the number of concurrent requests that a single instance of Cloud Run can handle
- Ensured a sufficient minimum of Cloud Run instances were available to serve user traffic to avoid cold-start latency
- Reviewed limits of Cloud Run maximum instance size for the region and increased the limits if necessary to handle peak load
- Set up smaller start-up times for Cloud Run containers, so that the application could quickly autoscale to handle a surge in traffic
- Optimized the memory and CPU configuration to handle processing requirements
CDN configuration
Cloud CDN was configured to ensure a high cache hit ratio > 99 %. This not only sped up the rendering of the images, but also reduced the load on Cloud Run, saving cost and improving performance.
Achieving zero downtime
Meesho followed well-established DevOps principles to achieve a zero-downtime migration:
- Metrics and alerts were configured in Cloud Monitoring to oversee the load balancer.
- The DNS server was configured to point to Cloud Load Balancer IP addresses in addition to their current load balancer, which served status assets.
- Weight-based DNS load balancing was employed to gradually shift the traffic to Google Cloud, while monitoring application performance and HTTP response codes.
- The initial migration process distributed .1% of traffic during non-peak hours. The metrics, end user performance and response codes were continuously monitored.
- Traffic was gradually incremented over a two-week period by increasing the weight of the Google Cloud Load balancer in DNS. By gradually shifting traffic, Meesho ensured a healthy cache-hit ratio, allowing Cloud Run to learn traffic patterns gradually and scale seamlessly.
Meesho learned a lot through this experience, and has the following advice for anyone undertaking a similar migration:
- While transferring data using Storage Transfer Service of Google Cloud, split the transfer process into multiple transfers.
- Ensure that applications do not pin certificates, which could create problems while migrating to the newer certificates in Google Cloud.
- Plan a gradual migration process to gradually increase the traffic to Google Cloud.
Summary
When all is said and done, Meesho considers its migration to Google Cloud a big success. After migrating the static images to Cloud CDN, Meesho held two major sales that each had three times the normal peak traffic, all with no issues. The CDN migration helped Meesho reduce its costs, improve performance and reduce load balancer errors when fetching static images. To learn more about Cloud CDN and how you can use it in your environment, check out the documentation.
More Relevant Stories for Your Company
Recommendations for Modelling SAP Data inside BigQuery
Over the past few years, many organizations have experienced the benefits of migrating their SAP solutions to Google Cloud. But this migration can do more than reduce IT maintenance costs and make data more secure. By leveraging BigQuery, SAP customers can complement their SAP investments and gain fresh insights by consolidating enterprise
ActivStat: Enhancing Live Sports Broadcast with Real-time Stats and Metrics
Millions of people annually view esports, as top players and teams compete in thrilling, fast-paced tests of reflexes, strategy, and teamwork. Fans are a diverse group, sharing a passion for action. A new, revolutionary project we’re working on with Call of Duty League just made that action a lot better.
What is Cloud SQL for SQL Server? And What Do IT Departments Say About It?
At Google Cloud, we develop our database services to meet the needs of enterprise teams wherever they are in their cloud journey. That is why we announce launched Cloud SQL for SQL Server and make it available to all of our customers. This addition to our database lineup means you can migrate
The Divercity Story: Using Google Cloud to Achieve a More Inclusive and Sustainable Workforce
Despite a growing number of diversity, equity, and inclusion (DEI) initiatives, Black and Latinx people remain highly underrepresented in tech. Although comprising 12.6% and 18% of the U.S. labor force respectively, Black professionals hold only 5% of tech positions, while Latinx professionals fill just 6% of tech roles. Long-standing biases
