How will your organization manage this year’s data growth and business requirements? Your actions and strategies involving data and AI will improve or undermine your organization’s competitiveness in the months and years to come. Our teams at Google Cloud have an eye on the future as we evolve our strategies to protect technology choice, simplify data integration, increase AI adoption, deliver needed information on demand, and meet security requirements. 

Google Cloud worked with* IDC on multiple studies involving global organizations across industries in order to explore how data leaders are successfully addressing key data and AI challenges. We compiled the results in our 2023 Data and AI Trends report. In it, you’ll find the metrics-rich research behind the top five data and AI trends, along with tips and customer examples for incorporating them into your plans. 

1: Show data silos the door

Given the increasing volumes of data we’re all managing, it’s no surprise that siloed transactional databases and warehousing strategies can’t meet modern demands. Organizations want to improve how they store, manage, analyze, and govern all their data, while reducing costs. They also want to eliminate conflicting insights from replicated data and empower everyone with fresh data.

A unified data cloud enables the integration of data and insights into transformative digital experiences and better decision making.

Andi Gutmans, GM and VP of Engineering for Databases, Google Cloud

Tweet this quote

In the report, you can learn how to adopt a unified data cloud that supports every stage of the data lifecycle so that you can improve data usage, accessibility, and governance. Inform your strategy by drawing on organizations’ examples such as a data fabric that improves customer experiences by connecting more than 80 data silos, as well as other unified data clouds that save money and simplify growth. 

2: Usher in the age of the open data ecosystem

Data is the key to unlocking AI, speeding up development cycles, and increasing ROI. To protect against data and technology lock-in, more organizations are adopting open source software and open APIs.

Tweet this quote

Understand how you can simplify data integration, facilitate multicloud analytics, and use the technologies you want with an open data ecosystem, as described in the report. Learn from metrics about global open source adoption and public dataset usage. And explore how global companies adopted open data ecosystems to improve patient outcomes, increase website traffic by 25%, and cut operating costs by 90%.

3: Embrace the AI tipping point

Pulling useful information out of data is easier with AI and ML. Not only can you identify patterns and answer questions faster but the technologies also make it easier to solve problems at scale.

We’ve reached the AI tipping point. Whether people realize it or not, we’re already using applications powered by AI—every day. Social media platforms, voice assistants, and driving services are easy examples.

June Yang, VP, Cloud AI and Industry Solutions, Google Cloud

Tweet this quote

Organizations share how they’re reaching their goals using AI and ML by empowering “citizen data scientists” and having them focus on small wins first. Gain tips from Yang and other experts for developing your AI strategy. And read how organizations achieve outcomes such as a reduction of 7,400 tons per year in carbon emissions and a more than 200% increase in ROI from ad spend by using pattern recognition and other AI capabilities. 

4: Infuse insights everywhere

Yesterday’s BI solutions have led to outdated insights and user fatigue with the status quo, based on generic metrics and old information. Research shows that as new tools come online, expectations for BI are changing, with companies revising their strategies to improve decision making, speed up the development of new revenue streams, and increase customer acquisition and retention by providing individuals with needed information on demand.

Organizations are equipping business decision-makers with the tools they need to incorporate required insights into their everyday workflows.

Kate Wright, Senior Director, Product Management, Google Cloud

Tweet this quote

In the report, you’ll discover why and how data leaders are rethinking their BI analytics strategies and applications to improve users’ trust and use of data in automated workflows, customizable dashboards, and on-demand reports. Global companies also share how they improve decision making with self-service BI, customer experiences with IoT analysis, and threat mitigation with embedded analytics.

5: Get to know your unknown data

Increasing data volumes can make it harder to know where and what data they store, which may create risk. Case in point: If a customer unexpectedly shares personally identifiable information during a recorded customer support call or chat session, that data might require specialized governance, which the standardized storage process may not provide.

If you don’t know what data you have, you cannot know that it’s accurately secured. You also don’t know what security risks you are incurring, or what security measures you need to take.

Anton Chuvakin, Senior Staff Security Consultant, Google Cloud

Tweet this quote

Check out the report to learn about data security risks that are often overlooked and how to develop proactive governance strategies for your sensitive data. You can also read how global organizations have increased customer trust and productivity by improving how they discover, classify, and manage their structured and unstructured data.  

Be ready for what’s next

What’s exciting about these trends is that they’re enabling organizations across industries to realize very different goals using their choice of technologies. And although all the trends depend on each other, research shows you can realize measurable benefits whether you adopt one or all five.

Review the report yourself and learn how you can refine your organization’s data and AI strategies by drawing on the collective insights, experiences, and successes of more than 800 global organizations. 

With a data strategy and data warehouse in place, EdTechs are building a data culture that helps everyone – from educators and administrators, to employees in marketing and accounting – make more informed decisions with their data.

So how do you build a data culture in your organization? It starts by asking these questions:

• Are users able to find answers to their data questions from available data?
• Can they act based on available metrics?
• Can they tell a compelling story with the data at hand?

How well is your company using analytics? The matrix  from Looker’s Analytical Maturity eBook, shows ranges from “vanity” (nice to see) metrics to “optimization.” 

Adopting and rolling out your organization’s new data platform

As outlined in the previous blog in this series, From data chaos to data-driven: How dedicated data teams can help EdTechs influence the future of education, a comprehensive data team is essential for building a data warehouse. That data team should include a core education intelligence team. This team can align with analytically savvy members of each department they support. It’s critical that the Education Intelligence team understand each department’s reporting needs and can make adjustments based on user feedback.

Include these key components to ensure smooth adoption for internal teams:

• Share a roadmap: An effective rollout team always knows what’s next on their roadmap and communicates that to the entire organization. Launching analytics across an entire company at once usually leads to slow movement, miscommunication, and lack of adoption. A roadmap will ease this transition.
• Train everyone: This approach makes sure everyone is consistent during the data rollout. Show users how to make the most of the platform your data team has created, and be sure they know where they can go for help.
• Monitor and optimize: Monitor your organization’s analytics usage. Understand which individuals are using your education intelligence system to drive their day-to-day operations. These individuals can provide further insight into what is working. Identifying those who may need more guidance and support ensures they don’t miss out on benefits.

Guild Education: Creating a data culture

Guild Education started working with Google Cloud to build a data culture and implemented Looker as their data platform. 

The company transformed their student success program—in which coaches work directly with connected employees throughout their educational journey—with Looker. “As enrollments increased, even spreadsheets were not able to accurately keep track of caseloads,” Sean McKeever, Senior Business Intelligence Analyst, recalls. “We needed a singular source of truth that could be updated in almost real-time.” In response, they created “Student Rosters” to manage a coach’s student outreach.

Users were thrilled with the new tools, and when the coaching team grew from 20 to 90, Sean created ambassador groups consisting of data specialists to support each department. This sparked Guild Education’s “data-driven evolution.”

“Unexpectedly, the Student Success task force also became an engine for new BI work,” Sean says. “They blew my highest expectations out of the water and started owning virtually the entire process: building requirements, prototyping, testing, and deploying new dashboards and panels—with hardly any help from the BI team.”

These successes have set Guild Education on its way to becoming a fully data-driven company. Unexpectedly, the Student Success task force also became an engine for new BI work…they blew my highest expectations out of the water and started owning virtually the entire process…with hardly any help from the BI team.Sean McKeever
Senior Business Intelligence Analyst, Guild Education

Start your journey from data chaos to data-driven

We hope this three-part series has shown how data analytics has an important role to play in transforming the future of education. EdTech companies are unlocking the power of big data to serve their customers. “The education landscape is changing rapidly, and EdTech has a major role to play as institutions adapt to the massive shift in learners’ preferences and expectations,” says Jesus Trujillo Gomez, Strategic Business Executive, Education & Research, Google Cloud. 

Feeling inspired? Let’s meet your EdTech challenges together and transform your data culture. Visit Google Cloud for education technology.

Database disaster recovery (DR) planning is an important component of a bigger DR plan, and for enterprises using Microsoft SQL Server on Compute Engine, it often involves critical data.

When you’re architecting a disaster recovery solution with Microsoft SQL Server running on Google Cloud Platform (GCP), you have some decisions to make to build an effective, comprehensive plan. 

Microsoft SQL Server includes a variety of disaster recovery strategies and features, such as Always On availability groups or Failover Cluster Instances.

And Google Cloud is designed from the start for resilience and availability. There are several types of data centers available within GCP where you can map SQL Server’s availability features based on your specific requirements: zones and regions. Zones are autonomous data centers co-located within a GCP region. These regions are available in different geographies such as North America or APAC. 

However, there is no single disaster recovery strategy to map Microsoft SQL Server DR features to Google Cloud’s data center topology that satisfies every possible combination of disaster recovery requirements.

As a database architect, you have to design a custom disaster recovery strategy based on your specific use cases and requirements.

Our new Disaster Recovery for Microsoft SQL Server solution provides information on Microsoft’s SQL Server disaster recovery strategies, and shows how you can map them to zones and regions in GCP based on your business’s particular criteria and requirements. One example is deploying an availability group within a region across three zones (shown in the diagram below). 

For successful DR planning, you should have a clear conceptual model and established terminology in place. In this solution, you’ll find a base set of concepts and terms in context of Google Cloud DR. This includes defining terms like primary database, secondary database, failover, switchover, and fallback.

You’ll also find details on recovery point objective, recovery time objective and single point of failure domain, since those are key drivers for developing a specific disaster recovery solution.

Building a DR solution with Microsoft SQL Server in GCP regions
To get started with implementing the availability features of Microsoft SQL Server in the context of Google Cloud, take a look at this diagram, which shows the implementation of an Always On availability group in a GCP region, using several zones:

In the new solution, you’ll see other availability features, like log shipping, along with how they map to GCP. In addition, features in Microsoft SQL Server that are not deemed availability features—like server replication and backup file shipping—can actually be used for disaster recovery, so those are included as well. 

Disaster recovery features of Microsoft SQL Server do not have to be used in isolation and can be combined for more complex and demanding use cases. For example, you can set up availability groups in two regions with log shipping as the transfer mechanism between the regions.

Disaster Recovery for Microsoft SQL Server also describes the disaster recovery process itself, how to test and verify a defined disaster recovery solution, and outlines a basic approach, step-by-step. 

Your product teams might ask – “Why does it take so long to build a feature or application?” Building applications is a heavy lift due to the technical complexity, which includes the complexity of backend services that are used to manage and store data. Every moment focused on this technical complexity is a distraction from  delivering on core business value. Firestore alters this by having Google Cloud manage your backend complexity through a complete backend-as-a-service! 

Firestore is a serverless, NoSQL document database that unlocks application innovation with simplicity, speed and confidence. 

It acts as a glue that intelligently brings together the complete Google Cloud backend ecosystem, in-app services from Firebase and core UI frameworks & OS from Google.

What is Firestore?

Firestore is a serverless, fully managed NoSQL document database that scales from zero to global scale without configuration or downtime. Here’s what makes Firestore unique:

• Ideal for rapid, flexible and scalable web and mobile development with direct connectivity to the database.
• Supports effortless real time data synchronization with changes in your database as they happen. 
• Robust support for offline mode, so your users can keep interacting with your app even when the internet isn’t available or is unreliable.
• Fully customizable security and data validation rules to ensure the data is always protected
• Built-in strong consistency, elastic scaling, high performance & best in class 99.999% availability 
• Integration with Firebase and Google Cloud services like Cloud Functions and BigQuery, serverless data warehouse.
• In addition to a rich set of Google Cloud service integrations, Firestore also offers deep one-click integrations with a growing set of 3rd party partners via Firebase Extensions to help you even more rapidly build applications.

Document-model database

Firestore is a Document-model database. All of your data is stored in “documents” and then “collections”.  You can think of a document as a JSON object. It’s a dictionary with a set of key-value mappings, where the values can be several different supported data types including strings, numbers or binary values..

These documents are stored in collections. Documents can’t directly contain other documents, but they can point to subcollections that contain other documents, which can point to subcollections, and so on. This structure brings with it a number of advantages. For starters, all queries that you make are shallow, meaning that you can grab a document without worrying about grabbing all the data underneath it. And this means that you can structure your data hierarchically in a way that makes sense to you logically, without having to worry about grabbing tons of unnecessary data. 

How to use Firestore?

Firestore can be used in two modes:

• Firestore in Native Mode:  This mode is differentiated by its ability to directly connect your web & mobile app to Firestore. Native Mode supports up to 10K writes per second, and over a million connections. 
• Firestore in Datastore Mode: This mode supports only server-side usage of Firestore, but supports unlimited scaling, including writes. 

Conclusion

Whatever your application use case may be, if you want to build a feature or an application quickly using Firestore backend-as-a-service. For a more in-depth look into Firestore check out the documentation. https://www.youtube.com/embed/moglAjmwmUQ?enablejsapi=1&

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.

Thirteen years’ experience in e-commerce has given Teddy Chan, Chief Executive Officer and Chief Technology Officer, AfterShip, a deep understanding of the challenges of shipping and tracking packages to customers worldwide.

“The key problem many merchants face is customers asking ‘where is my order?’ and ‘when I will get the package?’” Chan says. “When I considered this issue, I came up with the idea of AfterShip.” Chan helped found AfterShip in 2011 to enable merchants to keep track of packages sent to customers via a web portal or an API. AfterShip also allows merchants to notify customers of anticipated delivery times.

“Using AfterShip, merchants can provide the same experience to buyers regardless of which couriers they use,” Chan says. “Merchants can also improve their customer engagement by including up-selling or marketing content with their delivery notifications.”

Hong Kong-headquartered AfterShip continues to grow quickly and now has a 40-person team. Thirty members of this team are based on the China special administrative region and 10 are based in India.

“Google Cloud Platform has a network of global datacentres with deep connectivity that enables us to put our infrastructure close to our customers. In addition, the ability to horizontally scale our global database using tools such as Google Cloud Spanner eliminates any limits on our geographic expansion.”

—Teddy Chan, Chief Executive Officer and Chief Technology Officer, AfterShip

By October 2017, the business was tracking about 30 million packages per month and had expanded its services to include label and rate calculation and self-service return. Revenue, package transaction numbers and team size have doubled every year for the past three years, while more than 300,000 merchants and 426 couriers are signed up to the service. Key customers include Wish, Etsy and Groupon.

Close to half AfterShip’s customers are based in the United States, about one third in Europe and the remainder are located in Asia. AfterShip had initially delivered its applications and services from an incumbent public cloud service. However, the company wanted to continue its growth trajectory while automating key infrastructure processes, implementing a continuous deployment model and controlling costs.

The business needed to achieve these objectives while maintaining a global presence and high-quality service. AfterShip started reviewing its options and decided to migrate to Google Cloud Platform (GCP). “Google Cloud Platform has a network of global datacentres with deep connectivity that enables us to place our infrastructure close to our customers,” Chan says.

“In addition, the ability to horizontally scale our global database using tools such as Google Cloud Spanner eliminates any limits to our geographic expansion. Furthermore, the managed services provided through GCP would allow us to focus on building better features for online merchants.” The reliability provided by GCP would also enable AfterShip to meet the stringent service level requirements of large digital marketplaces in the United States, Asia and elsewhere.

“Google Cloud Platform could manage the high volumes and enable us to deliver the service levels that would realise our ambition of becoming the number one tracking API platform in the world,” Chan says. “For example, with Google Cloud Platform, we can provide a 99.95% monthly uptime service level to our customers.” Finally, GCP provided managed solutions, including Google Kubernetes Engine powered by open source container orchestrator Kubernetes, that would enable AfterShip to automate processes such as scaling and enable its team to focus on developing applications.

AfterShip has moved its websites into GCP infrastructure in three datacentres around the world and anticipates completing the migration in Q4 2017. “Google provided a lot of assistance, particularly early in the project when we needed it,” Chan says. “They briefed us on several services we hadn’t known about that could replace the equivalents in the public cloud we were using previously.” The business then completed the migration using its own skilled team members. As well as Google Kubernetes Engine and Google Cloud Spanner, AfterShip is using Google BigQuery to store and analyse transaction information.

“Google Cloud Platform could manage the high volumes and enable us to deliver the service levels that would realise our ambition of becoming the number one tracking API platform in the world.”

—Teddy Chan, Chief Executive Officer and Chief Technology Officer, AfterShip

Deployment times down from one hour to two minutes

With deployment times falling from up to one hour in its previous cloud environment to about two minutes in GCP, AfterShip has been able to adopt a continuous deployment model. “This has improved our service levels,” Chan says. “If there are any issues we can fix them quickly, while we can iterate faster to create new features in response to customer requests or changes in the market. “This enables us to continue to lead our competitors.”

Targeting a 30 percent reduction in costs

AfterShip is now targeting a 30% reduction in costs by optimising its use of Docker containerisation technology on GCP.

“By using Docker with Kubernetes, we have been able to fine-tune our use of compute resources and better control our costs,” Chan says. “We’re extremely pleased with Google Cloud Platform as it really is built for engineers,” he adds. “In addition, its documentation is extremely clear, allowing us to troubleshoot or carry out activities on the platform ourselves. “We look forward to continuing to grow and extend our package tracking and associated services with Google Cloud Platform.”