The capability gap between hyperscale cloud providers has begun to narrow; however, fierce competition for enterprise workloads extends to secondary markets worldwide. Infrastructure and operations leaders should evaluate cloud providers with a broad range of use cases and a wide market presence.

Market Definition/Description

Cloud computing is a style of computing in which scalable and elastic IT-enabled capabilities are delivered as a service using internet technologies. Cloud infrastructure and platform services (CIPS) are defined as standardized, highly automated offerings, in which infrastructure resources (e.g., compute, networking and storage) are complemented by integrated platform services. These include managed application, database and functions as-a-service offerings. The resources are scalable and elastic in near-real time and are metered by use. Self-service interfaces are exposed directly to the customer, including a web-based user interface (UI) and an API. The resources may be single-tenant or multitenant, and can be hosted by a service provider or on-premises in the customer’s data center.The scope of this Magic Quadrant has changed, compared with its predecessor, the “Magic Quadrant for Cloud Infrastructure as a Service.” Gartner has developed this Magic Quadrant to reflect the changing dynamics of cloud services offered and the ways that enterprise customers adopt them. Ultimately, hyperscale cloud providers, and the broad array of services they offer beyond infrastructure as a service (IaaS), have found strategic importance in Gartner’s enterprise clients and the Magic Quadrant needed to evolve to reflect as much.The scope of the Magic Quadrant for CIPS includes IaaS and integrated platform as a service (PaaS) platforms. These include application PaaS (aPaaS), functions as a service (FaaS), database PaaS (dbPaaS), application developer PaaS (adPaaS) and industrialized private cloud offerings that are often deployed in enterprise data centers.

Understanding the Vendor Profiles, Strengths and Cautions

CIPS providers that target enterprise and midmarket customers generally offer high-quality service, with excellent availability, good performance, high security and good customer support. Exceptions will be noted in this Magic Quadrant’s evaluations of individual providers. When we say “all providers,” we specifically mean “all the evaluated providers included in this Magic Quadrant,” not all CIPS providers in general. Keep the following in mind when reading the vendor profiles:

• All the providers have public cloud IaaS and PaaS offerings. Most also offer, or are in the process of building, industrialized private cloud offerings, in which every customer is on standardized infrastructure and cloud management tools. In some cases, the provider’s industrialized, on-premises offering may share similarities to hyperconverged infrastructure (HCI), but tethered to the cloud. However, this may not resemble the provider’s public cloud service in architecture or quality. A single architecture and feature set and cross-cloud management, for both public and private CIPS, make it easier for customers to combine and migrate across service models as their needs dictate. They also enable the provider to use its engineering investments more effectively. Gartner is beginning to describe the notion of cloud-provider-managed infrastructure, wherever it may exist, as “ distributed cloud.”
• All the providers target midmarket businesses and enterprises, as well as other companies that use technology at scale. Some of the providers may also target small businesses and startups. Just because a provider targets a segment, however, does not necessarily mean that it is well-suited to that segment’s needs. Furthermore, not all providers have the capacity to serve very-large-scale customers, and some have capacity constraints in particular regions.
• All the providers offer basic cloud IaaS — compute, storage and networking resources as a service. They also offer additional value-added capabilities, notably cloud software infrastructure services — typically middleware and databases as a service — including PaaS capabilities. These services, along with IT operations management (ITOM) capabilities as a service (especially DevOps-related services), are a vital differentiator in the market, especially for Mode 2 agile IT buyers.
• All the providers claim to have high security standards. However, the extent of the security controls provided to customers varies significantly. All the providers evaluated can offer solutions that will meet common regulatory compliance needs, unless otherwise noted. All the providers have undergone SOC 1, SOC 2 and SOC 3 audits, as well as SSAE 16, ISO/IEC 27001, ISO/IEC 27017 and ISO/IEC 27018 audits. This provides a relatively high level of assurance that the providers are adhering to generally accepted practices for the security of their systems, but it does not address the extent of controls offered to customers.
• Security is a shared responsibility. Customers need to correctly configure controls, and they may need to supply additional controls beyond what their providers offer. Furthermore, providers vary in their degree of transparency as to how services are architected, although customers typically have access to third-party assessment reports under a nondisclosure agreement (NDA).

• Monthly compute availability service-level agreements (SLAs) of 99.95% and higher are generally the norm. They are typically higher than availability SLAs for managed hosting. Service credits for outages in a given month are typically capped at 100% of the monthly bill; however, some providers have caps as low as 25%. This availability percentage is typically non-negotiable, because it is based on an engineering estimate of the underlying infrastructure reliability.
• Single-instance compute SLAs have become common for providers in this Magic Quadrant. It might be more accurate to say that there are usually two SLAs — one for the compute service, and one for individual instances. Some providers have a compute availability SLA that requires customers to use compute capabilities in at least two fault domains (sometimes known as “availability zones” or the like).
• Many providers have additional SLAs. These cover network availability and performance, customer service responsiveness and other service aspects.
• Infrastructure resources are not normally automatically replicated into multiple data centers. Customers are responsible for their own business continuity. Some providers offer optional disaster recovery solutions.
• All providers offer per-second metering of virtual machines (VMs). Some can offer shorter metering increments, which can be more cost-effective for short-term batch jobs. Unless otherwise noted, providers charge on a per-VM basis.
• Providers are increasingly offering bare-metal physical servers on a dynamic basis. These are priced by the second. Providers with a bare-metal option are noted as such.
• All the providers partner with carrier-neutral colocation exchanges. This enables customers to obtain connectivity from a variety of carriers that are located in these facilities. In addition, many customers require a small amount of supplemental colocation in low-latency proximity with their cloud provider. For example, they may have a large-scale database, specialized network equipment or legacy equipment, such as a mainframe.
• Some providers offer software marketplaces. In these marketplaces, software vendors specially license and package their software to run on that provider’s cloud IaaS offering. Marketplace software can be automatically installed, and can be billed through the provider, although the software vendor often provides support.
• All providers offer enterprise-class support with 24/7 customer service. This is provided via phone, email and chat, along with an account manager. Some offer a lower level of support, but allow customers to pay extra for enterprise-class support.
• All the providers will sign contracts with customers, can invoice and can consolidate bills from multiple accounts. All providers offer online sign-up and credit card billing, because they recognize that enterprise buyers prefer contracts and invoices. Some will sign “zero dollar” contracts that do not commit a customer to a certain volume.
• Some providers will sign a U.S. Health Insurance Portability and Accountability Act Business Associate Agreement (HIPAA BAA).
• Unless otherwise noted, all providers will sign the following contract addendums:
  • An EU Data Protection Directive (95/46/EC) data-processing agreement (DPA), which includes the model clauses
  • An EU General Data Protection Regulation (GDPR) DPA
• Managed and professional services are an optional but important accelerator for customer success. Almost all providers rely heavily on managed service providers (MSPs) and system integration (SI) partners for these services. However, most providers offer their own first-party professional services and some also offer first-party managed services offerings.
• All of the evaluated providers offer a portal, documentation, technical support, customer support and contracts in English. Some can provide one or more of these in languages other than English. Most providers can conduct business in local languages.

The service provider descriptions are accurate as of the time of publication. Our technical evaluation of service features took place between January 2020 and March 2020.

Format of the Vendor Descriptions

When describing each provider, we first summarize the nature of the company, then provide information about its industrialized cloud IaaS offerings in the following format:

• Locations: Cloud data center locations by country, languages in which the company does business and languages in which technical support can be conducted.
• Recommended Uses: These are the circumstances under which we recommend the provider. They are not the only circumstances in which it may be a useful provider, but they are the scenarios for which, in Gartner’s opinion, the provider is well-suited.

For a detailed technical description of CIPS offerings, along with a use-case-focused technical evaluation, see “Critical Capabilities for Cloud Infrastructure and Platform Services, Worldwide.”We also provide a detailed list of evaluation criteria in “Solution Criteria for Cloud Integrated IaaS and PaaS.” A detailed assessment of each provider against these criteria can be found in the Solution Scorecards. The results are also available in Gartner’s Cloud Decisions portal (see “Cloud Decisions’ Cloud Compare: Perform Real-Time IaaS Pricing and Performance Analysis”).

Magic Quadrant

Figure 1. Magic Quadrant for Cloud Infrastructure and Platform Services

Vendor Strengths and Cautions

Google

Google is a Leader in this Magic Quadrant.

Locations: Google has multiple regions across Japan and the U.S., as well as a presence in Belgium, Singapore, Finland, Germany, the Netherlands, the U.K., India, Australia, Brazil, Canada and, Switzerland, as well as the Hong Kong and Taiwan markets.

Recommended Uses: Google has evolved by enhancing its strengths and attacking its limitations to providing a strong offering in every use case, other than the edge use case. Google has a future focus on building out hybrid capabilities and partnerships with telco providers.

Strengths

• Google’s open-source contributions, such as Kubernetes and TensorFlow, have been market-moving innovations that have changed the course of enterprise IT. Such innovations have served to enable other cloud service providers, but also brought developer “mind share” to Google Cloud Platform (GCP). Google’s long-term strategy is to bring additional open-source-focused partners into GCP as managed services.
• During the past year, GCP has experienced a noticeable increase in year-over-year market share in terms of IaaS and dbPaaS, albeit from a lower base, relative to other providers in this Magic Quadrant. Google has also made significant gains by closing a number of critical capability gaps between GCP and Microsoft Azure, its nearest competitor in terms of market share and capabilities.
• Gartner clients continue to associate GCP with its big data and data science capabilities, stemming from the use of services such BigQuery and Dataproc. However, the company is pressing into new territory with Anthos, GCP’s container and Kubernetes-based middleware layer, which is designed to support the development and deployment of cloud applications in a hybrid and multicloud model.

Cautions

• Some of Gartner’s clients remain cautious about Google’s commitment to serving the needs of enterprise clients when put in the context of SAP’s preference for Microsoft Azure, and GCP’s slowness in executing on some highly touted partnerships. GCP lacks enterprise-focused aPaaS capabilities and support for Oracle, and it continues to struggle with having an enterprise mindset in the field.
• From a financial perspective, GCP’s revenue is a small fraction of overall Google revenue and GCP’s criticality to the overall business is not as clear as its competitors. Furthermore, GCP’s success may erode the company’s overall healthy gross margins.
• Google’s much-vaunted network capabilities have been the source of a number of GCP outages during the last year, with devastating impact on customers. One outage was multiregional in scope, affecting GCP customers and Google consumer services, such as G Suite and YouTube. This resulted in complete GCP network unavailability for some customers.

Working to put food on all of our tables, today’s farmers are facing a higher amount of instability from input supply chain issues to weather patterns. Adding to this challenge are problems farmers face when trying to correctly time grain purchases, sales and transport. Farmers have always been stewards of the land, but now the demand for sustainable products has them needing to better prove their regenerative practices.

At Bushel, we understand these problems can’t be solved overnight or by a single company. We focus on empowering agribusinesses and farmers to work even more closely together to build a more sustainable agricultural supply chain by rapidly responding to market changes. With Bushel, farmers can track market prices in real time, instantly buy and sell grain, analyze inventory and transactions, and securely share verified information with grain operators and other producers. We provide the digital tools to streamline how farmers buy and sell commodities throughout the agricultural industry’s supply chain to help address market inefficiencies that can lead to waste, and have the information and resources to help them flex and adapt as complexity increases in farming operations.

Approximately 40% to 50% of all U.S. grain transactions now pass through the Bushel platform. As we continue to grow, Bushel continues to focus on what digital tEmpowering Farmers: Bushel Drives 40-50% of US Grain Transactions, Fueling Sustainable Agricultureools can support each point in the supply chain. Many focus on the first mile at the farm or last mile at the store. But Bushel is focused on modernizing the middle where grain purchasing and processing sit. We aim to help local grain industries and stabilize regional agricultural supply chains.

Starting with a simple mobile app; now scaling into an agricultural ecosystem

Bushel began its journey in 2017 as a small-scale platform for farmers that delivered grain contracts, cash bids, and receipts. As Bushel evolved into a comprehensive agricultural ecosystem, we realized we needed knowledgeable technology partners to help us rapidly scale while saving time and administrative costs. That’s why we started partnering with the Google for Startups Cloud Program to get support from Google and work with Google Cloud Managed Services partner, DoiT International to help support our use of GKE and create a multi-regional deployment as well as migrate our CUDs to new Compute Engine families and continue to optimize our footprint. We’ll also use DoiT’s Flexsave technology to reduce the management overhead of CUDs in the future.

In just one year, we expanded to over 1,200 live grain receiving locations and quickly grew our services portfolio with electronic signature capabilities, commodity balances, and web development. Because that relationship between farmer and agribusiness is so important, we provide more than 200 grain companies with white-labled digital experiences so each farmer sees their local grain facility they do business with on both desktop and mobile. To further our extension into the digital infrastructure of agriculture, we subsequently acquired GrainBridge and FarmLogs to help farmers handle specific jobs and tasks, and provide the needed insights to improve their business operations. Over 2,000 grain receiving locations across the United States and Canada now use Bushel products. We accomplished all this on Google Cloud.

We leverage the secure-by-design infrastructure to protect millions of financial transactions and keep sensitive customer data safe. Our data is processed and stored in Google’s secure data centers, which maintain adherence to a number of compliance frameworks. We utilize Google Kubernetes Engine extensively as it reduces operational overhead and offers auto scaling up to 15,000 nodes.

Database provisioning, storage capacity management, and other time-consuming tasks are automated with our Cloud SQL usage. Query Insights for Cloud SQL streamlines database observability and seamlessly integrates with existing apps and Google Cloud services such as GKE and BigQuery.

Empowering farmers and agribusinesses in North America

The Google Cloud Account Team had been instrumental in helping Bushel build an expansive agricultural platform that powers APIs, apps, websites, and digital solutions. Google’s startup experts are incredibly responsive, with deep technical knowledge that can’t be found elsewhere. Google Cloud also has provided us credits to explore new ways of analyzing the vast amounts of data we generate, verify, and transfer with solutions such as BigQuery and Pub/Sub.

With BigQuery, we can run analytics at scale with 26%–34% lower three-year TCO than cloud data warehouse alternatives. BigQuery delivers actionable insights on a highly secure and scalable platform, includes built-in machine learning capabilities, and integrates with Pub/Sub to ingest and stream analytic events via Dataflow.

With Bushel, farmers across North America are rapidly responding to sudden market changes by tracking grain prices in real time and instantly buying and selling crops. We see a future where this business information becomes insights – where a farmer can not just know where to sell their grain, but when to sell. The burden right now to engage with carbon markets is high, full of paper-based binders and verification forms. We see a world where farming practices recorded digitally can be permissioned along the supply chain for a better picture of how our food is grown.

With the Bushel platform, millions of farmers around the world will have the digital tools to modernize local grain industries, build more sustainable agricultural supply chains, and help to address global food inequity.

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

Data teams across companies have continuous challenges of consolidating data, processing it and making it useful. They deal with challenges such as a mixture of multiple ETL jobs, long ETL windows capacity-bound on-premise data warehouses and ever-increasing demands from users. They also need to make sure that the downstream requirements of ML, reporting and analytics are met with the data processing. And, they need to plan for the future – how will more data be handled and how new downstream teams will be supported?

Checkout how Independence Health Group is addressing their enterprise data warehouse (EDW) migration in the video above.

Why BigQuery?

On-premises data warehouses become difficult to scale so most companies’ biggest goal is to create a forward looking system to store data that is secure, scalable and cost effective. GCP’s BigQuery is serverless, highly scalable, and cost-effective and is a great technical fit for the EDW use-case. It’s a multicloud data warehouse designed for business agility. But, migrating a large, highly-integrated data warehouse from on-premise to BigQuery is not a flip-a-switch kinda migration. You need to make sure your downstream systems dont break due to inconsistent results in migrating datasets, both during and after the migration. So..you have to plan your migration. 

Data warehouse migration strategy

 The following steps are typical for a successful migration: 

• Assessment and planning: Find the scope in advance to plan the migration of the legacy data warehouse 
  • Identify data groupings, application access patterns and capacities
  • Use tools and utilities to identify unknown complexities and dependencies 
  • Identify required application conversions and testing
  • Determine initial processing and storage capacity for budget forecasting and capacity planning 
  • Consider growth and changes anticipated during the migration period 
  • Develop a future state strategy and vision to guide design
• Migration: Establish GCP foundation and begin migration
  • As the cloud foundation is being set up, consider running focused POCs to validate data migration processes and timelines
  • Look for automated utilities to help with any required code migration
  • Plan to maintain data synchronization between legacy and target EDW during the duration of the migration. This becomes a critical business process to keep the project on schedule.
  • Plan to integrate some enterprise tooling to help existing teams span both environments
  • Consider current data access patterns among EDW user communities and how they will map to similar controls available in Big Query. 
  • Key scope includes code integration and data model conversions
  • Expect to refine capacity forecasts and refine allocation design. In Big Query there are many options to balance cost and performance to maximize business value. For example, you can use either on-demand or flat-rate slot pricing or a combination of both. 
• Validation and testing
  •  Look for tools to allow automated, intelligent data validation 
  • Scope must include both schema and data validation
  • Ideally solutions will allow continuous validation from source to target system during migration
  • Testing complexity and duration will be driven by number and complexity of applications consuming data from the EDW and rate of change of those applications 

A key to successful migration is finding Google Cloud partners with experience migrating EDW workloads. For example, our Google Cloud partner Datametica offers services and specialized Migration Accelerators for each of these migration stages to make it more efficient to plan and execute migrations.

Data warehouse migration: Things to consider

• Financial benefits of open source: Target moving to ‘Open Source’ where none of the services have license fees. For example BigQuery uses Standard SQL; Cloud Composer is managed Apache Airflow, Dataflow is based on Apache Beam. Taking these as managed services provides the financial benefits of open source, but avoids the burden of maintaining open source platforms internally. 
• Serverless: Move to “serverless” big data services. The majority of the services used in a recommended GCP data architecture scale on demand allowing more cost effective alignment with needs. Using fully managed services lets you focus engineering time on business roadmap priorities, not building and maintaining infrastructure. 
• Efficiencies of a Unified platform: Any data warehouse migration involves integration with services that surround the EDW for data ingest and pre-processing and advanced analytics on the data stored in the EDW to maximize business value. A cloud provider like GCP offers a full breadth of integrated and managed ‘big data’ services with built-in machine learning. This can yield significantly reduced long-term TCO by increasing both operational and cost efficiency when compared to EDW-specific point solutions. 
• Establishing a solid cloud foundation: From the beginning, take the time to design a secure foundation that will serve the business and technical needs for workloads to follow. Key features include: Scalable Resource Hierarchy, Multi-layer security, multi-tiered network and data center strategy and automation using Infrastructure-as-Code. Also allow time to integrate cloud-based services into existing enterprise systems such as CI/CD pipelines, monitoring, alerting, logging, process scheduling, and service request management. 
• Unlimited expansion capacity: Moving to cloud sounds like a major step, but really look at this as adding more data centers accessible to your teams. Of course, these data centers offer many new services that are very difficult to develop in-house and provide nearly unlimited expansion capacity with minimal up-front financial commitment. . 
• Patience and interim platforms: Migrating an EDW is typically a long running project. Be ready to design and operate interim platforms for data synchronization, validation and application testing. Consider the impact on up-stream and down-stream systems. It might make sense to migrate and modernize these systems concurrent with the EDW migration since they are probably data sources and sinks and may be facing similar growth challenges. Also be ready to accommodate new business requirements that develop during the migration. Take advantage of the long duration to have existing your operational teams learn new services from the partner leading the deployment so your teams are ready to take over post-migration. 
• Experienced partner: An EDW migration can be a major undertaking with challenges and risks during migration, but offers tremendous opportunities to reduce costs, simplify operations and offer dramatically improved capacities to internal and external EDW users. Selecting the right partner reduces the technical and financial risks, and allows you to plan for and possibly start leveraging these long-term benefits early in the migration process.

Example Data Warehouse Migration Architecture

• Setup foundational elements. In GCP these include, IAM for authorization and access, cloud resource hierarchy, billing, networking, code pipelines, Infrastructure as Code using Cloud Build with Terraform ( GCP Foundation Toolkit), Cloud DNS and a dedicated/partner Interconnect to connect to the current data centers.
• Activate monitoring and security scanning services before real user data is loaded using Cloud Operations for monitoring and logging and Security Command Center for security monitoring. 
• Extract files from on-premise legacy EDW and move to Cloud Storage and establish on-going synchronization using Big Query Transfer services. 
• From Cloud Storage, process the data in Dataflow and Load/Export data to BigQuery. 
• Validate the export using Datametica’s validation utilities running in a GKE cluster and Cloud SQL for auditing and historical data synchronization as needed. Application teams test against the validated data sets throughout the migration process. 
• Orchestrate the entire pipeline using Cloud Composer, integrated with on-prem scheduling services as needed to leverage established processes and keep legacy and new systems in sync. 
• Maintain close coordination with teams/services ingesting new data into the EDW and down-streams analytics teams relying on the EDW data for on-going advanced analytics. 
• Establish fine-grained access controls to data sets and start making the data in Big Query available to existing reporting, visualization and application consumption tools using BigQuery data connectors for ‘down-stream’ user access and testing. 
• Incrementally increase Big Query flat-rate processing capacity to provide the most cost-effective utilization of resources during migration. 

To learn more about migrating from on-premises Enterprise Data Warehouses (EDW) to Bigquery and GCP here.