Data teams across companies have continuous challenges of consolidating data, processing it and making it useful. They deal with challenges such as a mixture of multiple ETL jobs, long ETL windows capacity-bound on-premise data warehouses and ever-increasing demands from users. They also need to make sure that the downstream requirements of ML, reporting and analytics are met with the data processing. And, they need to plan for the future – how will more data be handled and how new downstream teams will be supported?

Checkout how Independence Health Group is addressing their enterprise data warehouse (EDW) migration in the video above.

Why BigQuery?

On-premises data warehouses become difficult to scale so most companies’ biggest goal is to create a forward looking system to store data that is secure, scalable and cost effective. GCP’s BigQuery is serverless, highly scalable, and cost-effective and is a great technical fit for the EDW use-case. It’s a multicloud data warehouse designed for business agility. But, migrating a large, highly-integrated data warehouse from on-premise to BigQuery is not a flip-a-switch kinda migration. You need to make sure your downstream systems dont break due to inconsistent results in migrating datasets, both during and after the migration. So..you have to plan your migration. 

Data warehouse migration strategy

 The following steps are typical for a successful migration: 

• Assessment and planning: Find the scope in advance to plan the migration of the legacy data warehouse 
  • Identify data groupings, application access patterns and capacities
  • Use tools and utilities to identify unknown complexities and dependencies 
  • Identify required application conversions and testing
  • Determine initial processing and storage capacity for budget forecasting and capacity planning 
  • Consider growth and changes anticipated during the migration period 
  • Develop a future state strategy and vision to guide design
• Migration: Establish GCP foundation and begin migration
  • As the cloud foundation is being set up, consider running focused POCs to validate data migration processes and timelines
  • Look for automated utilities to help with any required code migration
  • Plan to maintain data synchronization between legacy and target EDW during the duration of the migration. This becomes a critical business process to keep the project on schedule.
  • Plan to integrate some enterprise tooling to help existing teams span both environments
  • Consider current data access patterns among EDW user communities and how they will map to similar controls available in Big Query. 
  • Key scope includes code integration and data model conversions
  • Expect to refine capacity forecasts and refine allocation design. In Big Query there are many options to balance cost and performance to maximize business value. For example, you can use either on-demand or flat-rate slot pricing or a combination of both. 
• Validation and testing
  •  Look for tools to allow automated, intelligent data validation 
  • Scope must include both schema and data validation
  • Ideally solutions will allow continuous validation from source to target system during migration
  • Testing complexity and duration will be driven by number and complexity of applications consuming data from the EDW and rate of change of those applications 

A key to successful migration is finding Google Cloud partners with experience migrating EDW workloads. For example, our Google Cloud partner Datametica offers services and specialized Migration Accelerators for each of these migration stages to make it more efficient to plan and execute migrations.

Data warehouse migration: Things to consider

• Financial benefits of open source: Target moving to ‘Open Source’ where none of the services have license fees. For example BigQuery uses Standard SQL; Cloud Composer is managed Apache Airflow, Dataflow is based on Apache Beam. Taking these as managed services provides the financial benefits of open source, but avoids the burden of maintaining open source platforms internally. 
• Serverless: Move to “serverless” big data services. The majority of the services used in a recommended GCP data architecture scale on demand allowing more cost effective alignment with needs. Using fully managed services lets you focus engineering time on business roadmap priorities, not building and maintaining infrastructure. 
• Efficiencies of a Unified platform: Any data warehouse migration involves integration with services that surround the EDW for data ingest and pre-processing and advanced analytics on the data stored in the EDW to maximize business value. A cloud provider like GCP offers a full breadth of integrated and managed ‘big data’ services with built-in machine learning. This can yield significantly reduced long-term TCO by increasing both operational and cost efficiency when compared to EDW-specific point solutions. 
• Establishing a solid cloud foundation: From the beginning, take the time to design a secure foundation that will serve the business and technical needs for workloads to follow. Key features include: Scalable Resource Hierarchy, Multi-layer security, multi-tiered network and data center strategy and automation using Infrastructure-as-Code. Also allow time to integrate cloud-based services into existing enterprise systems such as CI/CD pipelines, monitoring, alerting, logging, process scheduling, and service request management. 
• Unlimited expansion capacity: Moving to cloud sounds like a major step, but really look at this as adding more data centers accessible to your teams. Of course, these data centers offer many new services that are very difficult to develop in-house and provide nearly unlimited expansion capacity with minimal up-front financial commitment. . 
• Patience and interim platforms: Migrating an EDW is typically a long running project. Be ready to design and operate interim platforms for data synchronization, validation and application testing. Consider the impact on up-stream and down-stream systems. It might make sense to migrate and modernize these systems concurrent with the EDW migration since they are probably data sources and sinks and may be facing similar growth challenges. Also be ready to accommodate new business requirements that develop during the migration. Take advantage of the long duration to have existing your operational teams learn new services from the partner leading the deployment so your teams are ready to take over post-migration. 
• Experienced partner: An EDW migration can be a major undertaking with challenges and risks during migration, but offers tremendous opportunities to reduce costs, simplify operations and offer dramatically improved capacities to internal and external EDW users. Selecting the right partner reduces the technical and financial risks, and allows you to plan for and possibly start leveraging these long-term benefits early in the migration process.

Example Data Warehouse Migration Architecture

• Setup foundational elements. In GCP these include, IAM for authorization and access, cloud resource hierarchy, billing, networking, code pipelines, Infrastructure as Code using Cloud Build with Terraform ( GCP Foundation Toolkit), Cloud DNS and a dedicated/partner Interconnect to connect to the current data centers.
• Activate monitoring and security scanning services before real user data is loaded using Cloud Operations for monitoring and logging and Security Command Center for security monitoring. 
• Extract files from on-premise legacy EDW and move to Cloud Storage and establish on-going synchronization using Big Query Transfer services. 
• From Cloud Storage, process the data in Dataflow and Load/Export data to BigQuery. 
• Validate the export using Datametica’s validation utilities running in a GKE cluster and Cloud SQL for auditing and historical data synchronization as needed. Application teams test against the validated data sets throughout the migration process. 
• Orchestrate the entire pipeline using Cloud Composer, integrated with on-prem scheduling services as needed to leverage established processes and keep legacy and new systems in sync. 
• Maintain close coordination with teams/services ingesting new data into the EDW and down-streams analytics teams relying on the EDW data for on-going advanced analytics. 
• Establish fine-grained access controls to data sets and start making the data in Big Query available to existing reporting, visualization and application consumption tools using BigQuery data connectors for ‘down-stream’ user access and testing. 
• Incrementally increase Big Query flat-rate processing capacity to provide the most cost-effective utilization of resources during migration. 

To learn more about migrating from on-premises Enterprise Data Warehouses (EDW) to Bigquery and GCP here.

When moving to the cloud, many organizations concentrate their focus on the change in technology, and overlook an area just as complex: cultural change. 

At Google, we’ve spent years nurturing our culture and workforce to best operate in the cloud, and the Google Cloud Professional Services team leverages the lessons we’ve learned for the benefit of enterprise customers embarking on their own cloud journeys. 

While it can be tempting to believe in a universally ‘correct’ strategy for change management, there is no one-size-fits-all answer. Every organization will have its own unique considerations. But with that said, there are some core strategies we’ve found to be relevant and useful across a broad range of businesses. 

1. Define your purpose for moving to Cloud
While pockets of cloud use and experimentation can evolve independently and in parallel across an organization, it’s important to make some deliberate decisions before starting a larger migration. At this stage, we recommend having a detailed answer to two key questions to ensure a successful cloud migration:

• Where do you want to go? (Or “What’s your cloud vision?”) 
• How do you plan to get there?

Start by having a conversation with leaders and those who will be key to the journey about how far you want to push your cloud vision. This alignment ensures everyone is on the same page—and will provide greater direction, allowing more deliberate action. 

2. Find the change path which is right for you 
Whether a ‘lift and shift’ approach to the cloud is right for you, or a more transformative approach with a lot of re-architecting—the most important thing is to find the flavor of change which is appropriate to your context and level of ambition.This will both shape your key migration activities, but also the level of impact to be managed within your organization.  

There are many ways to embark on a change journey for cloud migration (which one can find in the chart below). It is important to deeply understand the needs of your business and its people and determine what strategy makes the most sense.

3. Learn from best practices
Based on the lessons we’ve learned along our own journey, and the work we’ve done with customers, there are a number of recommendations we can share that can make a cloud migration more successful. We go into these in more detail in our new whitepaper, but below you can find the ones we think are most relevant: 

• Share the vision—and measure, measure, measure. Once you’ve crystallised your cloud vision with leadership and key stakeholders, share that vision widely. Set success goals and communicate them to hold yourself accountable.  
• Be clear about the capabilities you will need in the future—and where you’ll get them. For example, if your vision is to become a cloud-first, data and AI-led organization, ensuring you have the right data science skills and machine learning capabilities in your organization to achieve that vision becomes a critical step—be they home-grown or bought-in.
• Find the right balance between capabilities that should be under central control, and capabilities that should be decentralized, or agile. For example, should machine learning be something that sits centrally, or should it be spread across your organization? For every business, the solution will be a little different, and there’s no “one true answer.” There’ll be lots of different opinions about this, so the sooner the conversation starts, the better.  
• Start thinking about the needed tech and non-tech skills now, and how you’ll fill the gaps. Building the tech skills will take time, and not everyone will feel comfortable with the future picture of collaboration, innovation, and agility. 

To help businesses navigate their own cloud journeys, Google Cloud Professional Services has released a new whitepaper that can help guide organizations. “Managing Change in the Cloud” is closely aligned with the Google Cloud Adoption Framework and is a practical guide for organizations looking to maintain momentum in their cloud adoption. You can download the whitepaper here.

No discussion about the emerging technologies that are shaping our future is complete without a mention of cloud computing. Unfortunately, too many of these discussions are filled with misinformation and fear-mongering. And while it’s always advisable to approach new technologies with a degree of healthy skepticism, this should always be backed by facts.

Here are seven of the most common myths about cloud computing that you shouldn’t take at face value:

An On-Premises Data Centre is More Secure Than the Cloud
One of the first concerns cloud skeptics cite as the case against it is the possibility of a data breach or cyber attack. However, the overwhelming majority of professionals agree that cloud computing offers much more security than an on-premises data centre. Not only do cloud solutions come with the requisite infrastructure, but they’re also maintained by some of the finest security personnel in the world, who work for cloud service providers.

Cloud Computing is Too Expensive
The concerns surrounding the cost of cloud computing have been blown vastly out of proportion. While the cost of implementation varies from one business to another, cloud computing’s ‘pay-as-you-go’ model helps prevent unnecessary spending on infrastructure. In fact, some providers offer users their service without any up-front costs or termination fees. However, it is up to an enterprise to evaluate whether this model will be more cost-effective for their requirements, as opposed to building a data centre. 

Migration to the Cloud Will Result in Extended Downtime
There’s a reason that the old adage ‘Time is money’ is as popular as it is. Irrespective of its industry, geography, size or nature, no organisation can afford more than the bare minimum downtime. However, cloud migration doesn’t have to be synonymous with lost time and money. In fact, most popular cloud service providers offer seamless, live migration, and may only result in negligible downtime where the existing servers are extremely outdated.

My Business is Too Small for Cloud Computing
Cloud computing was once thought of as the prerogative of big companies that have massive computing power and infrastructure needs. However, a quick glance through the pros and cons of the technology shows that this isn’t the case. In fact, small and medium organisations might just find it to be more useful. Not only does cloud adoption save the cost of building a data centre, but also the human resource expenditure that comes with having to maintain it. Moreover, offerings like Google Apps for Business consolidate a multitude of services into one platform, making them easier to implement and maintain.

Cloud Computing is Only Good for Storage and Analytics
Data storage and analytics are among the cloud’s most popular uses. However, this doesn’t mean that the technology is only of use to technology and data science departments. Figures from 2011 showed that the bulk of cloud spending in the Asia Pacific region is concentrated in the functions of customer service, marketing, sales, manufacturing and human resource management. Seamless flow of accurate and up-to-date information is department-agnostic, and if leveraged in the right way, can be a game-changer for every organisation.

I Won’t Have Enough Control on the Cloud
Cloud solutions aren’t as rigid as most people assume. Between the public, private and hybrid cloud, there are a host of options that businesses can choose from when it comes to cloud migration. Each of these offers a different degree of customisation and flexibility, which companies can pick from, on the basis of their requirements and skill level.

Cloud Computing is Digital Transformation
Deploying cloud solutions can put organisations on the fast track to success in the digital age. However, it is far from the only component that is needed to make the big transition. It’s not unheard of for vendors and IT professionals to make cloud computing seem like the be-all and end-all solution to all of a business’ digital woes. This is why, senior management should have a clearly defined purpose and realistic expectations in mind before migrating to the cloud.

Enterprise agility and the ability to innovate, adapt and respond quickly to the ever-changing risk and regulatory landscape is no longer a choice, but the cornerstone of successful digital transformation and commercial growth. Traditional access to and ways of managing data invariably create challenges in dealing with multiple data repositories, reconciliations, fire-drills, etc.

In response, the move to cloud is increasing significantly. It enables risk analytics and regulatory reporting at scale in a secure environment with data storage, management and encryption capabilities as a standard. In addition, as regulatory reporting requirements become more granular, machine learning can help facilitate new insights and allow for risk management to become more embedded into operational processes.

This webinar will address the day-to-day challenges in risk management and regulatory compliance, while also exploring how technological innovations can provide massive improvements and potential.

Key themes

• Real-life data challenges in the eyes of risk managers: can compliance, fraud detection and identifying liquidity positions be improved through the use of AI?
• Innovative approaches to streamline regulatory reporting to derive deeper customer insights from data at the moment of truth.
• Reimagining operations: how to modernise the data infrastructure to accommodate data explosion, drive flexibility and deliver a more cost effective outcome.