BigQuery Admin Reference Guide Series: How to Optimize Data in Your Native Storage
3731
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
So far on the BigQuery Admin Reference Guide series, we’ve talked about the different logical resources available inside of BigQuery. Now, we’re going to begin talking about BigQuery’s architecture. In this post we’re diving into how BigQuery stores your data in native storage, and what levers you can pull to optimize how your data is stored.
Columnar storage format
BigQuery offers fully managed storage, meaning you don’t have to provision servers. Sizing is done automatically and you only pay for what you use. Because BigQuery was designed for large scale data analytics data is stored in columnar format.
Traditional relational databases, like Postgres and MySQL, store data row-by-row in record-oriented storage. This makes them great for transactional updates and OLTP (Online Transaction Processing) use cases because they only need to open up a single row to read or write data. However, if you want to perform an aggregation like a sum of an entire column, you would need to read the entire table into memory.
BigQuery uses columnar storage where each column is stored in a separate file block. This makes BigQuery an ideal solution for OLAP (Online Analytical Processing) use cases. When you want to perform aggregations you only need to read the column that you are aggregating over.
Optimized storage format
Internally, BigQuery stores data in a proprietary columnar format called Capacitor. We know Capacitor is a column-oriented format as discussed above. This means that the values of each field, or column, are stored separately so the overhead of reading the file is proportional to the number of fields you actually read. This doesn’t necessarily mean that each column is in its own file, it just means that each column is stored in a file block, which is actually compressed independently for increased optimization.
What’s really cool is that Capacitor builds an approximation model that takes in relevant factors like the type of data (e.g. a really long string vs. an integer) and usage of the data (e.g. some columns are more likely to be used as filters in WHERE clauses) in order to reshuffle rows and encode columns. While every column is being encoded, BigQuery also collects various statistics about the data — which are persisted and used later during query execution.
If you want to learn more about Capacitor, check out this blog post from Google’s own Chief BigQuery Officer.
Encryption and managed durability
Now that we understand how the data is saved in specific files, we can talk about where these files actually live. BigQuery’s persistence layer is provided by Google’s distributed file system, Colossus, where data is automatically compressed, encrypted, replicated, and distributed.
There are many levels of defense against unauthorized access in Google Cloud Platform, one of them being that 100% of data is encrypted at rest. Plus, if you want to control encryption yourself, you can use customer-managed encryption keys.
Colossus also ensures durability by using something called erasure encoding – which breaks data into fragments and saves redundant pieces across a set of different disks. However, to ensure the data is both durable and available, the data is also replicated to another availability zone within the same region that was designated when you created your dataset.
This means data is saved in a different building that has a different power system and network. The chances of multiple availability zones going offline at once is very small. But if you use “Multi-Region” locations – like the US or EU – BigQuery stores another copy of the data in an off-region replica. That way, the data is recoverable in the event of a major disaster.
This is all accomplished without impacting the compute resources available for your queries. Plus encoding, encryption and replication are included in the price of BigQuery storage – no hidden costs!
Optimizing storage for query performance
BigQuery has a built-in storage optimizer that helps arrange data into the optimal shape for querying, by periodically rewriting files. Files may be written first in a format that is fast to write but later BigQuery will format them in a way that is fast to query. Aside from the optimization happening behind the scenes, there are also a few things you can do to further enhance storage.
Partitioning
A partitioned table is a special table that is divided into segments, called partitions. BigQuery leverages partitioning to minimize the amount of data that workers read from disk. Queries that contain filters on the partitioning column can dramatically reduce the overall data scanned, which can yield improved performance and reduced query cost for on-demand queries. New data written to a partitioned table is automatically delivered to the appropriate partition.
BigQuery supports the following ways to create partitioned tables:
- Ingestion time partitioned tables: daily partitions reflecting the time the data was ingested into BigQuery. This option is useful if you’ll be filtering data based on when new data was added. For example, the new Google Trends Dataset is refreshed each day, you might only be interested in the latest trends.
- Time-unit column partitioned tables: BigQuery routes data to the appropriate partition based on date value in the partitioning column. You can create partitions with granularity starting from hourly partitioning. This option is useful if you’ll be filtering data based on the date value in the table, for example looking at the most recent transactions by including a WHERE clause for transaction_created_date
- INTEGER range partitioned tables: Partitioned based on an integer column that can be bucketed. This option is useful if you’ll be filtering data based on an integer column in the table, for example focusing on specific customers using customer_id. You can bucket the integer values to create appropriately sized partitions, like having all customers with IDs from 0-100 in the same partition.
Partitioning is a great way to optimize query performance, especially for large tables that are often filtered down during analytics. When deciding on the appropriate partition key, make sure to consider how everyone in your organization is leveraging the table. For large tables that could cause some expensive queries, you might want to require partitions to be used.
Partitions are designed for places where there is a large amount of data and a low number of distinct values. A good rule of thumb is making sure partitions are greater than 1 GB. If you over partition your tables, you’ll create a lot of metadata – which means that reading in lots of partitions may actually slow down your query.
Clustering
When a table is clustered in BigQuery, the data is automatically sorted based on the contents of one or more columns (up to 4, that you specify). Usually high cardinality and non-temporal columns are preferred for clustering, as opposed to partitioning which is better for fields with lower cardinality. You’re not limited to choosing just one, you can have a single table that is both partitioned and clustered!
The order of clustered columns determines the sort order of the data. When new data is added to a table or a specific partition, BigQuery performs free, automatic re-clustering in the background. Specifically, clustering can improve the performance for queries:
- Containing where clauses with a clustered column: BigQuery uses the sorted blocks to eliminate scans of unnecessary data. The order of the filters in the where clause matters, so use filters that leverage clustering first
- That aggregate data based on values in a clustered column: performance is improved because the sorted blocks collocate rows with similar values
- With joins where the join key is used to cluster the table: less data is scanned, for some queries this offers a performance boost over partitioning!
Looking for some more information and example queries? Check out this blog post!
Denormalizing
If you come from a traditional database background, you’re probably used to creating normalized schemas – where you optimize your structure so that data is not repeated. This is important for OLTP workloads (as we discussed earlier) because you’re often making updates to the data. If your customer’s address is stored every place they have made a purchase, then it might be cumbersome to update their address if it changes.
However, when performing analytical operations on normalized schemas, usually multiple tables need to be joined together. If we instead denormalize our data, so that information (like the customer address) is repeated and stored in the same table, then we can eliminate the need to have a JOIN in our query. For BigQuery specifically, we can also take advantage of support for nested and repeated structures. Expressing records using STRUCTs and ARRAYs can not only provide a more natural representation of the underlying data, but in some cases it can also eliminate the need to use a GROUP BY statement. For example, using ARRAY_LENGTH instead of COUNT.
Keep in mind that denormalization has some disadvantages. First off, they aren’t storage-optimal. Although, many times the low cost of BigQuery storage addresses this concern. Second, maintaining data integrity can require increased machine time and sometimes human time for testing and verification. We recommend that you prioritize partitioning and clustering before denormalization, and then focus on data that rarely requires updates.
Optimizing for storage costs
When it comes to optimizing storage costs in BigQuery, you may want to focus on removing unneeded tables and partitions. You can configure the default table expiration for your datasets, configure the expiration time for your tables, and configure the partition expiration for partitioned tables. This can be especially useful if you’re creating materialized views or tables for ad-hoc workflows, or if you only need access to the most recent data.
Additionally, you can take advantage of BigQuery’s long term storage. If you have a table that is not used for 90 consecutive days, the price of storage for that table automatically drops by 50 percent to $0.01 per GB, per month. This is the same cost as Cloud Storage Nearline, so it might make sense to keep older, unused data in BigQuery as opposed to exporting it to Cloud Storage.Thanks for tuning in this week! Next week, we’re talking about query processing – a precursor to some query optimization techniques that will help you troubleshoot and cut costs. Be sure to stay up-to-date on this series by following me on LinkedIn and Twitter!
Three New Features in Cloud SQL for SQL Server Extends its Functionality
3395
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
As a product with a long history in the database ecosystem, SQL Server offers numerous native capabilities that help provide scalability and security to its users. However, it can be time consuming and complex to take advantage of these features. Google Cloud SQL for SQL Server saves your team time by eliminating much of the unnecessary toil (OS patching, version upgrades, replica setup etc.) while still allowing you to leverage the functionality you’re used to. Three new features for Cloud SQL for SQL Server take its functionality even further.
A few months ago, we announced Active Directory (AD) integration had entered preview; now, it is generally available. Equally exciting, we are releasing Cross-Region Replicas (based on SQL Server’s Always On Availability Groups) in preview. Finally, you can try out this great new functionality in our managed database service with the latest release of SQL Server 2019, which is now generally available.
Simple and Secure Windows Authentication with Active Directory
As one of the most requested and critical security capabilities for Cloud SQL for SQL Server, we are pleased to now provide Windows Authentication via Managed Service for Microsoft Active Directory as generally available. Customers should feel confident onboarding their business critical production workloads to the managed service while still maintaining the authentication best practices they rely on today. While identities can be created and managed directly within the managed AD service, many customers choose to establish a trust relationship with their existing on-prem AD footprint to leverage existing identity objects.
What is Cross-Region Replica for SQL Server?
Bringing parity in the Cloud SQL portfolio alongside MySQL and PostgreSQL, Cross-region replica makes it easy to create a fully managed read replica in a different region than that of the primary instance. You can create a replica in any Google Cloud region. The difference for SQL Server is the Availability Group based architecture that paves the way for the service to continue to offer more core compatibility with the SQL Server features our customers depend on. Cloud SQL greatly simplifies the traditional process of provisioning Availability Groups and streamlines it into a few-step workflow.
Using read replicas will allow you to horizontally scale your read workloads. For example, you can configure a reporting dashboard to work against a read replica, and because it’s only reading, it will not affect the primary instance. You can also promote replicas to be Cloud SQL instances and that could help you reduce your recovery point objective (RPO) and recovery time objective (RTO). It can help you with the RPO because the data is constantly replicated and the replica is probably more up to date than your latest backup. It can help you with RTO because promoting the replica, especially in an automated way, is a relatively short process. To get started, check out the documentation for Cross-Region Replica
What’s new in SQL Server 2019?
Providing the most current major and minor versions is a key aspect of maintaining compatibility and security for your database workload. Cloud SQL provides an easy provisioning experience that will now allow you to select from four editions of SQL Server 2019 similar to our current SQL Server 2017 options of Enterprise, Standard, Web, and Express. A few key considerations as you are evaluating the new version should be:
- Compatibility level – A newly created database on a Cloud SQL for SQL Server 2019 Databases instance has a compatibility level of 150 by default.
- Accelerated Database Recovery – Allows instances to reduce the availability impact of restarts and shutdowns.
- TempDB changes – While we recently provided you more control to manage your tempdb files, 2019 also brings optimization to improve performance as well.
- Intelligent query processing – SQL Server 2019 provides direct improvements to the query engine itself which may improve overall query processing and performance.
- Many other performance improvements – capabilities such as verbose truncation warnings, resumable index build, and others. Learn more about supported features here.
To get started, check out documentation for SQL Server 2019.
In conclusion
These three features have been the most common requests from our enterprise customers. Finally, you can bring your own Active Directory domain for SQL Server authentication and authorization, use the latest features from SQL Server 2019 and scale your read workloads as well as leveraging the cross regional replicas for faster disaster-recovery.
To get started, check out the documentation for Cross-Region Replica, Active Directory, and SQL Server 2019. All are available with any new instance created via the console or API, simply follow the instructions in the documentation.
Home Depot’s Interconnected Retail Experience by Virtue of Google Cloud Migration for SAP Applications
8281
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
With nearly 2,300 stores, The Home Depot is the world’s largest home-improvement chain — a brand that professional contractors and DIYers alike have come to depend on. The home improvement industry continues to experience unprecedented demand and dramatic increases in online ordering accompanied by expanding consumer expectations for things like curbside pickup and same day delivery. The Home Depot’s decision to migrate to cloud-based infrastructure, including the migration of the company’s SAP applications on Google Cloud which began in 2017, has set it up for success in an increasingly digital world, and helped the company adapt to changing market conditions quickly.
Interconnected retail at scale
Building on a strong customer-first philosophy, The Home Depot aims to create what it calls interconnected retail—allowing customers to shop however, whenever, and wherever they want. “So many companies are focused on omni-channel retail,” explains Sam Moses, Vice President of Corporate Systems. “At The Home Depot, we wanted to take it to the next level. Interconnected retail puts the customer at the center of everything and enables them to shop in store, online, or both. Customers can begin a transaction online and continue in-store, or vice-versa.”
To support this strategy, the company’s SAP environment needed to be more agile. Running everything on-premises, from central finance to POS systems, meant that The Home Depot’s IT teams experienced redundancy and repetitive, manual processes. Their data warehouse needed an upgrade to process and analyze growing and increasingly diverse data sets. The Home Depot chose to migrate its SAP environment to Google Cloud to support both the velocity and scale needed for the business as well as critical analytics capabilities needed for its bold digital initiatives. “We chose Google Cloud to support our SAP implementation. Our decision had a lot to do with the relationship between Google Cloud and SAP and also for the applications and services that are offered by Google Cloud, like BigQuery, which are helping to enable data and analytics within our organization,” Moses explains.
After migrating its SAP applications—including S/4HANA, its customer activity repository (CAR), general ledger, e-commerce system, enterprise data warehouse and more to Google Cloud, the company now has the speed, scale and flexibility to tackle enormous spikes in the business, all while staying fully available for their customers. Additionally, The Home Depot was able to transform its financial systems and make them more agile to deliver critical information across multiple business functions in real time.
Maximizing data insights to support customer experiences
By migrating to Google Cloud, The Home Depot is leveraging Google Cloud analytics to build the industry’s most efficient supply chain including more robust demand forecasting, supplier lead times, estimated delivery times and more, all while maintaining better security than before. “We experienced unprecedented change in our customers’ behavior and their buying patterns, which puts a lot of pressure on our supply chain,” explains Moses. “So having the ability to leverage data and analytics gives us insights to know exactly what it is that our customers need.”
The company’s analysts now use BigQuery ML for machine learning directly against the company’s BigQuery data and use AutoML to determine the best model for predictions. The Home Depot’s engineers have also adapted BigQuery to monitor, analyze, and act on application performance data across all its stores and warehouses in real time—capabilities that were not as seamless in the on-premises environment.
With hundreds of projects on Google Cloud, The Home Depot’s cloud journey is well on track, but the company is always looking to the future. “As our customers’ needs have continued to evolve, and as technology has continued to evolve, our relationship with Google will continue to advance — to be able to innovate together, to be able to find new solutions together, to better serve our customers.”
Learn more about how The Home Depot is renovating its retail operation with SAP on Google Cloud.
Serverless for Startups, the Best Way to Succeed: Expert Says
7052
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
As Google Cloud has become a choice for more startups, I’ve experienced an increase in founders asking how they should think about cloud services. Though each startup is different and requirements may vary across industries and regions, I’ve seen a few core best practices that help startups to succeed—as well as several traps to avoid.
For example, If you go with a public cloud provider, you’re ideally not starting from ground zero like you would running your own data center, but it’s important not to introduce similar complexity in a virtualized environment. Just because you’re using public cloud infrastructure doesn’t mean you want to manage it.
Instead, you want to leverage platforms that abstract away complexity so your team can focus on delivering value to customers. That’s where serverless comes in. Serverless platforms are fully managed by the provider, offering automatic scaling for workloads, as well as provisioning, configuring, patching, and management of servers and clusters. Freed from these resource-intensive tasks, your technical talent can focus on the things that differentiate your business, not on IT curation.
This applies to not only running stateless applications, but also managing and analyzing data. You should be collecting data points about which features are most popular on your platform, what people are buying, the types of activities that help your customers get their jobs done, and so on. This information is essential to building and executing on a product roadmap that will serve your customers. However, it’s not enough to collect this data, you need to make your data accessible, secure, and easy for your team to analyze—so where do you run and host it?
On Google Cloud, this is where options like Spanner and Cloud SQL can play a large role, as can BigQuery for analysis. You won’t have to worry about standing up infrastructure or patching servers—you can just stream your data to our data management platforms, where it’s available whenever someone needs to run a query. By leveraging a serverless architecture, your startup can be data-driven without having to invest in the traditional complexity of database administration and management—and that can significantly change your playing field.
Serverless is just one of the factors you should consider as you build out your tech stack. To hear my thoughts on a range of other topics relevant to startups — such as security, cloud credits, and the differences among managed services — check out the below video or visit our Build and Grow page.
How Connected-Stories Uses BigQuery and AI/ML to Craft Personalized Ad Experiences
4018
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery
In the field of producing engaging video content such as ads, many marketers ignore the power of data to improve their creative efforts to meet the consumers’ need for personalized messages. The demand for creative tech to efficiently personalize is real as marketers need personalized video Ads to reach their audience with the right message at the right time. Data, Insights and Technology are the main ingredients to deliver this value while ensuring security and privacy requirements are met. The Connected-Stories team partnered with Google Cloud to build a platform for Ad personalization. Google Data Cloud and BigQuery are at the forefront to assimilate data, leverage ML models, create personalized ads, and capitalize on real-time intelligence as the core features of the Connected-Stories NEXT platform.
Connected-Stories NEXT is an end-to-end creative management platform to develop, serve, and optimize interactive video and display ads that scale across any channel. The platform ingests first-party data to create custom ML models, measure numerous third-party data points to help brands develop unique customer journeys and create videos that their data signals can drive. An intelligent feedback loop passes real-time data back, enabling brands to make data-driven and actionable video ads that take the brand’s campaigns to the next level.
The core use case of the NEXT platform revolves around collecting user’s interaction data and optimizing for precision and speed to create an actionable Ad experience that is personalized for each user. The platform processes complex data points to create interactive data visualizations that allow for accurate analysis. The platform uses Vertex AI to access managed tools, workflows, and infrastructure to build, deploy, and scale ML models that have improved the accuracy to identify segments for further analysis.
The platform ingests 200M data events with peaks and valleys of activity. These events are processed to generate dashboards that enable users to visualize metrics based on filters in real-time. These dashboards have high performance requirements in terms of a responsive user interface under constantly changing data dimensions.
Google Cloud’s serverless stack coupled with limitless data cloud infrastructure has been the core to the NEXT platform’s data-driven innovation. The growing volume of data ingested, streamed and processed were scaled uniformly across the compute, storage and analytical layers of solution. A lean development team at Connected-Stories were able to focus all-in on the solution, while the serverless stack scaled, lowered attack service in terms of security and optimized the cost footprint through pay-as-you-go features.
BigQuery has been the backbone to support the vast amounts of data spreading over multiple geos resulting in workloads running at petabyte scale. BigQuery’s fully managed serverless architecture, real-time streaming, built-in machine learning and rich business intelligence capabilities distinguishes itself from a cloud data warehouse. It is the foundation needed to approach data and serve users in an unlimited number of ways. For an application with zero tolerance for failure, given its fully managed nature, BigQuery handles replication, recovery, data distributed optimization and management.
The platform’s requirements include the need for low maintenance, constantly ingesting and refreshing data and smart-tuning of aggregated data. These capabilities can be implemented by BigQuery’s materialized views feature. Materialized views are useful for precomputed views that regularly cache query results for better performance. These views possess the innate feature to read only the delta change from base tables and calculate the up-to-date aggregations. Materialized views impart faster outputs and consume fewer resources while reducing the cost footprint.
Some key considerations in using Google cloud and focusing on the Serverless stack include: quick onboarding to development, prototyping in short sprints and ease of preparing data in a rapidly changing environment. Typical considerations around low code / no code include data transformation, aggregation and reduced deployment time. These considerations are fulfilled through using serverless capabilities within Google Cloud such as PubSub, Cloud Storage, Cloud Run, Cloud Composer, Dataflow and BigQuery as described in the Architecture diagram below. The use of each of these components and services are described below.
- Input/Ingest: At a high-level, microservices hosted in Cloud Run collect and aggregate incoming Ads events.
- Enrichment: The output of this stage is a Pub-Sub message enriched with more attributes based on a pre-configured campaign.
- Store: a Cloud Dataflow streaming job to create text files in Cloud Storage buckets.
- Trigger: Cloud Composer triggers the spark jobs based on text files to process and group them to produce desired output as one record per impression, a logical group of events.
- Deploy: Cloud Build is then used to automate all deployments.
Thus far, all Google cloud managed services work together to ingest, store and trigger the orchestration, all of which are scalable based on configurations including autoscaling capabilities.
- Visualization: A visualization tool reads data from BigQuery to compute pre-aggregations required for each dashboard.
- Data Model Evolution considerations: Though the solution served the purpose of creating pre-aggregations, as the data model evolved by adding a column or creating a new table, it led to recreating pre-aggregations and querying the data again. Alternatively, creating aggregate tables as an extra output of current ETLs seemed like a viable option. However, this would increase the cost and complexity of jobs. A similar situation to reprocess or update aggregated tables would occur as data is updated.
Precomputed views of data that is periodically cached are critical to reach the audience with the right message at the right time.
- Performance: In order to increase the performance of the platform, we need to have regularly precomputed views of the data, cached .
- Materialized Views: Consumers of these views needed faster response times, to consume fewer resources and output only the changes in comparison to a base table. BigQuery Materialized views were used to solve this very requirement. Materialized views have been highly leveraged to optimize the design resulting in lesser maintenance and access to fresh data with high performance with a relatively low technical investment in creating and maintaining SQL code.
- Dashboards: Application dashboards pointing to the Materialized views are highly performant and provide a view into fresh data.
- Custom Reports with Vertex AI Notebooks: Vertex AI notebooks directly read data from BigQuery to produce custom reports for a subset of customers. Vertex AI has been hugely beneficial to data analysts, where an environment with pre-installed libraries simplifies the readiness to use. Vertex AI Workbench notebooks are used to share these reports within the team allowing them to work always on the cloud without having the need to download data at any time. Besides, it increases the velocity to develop and test ML models faster.
The NEXT platform has yielded benefits such as customers having the ability to create unique consumer journeys powered by AI / ML personalization triggers, using first-party data and business intelligence tools to capitalize on real-time creative intelligence, which is a dashboard to measure campaign performance for cross-functional teams to analyze the impact of Ad content experience at a granular level. All of these while ensuring controlled access to data to enrich data without moving across clouds. The NEXT platform can keep up with increased demands for agility, scalability and reliability through the underlying usage of Google Cloud.
Partnering with Google, in the context of the Google Built with BigQuery program has surfaced the differentiated value in areas of creating interactive personalized Ads by using real-time data. In addition, by sharing this data across organizations as assets, ML models have fueled higher levels of innovation. Connected-Stories plan to deepen the penetration into the entire spectrum of services offered in the AI/ML area to enhance core functionality and provide newer capabilities to the platform.
Click here to learn more about Connected-Stories NEXT Platform capabilities.
The Built with BigQuery Advantage for ISVs
Through Built with BigQuery, launched in April ‘22 as part of Google Data Cloud Summit, Google is helping tech companies like Connected-Stories co-innovate in building applications that leverage Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs. Participating companies can:
- Get started fast with a Google-funded, pre-configured sandbox.
- Accelerate product design and architecture through access to designated technical experts from the ISV Center of Excellence who can share insights from key use cases, architectural patterns, and best practices encountered in the field.
- Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.
The Google Data Cloud spectrum of products and specifically BigQuery give ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge and expanding partner ecosystem and support for multi-cloud, open source tools and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in and exercise choice.
We thank the Google Cloud and Connected-Stories team members who co-authored the blog: Connected-Stories: Luna Catini, Marketing Director, Google: Sujit Khasnis, Cloud Partner Engineering
Gartner Identifies Critical Capabilities for Data Management Solutions
DOWNLOAD WHITEPAPER4001
Of your peers have already downloaded this article
3:30 Minutes
The most insightful time you'll spend today!
Data management solutions for analytics offerings are consolidating, with major vendors able to address a range of use cases and smaller vendors addressing a subset of use cases. Data and analytics leaders can use this research to guide evaluation and initial vendor selection for DMSA offerings.
For data and analytics leaders responsible for data management solutions as part of strategizing and planning information infrastructure should:
- Evaluate the capabilities of incumbent solution(s) against new use cases, to determine if existing expertise could be used to reduce development time with a good-enough solution already in place.
- Plan on using a heterogeneous solution landscape overall, but try and reduce duplication of effort by categorizing use cases with regard to their target deployment platform.
- Use a logical data warehouse architecture when you need to integrate separate data repositories efficiently, keeping in mind performance SLAs that may be impacted by remote access.
- Plan for eventual integration with other data silos when scoping the effort needed to implement a specific solution, to avoid crippling overhead caused by proliferating data silos.
Download this report to know more.
More Relevant Stories for Your Company
Over 700 SaaS Companies Trust Google’s Data Cloud to Build Intelligent Apps!
Today, a typical enterprise uses over 100 SaaS applications while large organizations commonly use over 400 apps. These applications contain valuable data on customers, suppliers, employees, products and more, offering the potential for valuable insights and powerful workflows. However, in the past, this data has often remained siloed and underutilized,
Quick Watch on Six Database Trends
In a data-driven, global, always-on world, databases are the engines that let businesses innovate and transform. As databases get more sophisticated and more organizations look for managed database services to handle infrastructure needs, there are a few key trends we’re seeing. Here’s what to watch.
The Query Execution Graph: Your Key to Better BigQuery Analytics
BigQuery offers strong query performance, but it is also a complex distributed system with many internal and external factors that can affect query speed. When your queries are running slower than expected or are slower than prior runs, understanding what happened can be a challenge. The query execution graph provides
Two Ways to Deploy SAP HANA System on Google Cloud
Many of the world’s leading companies run on SAP—and deploying it on Google Cloud extends the benefits of SAP even further. Migrating your current SAP S/4HANA deployment to Google Cloud—whether it resides on your company’s on-premises servers or another cloud service—provides your organization with a flexible virtualized architecture that lets
