Become Truly Data-driven with Unified Analytics Platform Built on Google Cloud
3674
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Every company these days is becoming a data company whether they know it or not. This results in preparing the company to create an ecosystem for data processing. Traditionally, organisations’ data ecosystems consisted of point solutions that used to provide data services. For example, one of the most common questions we get from customers is, “Do I need a data lake, or should I consider a data warehouse? Do you recommend I consider both?” Traditionally, these two architectures have been viewed as separate systems, applicable to specific data types and user skill sets. Increasingly, we are seeing a blurring of lines between data warehouses and data lakes, which provide customers with an opportunity to create a more comprehensive platform that gives them the best of both worlds.
What if we don’t need to compromise? And we create an end-to-end solution compromising the entire data management and processing stages, from data collection to serving. Thus, Data platforms are typically used to store vast amounts of data in varying formats and doing so without compromising on latency. At the same time, providing a platform for all users throughout the data lifecycle.
Emerging Trends
There are data solutions and architectures we’re seeing and anticipating. Emerging concepts include data lakehouses, data meshes, and data vaults. Some are not new and have been around in different shapes and formats, however, all of them work naturally within a Google Cloud environment. Lets look into both ends of the spectrum of enabling data and enabling teams.
Data mesh facilitates a decentralized approach to data ownership, allowing individual lines of business to publish and subscribe to data in a standardized manner, instead of forcing data access and stewardship through a single, centralized team. On the other hand, a data lake house brings raw and processed data closer together, allowing for a more streamlined and centralized repository of data needed throughout the organization. Processing can be done in transit via ELT, reducing the need to copy datasets across systems. This allows for easier data exploration and easier governance. Data vault is designed to separate data driven and model driven parts, so the way data is integrated in the raw vault enables parallel loading so that large implementations can scale out easily.
In Google Cloud, there is no need to keep them separate. In fact, with interoperability among our portfolio of data analytics products, you can easily provide access to data residing in different places, effectively bringing your data lake and data warehouse together on a single platform.
Let’s look at some of the technological innovations that make this reality. BigQuery’s storage API allows treating a data warehouse like a data lake, letting you access the data residing in BigQuery. For example, you can use Spark to access data residing in the data warehouse without it affecting performance of any other jobs accessing it. This is all made possible by the underlying architecture, which separates compute and storage.
We continue to offer specialized products and solutions around data lake and data warehouse functionality but over time we expect to see a significant enough convergence of the two systems that the terminology will change. At Google Cloud, we consider this combination an “analytics data platform”.
Tactical or Strategical
Key differentiators of Google Cloud’s data analytics platform are being open, intelligent, flexible, and tightly integrated. There are many technologies in the market which provide tactical solutions that may feel comfortable and familiar. However, this can be a rather short-term approach that simply lifts and shifts a siloed solution into the cloud. In contrast, an analytics data platform built on Google Cloud provides modern data warehousing and data lake capabilities with close integration to our AI Platform. It also provides built-in streaming, ML, and geospatial capabilities and an in-memory solution for BI use cases. Depending on your organizational data needs, Google Cloud has the set of products, tools, and services to create the right data platform for you.
To become a truly data-driven organization, the first step is to design and implement an analytics data platform that meets your technical and business needs. Whether you want to empower teams to own, publish, and share their data across the organization, or you want to create a streamlined store of raw and processed data for easier discovery, there is a solution that best meets the needs of your company.
To learn more about the elements of a unified analytics data platform built on Google Cloud, and the differences in platform architectures and organizational structures, read our Unified Analytics Platform paper.
How We Built a Brand New Bank on Google Cloud and Cloud Spanner: The First Scalable, Enterprise-grade, Database Service
6567
Of your peers have already read this article.
6:30 Minutes
The most insightful time you'll spend today!
Editor’s note: Technology today lets companies of any size take on entire industries simply with an innovative business model plus digital distribution. Take Shine, a French startup whose platform helps freelancers manage their finances — and their administrative commitments. Here, Raphael Simon, Shine’s CTO and co-founder, talks about why Shine built a new bank on Google Cloud Platform, and in particular Cloud Spanner.
More and more people are deciding to take the plunge and start a freelance career. Some of them by choice, others out of necessity. One of their biggest pain points is dealing with administrative tasks.
In some countries, especially in Europe, the administrative burden of being a freelancer is similar to what a company of 10 or more people deals with. A freelancer doesn’t necessarily have the time or skills to manage all this paperwork. So we are building a new bank for freelancers from the ground up that helps automate administrative tasks associated with their business.
Shine’s banking services and financial tools make it as easy to work as a freelancer as it is to work for a larger company. We deal with administrative tasks on behalf of the freelancer so that he or she can focus on their job: finding and wowing clients.
Building our infrastructure
As a new bank, we had the opportunity to build our infrastructure from the ground up. Designing an infrastructure and choosing a database presents tough decisions, especially in the financial services world. Financial institutions come under tremendous scrutiny to demonstrate stability and security. Even a tiny leak of banking data can have tremendous consequences both for the bank and its clients, and any service interruption can trigger a banking license to be suspended or a transaction to be declined.
At the same time, it’s vital for us to optimize our resources so we can maximize the time we spend developing user-facing features. In our first six months, we iterated and validated a prototype app using Firebase, and secured our seed funding round (one of the largest in Europe in 2017).
Based on our positive experience with Firebase, plus the ease-of-use and attractive pricing that Google Cloud offered, we decided to build our platform on Google Cloud Platform (GCP).
We were drawn to GCP because it has a simple, consistent interface that is easy to learn. We chose App Engine flexible environment with Google Cloud Endpoints for an auto-scaling microservices API. These helped us reduce the time, effort, and cost in terms of DevOps engineers, so we could invest more in developing features, while maintaining our agility.
We use Cloud Identity and Access Management (Cloud IAM) to help control developer access to critical parts of the application such as customer bank account data. It was quite a relief to lean on a reliable partner like Google Cloud for this.
Database decisions
Next came time to choose a database. Shine lives at the financial heart of our customers’ businesses and provides guidance on things like accounting and tax declaration. The app calculates the VAT for each invoice and forecasts the charges they must pay each quarter.
Due to the sensitivity of our customers’ data, the stakes are high. We pay careful attention to data integrity and availability and only a relational database with support for ACID transactions (Atomicity, Consistency, Isolation, Durability) can meet this requirement.
At the same time, we wanted to focus on the app and user experience, not on database administration or scalability issues. We’re trying to build the best possible product for our users, and administering a database has no direct value for our customers. In other words, we wanted a managed service.
Cloud Spanner combines a globally distributed relational database service with ACID transactions, industry-standard SQL semantics, horizontal scaling, and high availability. Cloud Spanner provided additional security, high-availability, and disaster recovery features out-of-the-box that would have taken months for us to implement on our own. Oh, and no need to worry about performance — Cloud Spanner is fast. Indeed, Cloud Spanner has been a real asset to the project, from the ease-of-use of creating an instance to scaling the database.
Cloud Spanner pro tips
We began working with Cloud Spanner and have learned a lot along the way. Here are some technical notes about our deployment and some best practices that may be useful to you down the road:
- Cloud Spanner allows us to change a schema in production without downtime. We always use a NOT NULL constraint, because we generally think that using NULL leads to more errors in application code. We always use a default value when we create an entity through our APIs and we use Cloud Dataflow to set values when we change a schema (e.g., adding a field to an entity).
- With microservices, it’s generally a good practice to make sure every service has its own database to ensure data isolation between the different services. However, we adopted a slightly different strategy to optimize our use of Cloud Spanner. We have an instance on which there are three databases — one for production, one for staging and one for testing our continuous integration (CI) pipeline. Each service has one or more interleaved tables that are isolated from others services’ tables (we do not use foreign-keys between tables from different services). This way our microservices data are not tightly “coupled”.
- We created an internal query service that performs read-only queries to Cloud Spanner to generate a dashboard or do complex queries for analytics. It is the only service where we allow joins between tables across services.
- We take advantage of Cloud Spanner’s scalability, and thus don’t delete any data that could one day be useful and/or profitable.
- We store all of our business logs on Cloud Spanner, for example connection attempts to the application. We append the ‘-Logs’ suffix to them.
- When possible, we always create an interleave.
In short, implementing Cloud Spanner has been a good choice for Shine:
- It’s saved us weeks, if not months, of coding.
- We feel we can rely on it since it’s been battle-tested by Google.
- We can focus on building a disruptive financial services product for freelancers and SMBs.
And because Cloud Spanner is fully managed and horizontally scalable, we don’t have to worry about hardware, security patches, scaling, database sharding, or the possibility of a long and risky database migration in the future. We are confident Cloud Spanner will grow with our business, particularly as we expand regionally and globally. I strongly recommend Cloud Spanner to any company looking for a complete database solution for business-critical, sensitive, and scalable data.
Lending DocAI Shortens Borrowers’ Journey on Roostify
11828
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
The home lending journey entails processing an immense number of documents daily from hundreds of thousands of borrowers. Currently, home lending document processing relies on some outdated digital models and a high dependency on manual labor, resulting in slow processing times and higher origination costs. Scaling a business that sorts through millions of documents daily, while increasing efficacy and accuracy, is no small feat. When it comes to applying for a mortgage loan, consumers expect a digital experience that’s as good as the in-person one. Roostify simplifies the home lending journey for lenders and their customers.
No time to spare: Overcoming document processing challenges with AI
Roostify provides enterprise cloud applications for mortgage and home lenders. In order to empower its customers to deliver a better, more personalized lending experience, they needed to automate and scale their in-house document parsing functionality.
As a key component of its document intelligence service, Roostify is leveraging Google Cloud’s Lending DocAI machine learning platform to automate processing documents required during a home loan application process, such as tax returns or bank statements with multi-language support. This partnership delivers data capture at scale, enabling Roostify customers to automatically identify document types from the uploaded file and to extract relevant entities such as wages, tax liabilities, names, and ID numbers for further processing, and make things move faster in the cumbersome lending process.
Roostify’s solutions leverage Google Cloud’s Lending DocAI, which is built on the recently announced Document AI platform, a unified console for document processing. Customers can easily create and customize all the specialized parsers (e.g., mortgage lending documents and tax returns parsers) on the platform without the need to perform additional data mapping or training. All Google Cloud’s specialized parsers are fine-tuned to achieve industry-leading accuracy, helping customers and partners confidently unlock insights from documents with machine learning. Learn more about the solution from the GA launch blog and the overview video.
Integrating Lending DocAI’s intelligent document processing capabilities into the Roostify platform means more innovation for their customers and tangible results: faster loan processing times, fewer document intake errors, and lower origination costs. Additional support in Google Lending DAI for other languages and more documents like global Know Your Customer (KYC) documents or payroll reports is in the near future.
Full integration of AI solutions
Working together with Roostify’s platform team, we were able to help them solve their document processing challenge through integration of various GCP products such as Lending DocAI (LDAI), Data Loss Prevention (DLP) for redacting sensitive data, BigQuery for data warehousing and analytics, and Firestore for API status. To make it very safe and secure, all data was encrypted end-to-end at Rest and in Transit. LDAI won’t require any training data to process. It is an easy plug and play API.
Here is a sneak peek in the high level deployment architecture for LDAI in Roostify environment:
Here are the steps for processing data:
- Receives document processing request from the client.
- API Function directs requests to the pre-processing service. For Async requests a processing ID is generated and returned to the caller.
- Pre-processing service sends the request for further processing (Long/short PDF conversion), calling other microservices and receives back the responses. Any error in the response received is then sent to the response processing service.
- If the response is synchronous, the pre-processing service directs it to the LDAI Invoker service.
- If the response is asynchronous, the pre-processing service feeds it into the Cloud Pub/Sub service.
- Cloud Pub/Sub service feeds the response back to the LDAI Invoker service.
- LDAI Invoker service routes the request to the Google LDAI API for classification if there are multiple pages in the document.
- Document will be split based on LDAI response and then saved in a GCS bucket for temporary storage.
- LDAI entity interface for single page processing and then LDAI Invoker sends LDAI results to LDAI Response Processing
- If a request is a synchronous request the LDAI Response Processor sends results to the API Function so that it can complete the synchronous call and respond to the rConnect caller.
- If the request is an asynchronous request the LDAI Response Processor will respond to the caller’s webhook and complete the transaction.
- Finally, Data stored in the GCP bucket will be deleted.
All the responses that come from the LDAI API can optionally feed into BigQuery via the Response Processor, after parsing it through Data Loss Prevention (DLP) API to redact the PII/sensitive information. Throughout the processing of both asynchronous and synchronous requests all transactions are logged using Cloud Logging. For asynchronous transactions, the state is maintained throughout the process using Cloud Firestore.
Roostify currently uses this technology to power two different solutions: Roostify Document Intelligence and Roostify Beyond™. Roostify Document Intelligence is a real-time document capture, classification, and data extraction solution built for home lenders. It ingests documents uploaded by borrowers and loan officers, identifies the relevant documents, and extracts and classifies key information. Roostify Document Intelligence is available as a standalone API service to any home lender with any digital lending infrastructure already in place.
Roostify Beyond™ is a robust suite of AI-powered solutions that enables home lenders to create intelligent experiences from start to close. It combines powerful data, insightful analytics, and meaningful visualization to streamline the underwriting process. Roostify Beyond™ is currently available only to Roostify customers as part of an Early Adopter program and will be rolled out to the market later this year.
Through this partnership, Roostify has enabled its customers to adopt a data-first approach to their home lending processes, which will lead to improved user experiences and significantly reduced loan processing times.
Fast track end-to-end deployment with Google Cloud AI Services (AIS)
Google AIS (Professional Services Organization), in collaboration with our partner Quantiphi, helped Roostify deploy this system into production and fast-tracked the development multifold to generate the final business value.
The partnership between Google Cloud and Roostify is just one of the latest examples of how we’re providing AI-powered solutions to solve business problems.
4028
Of your peers have already watched this video.
54:30 Minutes
The most insightful time you'll spend today!
Quick Tips to Get the Most Out of Cloud Spanner
Today, IT Admins and DBAs are inundated with thankless tasks. But they no longer have to deal with that. With Cloud Spanner, they can focus on value-add and innovation instead of maintenance. Creating or scaling a globally replicated database for mission-critical apps now takes a handful of clicks.
Industry-leading high-availability and Google-grade security as defaults, not expensive add-ons, help ensure your apps stay online and more secure.
Cloud Spanner is a powerful product, but many users do not maximize its benefits.
It is the first scalable, enterprise-grade, globally-distributed, and strongly consistent database service built for the cloud specifically to combine the benefits of relational database structure with a non-relational horizontal scale.
This combination delivers high-performance transactions and strong consistency across rows, regions, and continents with an industry-leading 99.999% availability SLA, no planned downtime, and enterprise-grade security. Cloud Spanner revolutionizes database administration and management and makes application development more efficient.
This video highlights best practices, strategies for optimizing applications and workloads, and ways to improve performance and scalability. This live demos shows real-time speed-ups of transactions, queries, and overall performance.
Additionally, this talk explores techniques for monitoring Cloud Spanner to identify performance bottlenecks. Watch now to learn more.
Qlik and Google Cloud Combo: Extending Integration of SAP Data on BigQuery
5476
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
If your organization is one of the 52% of SAP customers whose top analytics pain point is data integration1, Google Cloud has got you covered. By working with partners like Qlik, we are expanding our integration options and bringing real-time replication capability for SAP to BigQuery.
Integrated data for accelerated insights
BigQuery—our fully managed, enterprise data warehouse that can scale up to petabytes on demand and execute queries in seconds—allows SAP customers to consolidate enterprise data silos and confidently derive more use and value from their data. Customers can accelerate and simplify the delivery of SAP data on BigQuery with the latest Qlik Data Integration platform offering for Google Cloud which allows data integration using an automated, near real-time data pipeline through Qlik Replicate, whether data originates from legacy SAP environments, SAP HANA, or SAP application servers. Additionally, Qlik Compose for Data Warehouses can be used to easily generate and automate logical data models from SAP directly in BigQuery freeing up more time for data analysts to leverage advanced built-in capabilities such as BigQuery ML to derive greater value and insights using standard SQL without the need for advanced programming expertise.
Delivering faster results with real-time
Traditional extract-transform-load (ETL) solutions operate on a batch basis, pulling data sets from SAP daily, hourly, or every minute. These tools often require manual mapping of multiple data fields so that data flows accurately. Given SAP’s highly complex table relationships, in which a single transaction can result in multiple changes, this can be a time consuming and tedious process. ETL can also increase the burden on your SAP systems.
Qlik Replicate simplifies this with its intuitive user interface where you can set up real-time data replication between SAP and BigQuery, eliminating the need for manual coding. And, to prevent system overhead, as soon as a new transaction is entered into SAP, the resulting data is replicated into BigQuery in a process known as change data capture (CDC) from SAP’s log layer. This means that data transfer can benefit from high performance with minimal impact on the source system’s resources. Read our latest white paper to learn how to extract SAP data into BigQuery leveraging Qlik Replicate.
Solution expertise for all core SAP workloads
It doesn’t matter what database your SAP system runs on, Qlik Replicate supports all core SAP systems. It automates real-time data replication and decodes SAP’s complex, application-specific data structures into formats that flow smoothly into BigQuery. This ensures that anyone who depends on data analytics has the most current and relevant SAP data they need. For its robust solution capabilities Qlik has received a new “SAP on Google Cloud Expertise” designation for supporting:
- Fast onboarding and accelerated replication of SAP data into Google Cloud
- Real-time and continuous data replication from SAP applications to BigQuery
- Support for all core SAP modules and a broad set of data sources
- Automated data integration, which cuts resource requirements for initial delivery and ongoing maintenance
Proven customer results
Many SAP customers have experienced the benefits of leveraging Qlik alongside BigQuery for data analytics and AI at scale, including German luxury department store chain Breuninger. In order to meet its customers’ growing and changing expectations,Breuninger needed to accelerate its time to insight from data sources across a highly dispersed landscape of on-premises databases and systems, including SAP. The company uses Qlik Replicate to feed corporate data from modules in its SAP system into BigQuery and integrate its varying on-premises databases with the Google Cloud environment. This has yielded game-changing, real-time customer insights for the retailer.
What could your business do with faster, more integrated insights? Learn more about BigQuery for SAP customers and also how Qlik and Google Cloud can help you modernize and automate data integration and analytics.
Making Mothers’ Day Special: How Google Cloud Migration for 1-800-FLOWERS.COM, Inc Impacts CX
6851
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Editor’s note: In honor of Mother’s Day, we look at how 1-800-FLOWERS.COM, Inc. migrated to Google Cloud as part of its digital transformation to quickly deploy seamless and convenient customer experiences across multiple brands on Mother’s Day and every day.
As a leading provider of gifts designed to help customers express, connect, and celebrate, 1-800-FLOWERS.COM, Inc. has embraced cloud technologies to grow and transform its business through constant innovation. As part of our digital transformation, we recently completed the migration of our ecommerce platform and other services to Google Cloud. We’ve transitioned from a monolithic to a microservices platform, moved many workloads from our on-premises data centers to our Google Cloud environment, and scaled both horizontally and vertically.
Since our migration, we’ve developed efficient processes to launch new brands, improved the customer experience across all brands, and seen significantly increased site traffic.
Nurturing a more delightful customer journey
Customer delight is at the core of everything we do. Whether it be with a flower bouquet, a sweet treat, or a personalized keepsake, our mission is to deliver smiles. With the rise of the COVID-19 pandemic, we’ve all been challenged to find unique and safe ways to continue honoring the special connections in our lives and celebrating occasions with loved ones. Our customers have adapted by doing things such as sending gifts to isolated loved ones, sharing the same meal together virtually, or using video to engage with others through group activities like flower arranging and building charcuterie boards.
The customer experience is a top priority for us, and we constantly look for innovative ways to enhance the customer journey across our ecommerce platform of more than a dozen brands. As a result, we’ve continued to see a rise in demand as customers enjoy the ease and convenience of our site and discover our full family of brands.
Migrating to a cloud-first mindset
As we’ve continued to innovate and iterate on the customer experience, we knew we wanted to evolve our platform. We wanted to shift to a microservices platform, which would allow our team the opportunity to release updates to our site more often and set up the right continuous integration/continuous deployment (CI/CD) practices.
Working with Google Cloud, we were able to move our ecommerce platform to the cloud and standardize our site and brand deployment by building one release that could then be repeated across all of our brands. We built everything in a modular fashion, including microservices and code libraries, so that sites could be easily constructed and replicated for each brand. And because of this, we were able to launch Shari’s Berries extremely quickly after we acquired the brand in 2019.
Moving our platform to a completely homegrown solution of microservices was a daunting task. But our team handled it beautifully through load-testing, stress-testing, and building new monitoring tools. And with Google Cloud supporting us all along the way, managing the migration process was simple and easy from start to finish.
Arranging a better bouquet of services
Currently, we’ve migrated every customer-facing touchpoint for all of our brands to Google Cloud—whether it’s on the web or mobile, our AI bots, or our chat interfaces.
- We run on Google Kubernetes Engine and Istio.
- We have nearly 200 microservices built to help power our entire ecommerce stack across several cloud services running on Google Cloud.
- We’re utilizing BigQuery for our offline intelligence.
Results are coming up roses
Our new stack on Google Cloud has benefits for both us and our customers. We moved from a session-based to a token-based system, which provides enhanced security as well as a consistent, convenient experience across all our brands. Using service workers and a single-page app, we are able to download all the relevant site content to the browser in under two seconds to create an instant-click experience for each and every customer. We also use Google Analytics to measure our user interactions and provide personalized results to each customer. Our hope is that with this new system, we can learn from customer behavior to offer gift givers a more personalized shopping experience during each visit.
The benefits of our new tech stack have not only helped us enhance the solutions we offer to customers today, they’ve also enabled us to offer new ones at lightning speed. With our legacy system, we used to release new code once a week or once a month. Now, even during our peak periods, we’re able to release 10 to 15 times a day and can deploy and pivot quickly to create new microservices and microsites on the fly—often without having to touch any code.
Efficiencies abound
The benefits of moving our platform to Google Cloud have extended to our internal teams as well. Before the migration, we had only two environments for developing and testing, which made it time-consuming to test updates before they went into production. Now with Google Cloud, we have several different journey teams—which are made up of developers, product owners, and technical owners—all working in several different environments, solving problems, and creating new solutions together.
Everyone is now empowered to be self-sufficient, developing and releasing microservices on their own when they’re ready. This has given our developers more time to take part in continued development and learning opportunities. For example, we offer lunch-and-learn sessions as well as other resources for everyone to take advantage of so they can continue to learn and refine their skills.
Planting the seeds for future growth
As we look to the future and think about how we help our customers express, connect, and celebrate, we’ll continue to collaborate across teams to deliver solutions that spread smiles. Specifically, we’re exploring additional use of AI to help us better serve our customers across all our brands.
We’ve enjoyed the ongoing support we’ve received from the Google Cloud team as they help us build new solutions and design a road map for the future. Their support has helped the 1-800-FLOWERS.COM, Inc. team to realize the power of the cloud and bring the very best experience to our customers.
Learn more about 1-800-FLOWERS.COM, Inc., or check out our recent blog about cloud migration for the real world.
More Relevant Stories for Your Company
Meesho’s Zero Downtime Success: Cloud CDN Migration Made Easy
Meesho is an Indian online marketplace that serves millions of customers every day. Recently, the company decided to adopt a multi-cloud strategy, leveraging Google Cloud’s scalable and reliable infrastructure to drive operational efficiency, modernize and scale for growth. To do so, they needed to migrate billions of static files and images
Pega Systems Migrates SAP Servers to Google Cloud in Just 9 Weeks!
Pega Systems' financial data on SAP environs were on a hosting platform that lacked agility. By moving nearly 30 SAP servers to Google Cloud in just 9 weeks, Pega Systems was able to unlock data and integrate BigQuery into SAP HANA to deliver personalization for clients and embark on an
Google Leads the Pack in Big Data NoSQL Market: Forrester Research
NoSQL has become critical for all businesses to support modern business applications. It has gone from supporting simple schemaless apps to becoming a mission-critical data platform for large Fortune 1000 companies. It has already disrupted the database market, which was dominated for decades by relational database vendors. Today, half of
Ahead of the Curve: 5 Data and AI Trends Set to Shape 2023
How will your organization manage this year's data growth and business requirements? Your actions and strategies involving data and AI will improve or undermine your organization's competitiveness in the months and years to come. Our teams at Google Cloud have an eye on the future as we evolve our strategies
