Cloud Bigtable brings database stability and performance to Precognitive - Build What's Next
Case Study

Cloud Bigtable brings database stability and performance to Precognitive

5725

Of your peers have already read this article.

3:40 Minutes

The most insightful time you'll spend today!

Precognitive develops technology to interpret data to improve the accuracy of fraud detection and prevention, to reduce false positives and customer disruption. Read how the quest for the right database led them to Cloud Bigtable.

At Precognitive, we were able to start with a blank technology slate to support our fraud detection software products. When we started building the initial version of our platform in 2017, we had some decisions to make: What coding language to use? What cloud infrastructure provider to choose? What database to use? The majority of the decisions were straightforward, but we struggled to decide upon a database. We had plenty of collective experience with relational databases, but not with a wide-column database like Cloud Bigtable—which we knew we’d need to scale our behavior and device workloads. At launch, our products were supported by a self-managed database, but we quickly migrated to Cloud Bigtable, and we love it.  

To efficiently support our bursty, real-time fraud detection workloads, we needed a cloud database that could satisfy the following key requirements:

  • Stability to keep up with increased adoption of our products
  • Intelligent scaling that avoids bottlenecks
  • Native integrations with BigQuery and Cloud Dataproc
  • Managed services that free up our engineers’ time to work on our products

Adding Cloud Bigtable as our performance database

As we scaled our services and added customers, our data collection services for our Device Intelligence and Behavioral Analytics products were seeing thousands of events per second. Cloud Bigtable provided a stable managed database that could handle the volume we were receiving during peak hours. We weren’t always able to handle this scale, as an early version of our product utilized a self-managed database.

Every month, two or three engineers spent hours managing the database instances. Whenever the instances crashed, it would cost at least one engineer a day or two of productivity attempting to restore the instances and recovering any data from our backup database. Managing this database internally was taking precious time away from product development.

We circled back to Cloud Bigtable. After two weeks of R&D, we decided to switch the Device Intelligence and Behavioral Analytics services to Cloud Bigtable.

Cloud Bigtable solved our scaling issues. Cloud Bigtable had been attractive to us from the start because it was fully managed, and offered regional replication and other features we were lacking in our own managed instances. Cloud Bigtable provides horizontal scaling and automatically rebalances row keys (equivalent to a shard key) over time to prevent “hot” nodes. In addition, Cloud Bigtable provides a connector to BigQuery and Cloud Dataproc that allows us to analyze the terabytes of data we are processing and use that data for unsupervised machine learning.

The perks of using Cloud Bigtable

After the migration to Cloud Bigtable, we noticed a number of additional benefits: improved I/O performance, a significant cost reduction, and a sizable decrease in hours spent on database maintenance.

We measured some of our typical metrics before and after implementing Cloud Bigtable. Our request latency dropped by about 30 ms on average (to sub-10 ms) for API requests. Prior to the change, we were seeing latencies of 40+ ms on average. This latency drop on our Behavioral Analytics and Device Intelligence products allowed us to trim about an additional 10 to 15 ms off our average response time across all dependent services.

cloud_bigtable_latency.png
By switching to Cloud Bigtable, we cut database infrastructure costs by approximately 35%.

Before we moved to Cloud Bigtable, we had to scale our database instances every time a new customer was onboarded. We were over-scaling in an attempt to avoid constantly resizing our database servers. By sunsetting our self-managed database and switching to Cloud Bigtable, we cut database infrastructure costs by approximately 35% and can now scale as needed, with a couple of clicks, during onboarding.

We have spent zero hours managing a Cloud Bigtable database since launch, and we put the time we are saving every month toward product development.

Moving forward with Cloud Bigtable

As an engineering team, we love working with Cloud Bigtable. We are not only seeing improved developer experience and reduced latency, which keeps the engineers happy, but also reduced costs, which keeps the business happy. We’re able to build more product, too, with the time we’ve saved by switching to Cloud Bigtable. Stay tuned to our engineering blog for more on the lessons we’ve learned and our contributions to the wider Cloud Bigtable community.

Case Study

How We Built a Brand New Bank on Google Cloud and Cloud Spanner: The First Scalable, Enterprise-grade, Database Service

6568

Of your peers have already read this article.

6:30 Minutes

The most insightful time you'll spend today!

Shine, a French startup whose platform helps freelancers manage their finances, shares how using Cloud Spanner saved it months of coding, allowed it to ensure security, availability, and scalability, and ensured it could focus on building a disruptive financial services product.

Editor’s note: Technology today lets companies of any size take on entire industries simply with an innovative business model plus digital distribution. Take Shine, a French startup whose platform helps freelancers manage their finances — and their administrative commitments. Here, Raphael Simon, Shine’s CTO and co-founder, talks about why Shine built a new bank on Google Cloud Platform, and in particular Cloud Spanner.

More and more people are deciding to take the plunge and start a freelance career. Some of them by choice, others out of necessity. One of their biggest pain points is dealing with administrative tasks.

In some countries, especially in Europe, the administrative burden of being a freelancer is similar to what a company of 10 or more people deals with. A freelancer doesn’t necessarily have the time or skills to manage all this paperwork. So we are building a new bank for freelancers from the ground up that helps automate administrative tasks associated with their business.

Shine’s banking services and financial tools make it as easy to work as a freelancer as it is to work for a larger company. We deal with administrative tasks on behalf of the freelancer so that he or she can focus on their job: finding and wowing clients.

image63w3u.PNG
image82pol.PNG

Building our infrastructure 

As a new bank, we had the opportunity to build our infrastructure from the ground up. Designing an infrastructure and choosing a database presents tough decisions, especially in the financial services world. Financial institutions come under tremendous scrutiny to demonstrate stability and security. Even a tiny leak of banking data can have tremendous consequences both for the bank and its clients, and any service interruption can trigger a banking license to be suspended or a transaction to be declined.

At the same time, it’s vital for us to optimize our resources so we can maximize the time we spend developing user-facing features. In our first six months, we iterated and validated a prototype app using Firebase, and secured our seed funding round (one of the largest in Europe in 2017).

Based on our positive experience with Firebase, plus the ease-of-use and attractive pricing that Google Cloud offered, we decided to build our platform on Google Cloud Platform (GCP).

We were drawn to GCP because it has a simple, consistent interface that is easy to learn. We chose App Engine flexible environment with Google Cloud Endpoints for an auto-scaling microservices API. These helped us reduce the time, effort, and cost in terms of DevOps engineers, so we could invest more in developing features, while maintaining our agility.

We use Cloud Identity and Access Management (Cloud IAM) to help control developer access to critical parts of the application such as customer bank account data. It was quite a relief to lean on a reliable partner like Google Cloud for this.

Database decisions 

Next came time to choose a database. Shine lives at the financial heart of our customers’ businesses and provides guidance on things like accounting and tax declaration. The app calculates the VAT for each invoice and forecasts the charges they must pay each quarter.

Due to the sensitivity of our customers’ data, the stakes are high. We pay careful attention to data integrity and availability and only a relational database with support for ACID transactions (Atomicity, Consistency, Isolation, Durability) can meet this requirement.

At the same time, we wanted to focus on the app and user experience, not on database administration or scalability issues. We’re trying to build the best possible product for our users, and administering a database has no direct value for our customers. In other words, we wanted a managed service.

Cloud Spanner combines a globally distributed relational database service with ACID transactions, industry-standard SQL semantics, horizontal scaling, and high availability. Cloud Spanner provided additional security, high-availability, and disaster recovery features out-of-the-box that would have taken months for us to implement on our own. Oh, and no need to worry about performance — Cloud Spanner is fast. Indeed, Cloud Spanner has been a real asset to the project, from the ease-of-use of creating an instance to scaling the database.

Cloud Spanner pro tips 

We began working with Cloud Spanner and have learned a lot along the way. Here are some technical notes about our deployment and some best practices that may be useful to you down the road:

  • Cloud Spanner allows us to change a schema in production without downtime. We always use a NOT NULL constraint, because we generally think that using NULL leads to more errors in application code. We always use a default value when we create an entity through our APIs and we use Cloud Dataflow to set values when we change a schema (e.g., adding a field to an entity). 
  • With microservices, it’s generally a good practice to make sure every service has its own database to ensure data isolation between the different services. However, we adopted a slightly different strategy to optimize our use of Cloud Spanner. We have an instance on which there are three databases — one for production, one for staging and one for testing our continuous integration (CI) pipeline. Each service has one or more interleaved tables that are isolated from others services’ tables (we do not use foreign-keys between tables from different services). This way our microservices data are not tightly “coupled”. 
  • We created an internal query service that performs read-only queries to Cloud Spanner to generate a dashboard or do complex queries for analytics. It is the only service where we allow joins between tables across services. 
  • We take advantage of Cloud Spanner’s scalability, and thus don’t delete any data that could one day be useful and/or profitable. 
  • We store all of our business logs on Cloud Spanner, for example connection attempts to the application. We append the ‘-Logs’ suffix to them. 
  • When possible, we always create an interleave. 

 In short, implementing Cloud Spanner has been a good choice for Shine:

  • It’s saved us weeks, if not months, of coding. 
  • We feel we can rely on it since it’s been battle-tested by Google. 
  • We can focus on building a disruptive financial services product for freelancers and SMBs.

And because Cloud Spanner is fully managed and horizontally scalable, we don’t have to worry about hardware, security patches, scaling, database sharding, or the possibility of a long and risky database migration in the future. We are confident Cloud Spanner will grow with our business, particularly as we expand regionally and globally. I strongly recommend Cloud Spanner to any company looking for a complete database solution for business-critical, sensitive, and scalable data.

4553

Of your peers have already watched this video.

3:30 Minutes

The most insightful time you'll spend today!

Trend Analysis

Strategies for Migrating to the Cloud

What are the technologies that are helping enterprises scale, adapt, and modernize? Are there any strategies that enterprises can adopt for moving to the cloud?

What this webinar to find out the different migration patterns to the cloud and learn how enterprises can choose the right strategies based on their business and technical environments, and the tooling that can help them get there.

Blog

NVIDIA CloudXR Streaming from Cloud to Transform Gaming and Enterprise AR/VR Experience

3298

Of your peers have already read this article.

3:00 Minutes

The most insightful time you'll spend today!

Experience high-fidelity AR and VR applications from anywhere with just good internet connection! NVIDIA CloudXR combines their powerful GPUs running NVIDIA RTX Virtual Workstations (vWS) in Google Cloud data centers. Learn more!

The Opportunity for Streamed AR/VR Content 

What if you could get a high quality AR/VR experience without a dedicated physical computer—or even without a physical tether?  

In the past, interacting with VR required a dedicated, high-end workstation and, depending on the headset, wall-mounted sensors and a dedicated physical space. Complex tasks in VR can push the limits of sensor reach, cable-length, and spatial boundaries, entangling the artist and restricting their movement. This solution was not scalable beyond a handful of advanced use cases.

Recently, tetherless VR headsets from manufacturers such as HTC and Oculus have emerged that free the user from these odious restrictions, enabling a new freedom to experience VR and AR from just about anywhere. The enhanced portability and reduced cost has led to substantially increased adoption among consumers and opened up many new exploratory use cases in enterprise.

However, while these headsets are more accessible and portable, a tradeoff in compute power was required in order to achieve these goals. The limited on-device compute power of this new generation of Head Mounted Displays (HMDs) is acceptable for many consumer applications such as casual gaming. However, advanced enterprise workloads with heavy graphics, compute, or memory requirements can be difficult or even impossible to run on these lightweight devices.

By combining NVIDIA CloudXR with their powerful GPUs running NVIDIA RTX Virtual Workstations (vWS) in Google Cloud data centers, you can experience high-fidelity VR and AR applications from just about anywhere with a good internet connection. The heavy computations are performed in the cloud on a GPU-attached VM and the content streams to any CloudXR compatible headset. 

The combination of Google Cloud’s private fiber optic network — the same network we built for global delivery of YouTube content — and CloudXR’s QoS technologies provides the user with the highest possible quality of service. In fact, the streaming experience is comparable to that of a headset tethered to a powerful physical workstation, but without the friction of hardware and cables. 

This combination of power and portability sets the stage to unlock the potential of high-quality gaming and enterprise AR/VR experiences anywhere on Earth.

Case Study: Creating a Masterpiece

Digital character creation is a core skill for many 3D artists today. One of the preferred methods of 3D character creation is digital sculpting, which lets artists create both hard-surface and organic shapes with high levels of accuracy and control.

Sculpting is one of the many tasks in the character designer’s skillset. Artists must also master the texturing, rigging, and posing of characters as part of the digital character creation pipeline. 

Mastering all these tasks can be challenging, and often, the technology gets in the way; working in multiple applications requires switching contexts and workflows, breaking the artist’s creative flow. The traditional user experience can also be unintuitive, forcing the creator to translate what they want to do with their hands and head into mouse movement and keyboard presses.

Masterpiece Studio Pro revolutionizes this character creation workflow by giving artists the first fully immersive 3D creation pipeline. Artists work in VR, giving them a far more intuitive and seamless way to work which combines the best of the digital and physical worlds.

In Masterpiece Studio Pro, the artist can work on the character or object at any scale, using familiar tools and hand gestures to sculpt a model, much as they would a real clay figure. Performing other tasks such as skeleton creation is simple, allowing the artist to work directly with the limbs of a character to place and adjust the joints.

Bringing It All Together

NVIDIA CloudXR streaming from the cloud provides tremendous opportunities for new creative use cases within gaming and across the enterprise. This solution joins Virtual Studio for Gaming as the latest in our series to help developers build better games.

The Masterpiece Studio use case is a powerful demonstration of new modalities for content creation and collaboration. To try CloudXR with NVIDIA RTX vWS on Google Cloud for yourself, see this tutorial. Masterpiece Studio Pro also offers extensive learning materials and a free trial.

Blog

Three Benefits of VPC Network Peering for SAP Managed Apps

3531

Of your peers have already read this article.

3:00 Minutes

The most insightful time you'll spend today!

Google Cloud's VPC Network peering capability to connect SAP managed environment with Google Cloud apps and services can strengthens security, and deliver benefits in three key areas. Read to learn how VPC Network peering enhances value of SAP apps!

Over the past year, RISE with SAP has emerged as a valuable solution for SAP customers seeking a faster, simpler, and more affordable path to the cloud. RISE with SAP is a subscription-based offering that typically includes a number of fully managed cloud application options, including SAP S/4HANA Cloud and elements of the SAP Business Technology Platform. It’s a great way to migrate your business to modern cloud ERP applications, while handing off the implementation and management to SAP and freeing your own IT staff to focus on other projects.

As a strategic partner for RISE with SAP, Google Cloud works closely with SAP to ensure reliable, secure, high-performance connectivity between your SAP managed applications and your other Google Cloud applications and services. Today, we’re focusing on a Google Cloud capability that plays a big part in achieving this goal: VPC network peering. If your company uses Google Cloud to host its RISE with SAP managed applications (or any other SAP-managed offering, such as SAP Enterprise Cloud Services), it’s worth learning more about why VPC network peering is an important capability and how your network admin can implement it.

We recently co-authored a white paper with SAP that goes into technical detail about how VPC network peering works and includes step-by-step configuration instructions. But it is useful to understand, at a higher level, why it matters for RISE with SAP subscribers.

When SAP implements your RISE with SAP managed applications on Google Cloud, it also provisions a virtual private cloud (VPC) — a virtualized network that provides the same functionality, performance, and security benefits as a dedicated physical network — for this SAP environment. In fact, a VPC actually gives you some capabilities that a physical network can’t match, such as the ability to increase IP space without downtime.

Using VPC network peering to connect your SAP managed environment with your Google Cloud applications and services gives you three key benefits: 

  • Network latency: Traffic that remains inside Google’s network enjoys lower latency than connectivity that uses external addresses.
  • Network security: Service owners do not need to have their services exposed to the public internet and deal with its associated risks.
  • Network cost: Rather than pay egress bandwidth costs for networks using external IPs to communicate, peered networks can use internal IPs to communicate, saving you money on those egress costs. Regular network pricing still applies to all traffic.

Many customers also appreciate the flexibility they get from VPC network peering: It supports the ability to peer your own VPC networks with other VPC networks that reside in different projects and even in different organizations.  

Using VPC network peering to strengthen your SAP security posture

Most Google Cloud customers find VPC network peering most useful to ensure seamless connectivity between their SAP managed applications and the rest of their Google Cloud environment. But some companies also rely on VPC network peering to gain greater control over the security of their cloud environments. This can include:

  • Implementing additional network security capabilities, such as advanced firewalls, intrusion detection, and network package inspection, that go above and beyond standard SAP security measures.
  • Leveraging Google Cloud Interconnect to link your on-premises and Google Cloud environments, including your RISE with SAP managed applications, without sending traffic across the public internet.

Once your company has configured VPC network peering, things get even more interesting. Google Cloud offers plenty of ways to complement and enhance the value of your SAP applications: using Google BigQuery to consolidate and enrich your SAP application data; relying on Google AI/ML capabilities to sharpen your predictive analytics and real-time decision-making; or leveraging Google Kubernetes Engine to jump-start your own cloud-native application development efforts, among many other examples

Some of the world’s biggest SAP customers, including The Home Depot, and Cardinal Health,  have chosen Google Cloud to migrate their business-critical SAP applications. And we’ve seen how useful VPC network peering can be for SAP customers to unlock the full value and potential of Google Cloud. Be sure to download our white paper so you can get VPC network peering configured for your RISE with SAP managed applications, and discover for yourself just what’s possible when you run SAP applications on Google Cloud.

Blog

Strategic Consulting, Training & Implementation Guidelines with Public Sector PSO Helps Governments Stay Compliant

4717

Of your peers have already read this article.

3:00 Minutes

The most insightful time you'll spend today!

Google Cloud's Public Sector Professional Service Organization (PSO) is dedicated towards helping enterprises meet their cloud security and compliance needs by offering consulting. Read about Google Cloud's commitment towards scaling cloud adoption!

Did you know that by 2025, enterprise IT spending on public cloud computing will overtake traditional IT spending? In fact, 51% of IT spend in application software, infrastructure software, business process services, and system infrastructure will transition to the public cloud, compared to 41% in 20221.. As enterprises continue to rapidly shift to the cloud, government agencies must prioritize and accelerate security and compliance implementation.

In May 2021, the White House issued an Executive Order requiring US Federal agencies to accelerate cloud adoption, embrace security best practices, develop plans to implement Zero Trust architectures, and map implementation frameworks to FedRAMP. The Administration’s focus on secure cloud adoption marks a critical shift to prioritizing cybersecurity at scale. Google Cloud’s Public Sector Professional Services Organization (PSO) has committed to helping customers meet security and compliance requirements in the cloud through specialized consulting engagements.

Accelerating Authority to Operate (ATO)


The Federal Risk and Authorization Management Program (FedRAMP) was established in 2011 as a government-wide program that promotes the adoption of secure cloud services across the federal government. FedRAMP provides a standardized approach to security and risk assessment for cloud technologies and federal agencies. US Federal agencies are required to utilize and implement FedRAMP cloud service offerings as part of the “Cloud First” federal cloud computing strategy.

While Google Cloud provides a FedRAMP-authorized cloud services platform and a robust catalog of FedRAMP-approved products and services (92 services and counting), customers are still tasked with achieving Agency ATO for the products and services they use, and Google Cloud provides many resources to assist customers with this journey. Google Cloud’s FedRAMP package can be accessed by completing the FedRAMP Package Access Request Form and submitting it to info@fedramp.gov. Additionally, customers can use Google’s NIST 800-53 ATO Accelerator as a starting point for documenting control implementation. Finally, Google Cloud’s Public Sector PSO offers the following strategic consulting engagements to help customers streamline the Agency ATO process.

  • Cloud Discover: FedRAMP is a six-week interactive workshop to support customers that are just getting started with the ATO process on Google Cloud. Customers are educated on FedRAMP fundamentals, Google’s security and compliance posture, and how to approach ATO on Google Cloud. Through deep-dive interviews and design sessions, PSO helps customers craft an actionable ATO plan, assess FedRAMP readiness, and develop a conceptual ATO boundary. This engagement helps organizations establish a clear understanding and roadmap for FedRAMP ATO on Google Cloud.
  • FedRAMP Security Review is a ten to twelve week engagement that aids customers in FedRAMP operational readiness. PSO consultants perform detailed FedRAMP architecture reviews to identify potential gaps in NIST 800-53 security control implementation and Google Cloud secure architecture best practices. Findings from the security reviews are shared with the customer along with configuration guidance and recommendations. This engagement helps organizations prepare for the third-party or independent security assessment that is required for FedRAMP ATO.
  • Cloud Deploy: FedRAMP is a multi-month engagement designed to help customers document the details of their FedRAMP System Security Plan (SSP) and corresponding NIST 800-53 security controls, in preparation for Agency ATO on Google Cloud at FedRAMP Low, Moderate, or High. PSO collaborates with customers to develop a detailed technical infrastructure design document and security control matrix capturing evidence of the FedRAMP system architecture, security control implementation, data flows and system components. PSO can also partner with a third-party assessment organization (3PAO) or an independent assessor (IA) to support customer efforts for FedRAMP security assessment. This engagement helps customer system owners prepare for Agency ATO assessment and package submission.

Developing a Zero Trust Strategy


In addition to providing FedRAMP enablement, Public Sector PSO has partnered with the Google Cloud Chief Information Security Officer (CISO) team to assist organizations with developing a zero trust architecture and strategy.


Zero Trust Foundations is a seven-week engagement co-delivered by Google Cloud’s CISO and PSO teams. CISO and PSO educate customers on zero trust fundamentals, Google’s journey to zero trust through BeyondCorp, and defense in depth best practices. The CISO team walks customers through a Zero Trust Assessment (ZTA) to understand the organization’s current security posture and maturity. Insights from the ZTA enable the CISO team to work with the customer to identify an ideal first-mover workload for zero trust adoption. Following the CISO ZTA, PSO facilitates a deep-dive Zero Trust Workshop (ZTW), collaborating with key customer stakeholders to develop a NIST 800-207 aligned, cloud-agnostic zero trust architecture for the identified first-mover workload. The zero trust architecture is part of a comprehensive zero trust strategy deliverable that is based on focus areas called out in the Office of Management and Budget (OMB) Federal Zero Trust Strategy released January 2022.

Scaling Secure Cloud Adoption with PSO


Public Sector PSO enables customer success by sharing our technical expertise, providing cloud strategy, implementation guidance, training and enablement using our proven methodology. As enterprise IT, operations, and organizational models continue to evolve, our goal is to help government agencies accelerate their security and compliance journeys in the cloud. To learn more about the work we are doing with the federal government, visit cloud.google.com/solutions/federal-government.

1 Gartner Says More Than Half of Enterprise IT Spending in Key Market Segments Will Shift to the Cloud by 2025

More Relevant Stories for Your Company

Case Study

Google Cloud Helps DSW Engage Over 28M Shoe Lovers in Real Time

DSW Shoes turned to Google Cloud and Google Cloud Partner – Slalom to develop a real­time, engaging loyalty program that serves more than 28 million active members instantly. The major move from on ­premise to a cloud-­based, data-driven platform that offers speed, flexibility, and scalability has helped increase DSW's new

How-to

How Google Cloud Helps SAP Admins Create Scalable, Secure Networks

SAP forms the critical backbone of thousands of enterprises, supporting critical business functions such as finance, supply chain, warehouse management, and more. Google Cloud provides a highly scalable and resilient infrastructure to run such workloads and offers tools, such as Smart Analytics and Machine Learning that can accelerate your organization’s digital transformation.  In fact, a

Blog

Google Introduces Cloud Career Jump Start Certification for the U.S. Underrepresented Communities

It’s no question that the cloud computing industry is booming and cloud experts are in high demand.  In 2020, 67 percent of organizations surveyed in the IDG Cloud Computing Report added new cloud roles and functions (source).  So there are plenty of cloud jobs out there, but if you don’t

Blog

How AI-powered ML Models Helps Run Unemployment Claims Verification at Scale

With unemployment application submissions reaching record numbers over the past year, state and local agencies in the United States have faced the challenge of processing unprecedented numbers of claims per week. The digital infrastructure most agencies have in place is unable to handle this volume, resulting in constituents waiting longer,

SHOW MORE STORIES