Cloud Bigtable Helps Fraud-detection Company Meet Scalability Demands and Secure Customer Data
7106
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Editor’s note: Today we are hearing from Jono MacDougall , Principal Software Engineer at Ravelin. Ravelin delivers market-leading online fraud detection and payment acceptance solutions for online retailers. To help us meet the scaling, throughput, and latency demands of our growing roster of large-scale clients, we migrated to Google Cloud and its suite of managed services, including Cloud Bigtable, the scalable NoSQL database for large workloads.
As a fraud detection company for online retailers, each new client brings new data that must be kept in a secure manner and new financial transactions to analyze. This means our data infrastructure must be highly scalable and constantly maintain low latency. Our goal is to bring these new organizations on quickly without interrupting their business. We help our clients with checkout flows, so we need latencies that won’t interrupt that process—a critical concern in the booming online retail sector.
We like Cloud Bigtable because it can quickly and securely ingest and process a high volume of data. Our software accesses data in Bigtable every time it makes a fraud decision. When a client’s customer places an order, we need to process their full history and as much data as possible about that customer in order to detect fraud, all while keeping their data secure. Bigtable excels at accessing and processing that data in a short time window. With a customer key, we can quickly access data, bring it into our feature extraction process, and generate features for our models and rules. The data stays encrypted at rest in Bigtable, which keeps us and our customers safe.
Bigtable also lets us present customer profiles in our dashboard to our client, so that if we make a fraud decision, our clients can confirm the fraud using the same data source we use.
We have configured our bigtable clusters to only be accessible within our private network and have restricted our pods access to it using targeted service accounts. This way the majority of our code does not have access to bigtable and only the bits that do the reading and writing have those privileges.
We also use Bigtable for debugging, logging, and tracing, because we have spare capacity and it’s a fast, convenient location.
We conduct load testings against Bigtable. We started at a low rate of ~10 Bigtable requests per second and we peaked at ~167000 mixed read and write requests per second at absolute peak. The only intervention that was done to achieve this was pressing a single button to increase the number of nodes in the database. No other changes were made.
In terms of real traffic to our production system, we have seen ~22,000 req/s (combined read/write) on Bigtable in our live environment as a peak within the last 6 weeks.
Migrating seamlessly to Google Cloud
Like many startups, we started with Postgres, since it was easy and it was what we knew, but we quickly realized that scaling would be a challenge, and we didn’t want to manage enormous Postgres instances. We looked for a kind of key value store, because we weren’t doing crazy JOINS or complex WHERE clauses. We wanted to provide a customer ID and get everything we knew about it, and that’s where key value really shines.
I used Cassandra at a previous company, but we had to hire several people just for that chore. At Ravelin we wanted to move to managed services and save ourselves that headache. We were already heavy users and fans of BigQuery, Google Cloud’s serverless, scalable data warehouse, and we also wanted to start using Kubernetes. This was five years ago, and though quite a few providers offer Kubernetes services now, we still see Google Cloud at the top of that stack with Google Kubernetes Engine (GKE). We also like Bigtable’s versioning capability that helped with a use case involving upserts. All of these features helped us choose Bigtable.
Migrations can be intimidating, especially in retail where downtime isn’t an option. We were migrating not just from Postgres to Bigtable, but also from AWS to Google Cloud. To prepare, we ran in AWS like always, but at the same time we set up a queue at our API level to mirror every request over to Google Cloud. We looked at those requests to see if any were failing, and confirmed if the results and response times were the same as in AWS. We did that for a month, fine tuning along the way.
Then we took the big step and flipped a config flag and it was 100% over to Google Cloud. At the exact same time, we flipped the queue over to AWS so that we could still send traffic into our legacy environment. That way, if anything went wrong, we could fail back without missing data. We ran like that for about a month, and we never had to fail back. In the end, we pulled off a seamless, issue-free online migration to Google Cloud.
Flexing Bigtable’s features
For our database structure, we originally had everything spread across rows, and we’d use a hash of a customer ID as a prefix. Then we could scan each record of history, such as orders or transactions. But eventually we got customers that were too big, where the scanning wasn’t fast enough. So we switched and put all of the customer data into one row and the history into columns. Then each cell was a different record, order, payment method, or transaction. Now, we can quickly look up the one row and get all the necessary details of that customer. Some of our clients send us test customers who place an order, say, every minute, and that quickly becomes problematic if you want to pull out enormous amounts of data without any limits on your row size. The garbage collection feature makes it easy to clean up big customers.
We also use Bigtable replication to increase reliability, atomicity, and consistency. We need strong consistency guarantees within the context of a single request to our API since we make multiple bigtable requests within that scope. So within a request we always hit the same replica of Bigtable and if we have a failure, we retry the whole request. That allows us to make use of the replica and some of the consistency guarantees, a nice little trade-off where we can choose where we want our consistency to live.https://www.youtube.com/embed/0-eH5u7rrQQ?enablejsapi=1&
We also use BigQuery with Bigtable for training on customer records or queries with complicated WHERE clauses. We put the data in Bigtable, and also asynchronously in BigQuery using streaming inserts, which allows our data scientists to query it in every way you can imagine, build models, and investigate patterns and not worry about query engine limitations. Since our Bigtable production cluster is completely separate, doing a query on BigQuery has no impact on our response times. When we were on Postgres many years ago, it was used for both analysis and real time traffic and it was not the optimal solution for us. We also use Elasticsearch for powering text searches for our dashboard.
If you’re using Bigtable, we recommend three features:
- Key visualizer. If we get latency or errors coming back from Bigtable, we look at the key visualizer first. We may have a hotkey or a wide row, and the visualizer will alert us and provide the exact key range where the key lives, or the row in question. Then we can go in and fix it at that level. It’s useful to know how your data is hitting Bigtable and if you’re using any anti-patterns or if your clients have changed their traffic pattern that exacerbated some issue.
- Garbage collection. We can prevent big row issues by putting size limits in place with the garbage collection policies.
- Cell versioning. Bigtable has a 3d array, with rows, columns, and cells, which are all the different versions. You can make use of the versioning to get history of a particular value or to build a time series within one row. Getting a single row is very fast in Bigtable so as long as you can keep the data volume in check for that row, making use of cell versions is a very powerful and fast option. There are patterns in the docs that are quite useful and not immediately obvious. For example, one trick is to reverse your timestamps (MAXINT64 – now) so instead of the latest version, you can get the oldest version effectively reversing the cell version sorting if you need it.
Google Cloud and Bigtable help us meet the low-latency demands of the growing online retail sector, with speed and easy integration with other Google Cloud services like BigQuery. With their managed services, we freed up time to focus on innovations and meet the needs of bigger and bigger customers.
Learn more about Ravelin and Bigtable, and check out our recent blog, How BIG is Cloud Bigtable?
How One Company Improved Security Significantly–Without Increasing Staff
6447
Of your peers have already read this article.
2:45 Minutes
The most insightful time you'll spend today!
Quanta Services is the leading specialty contractor with the largest and highly-skilled trained workforce in North America. It provides fully-integrated solutions for the electric power
pipeline industrial and telecommunications industries the company’s geographic footprint which includes North America Latin America and Australia.
It’s network of companies ensures world-class execution with local delivery and has over 40,000 employees.
“The exciting thing about Backstory is it allows us to land massive amounts of data from all of our different security tools and then Chronicle worries about how to correlate and aggregate this information for us. We can then focus on the highest priority threats.”
Richard Breaux, Manager Security Operations Quanta Services
Because Quanta Services’ customers provide energy and telecommunications to their customers, they have world-class cybersecurity requirements.
“We must ensure that Quanta leads the industry given the fact that our business is building the core infrastructure that powers people’s lives. It’s crucial that we meet their cybersecurity requirements,” says James Stinson, VP-IT, Quanta Services.
To support this goal Quanta implemented a number of security tools that generate terabytes of security telemetry. Over time, however, these systems generated an information overload for the company’s limited pool of qualified security resources.
“Our logging tools also couldn’t keep up with the rapidly growing information. With limited resources we need to focus our security analysts time on high quality work instead of digging through mountains of data,” says Stinson.
They achieve this by implementing Backstory and Chronicle, which is on Google Cloud.
“The exciting thing about Backstory is it allows us to land massive amounts of data from all of our different security tools and then Chronicle worries about how to correlate and aggregate this information for us. We can then focus on the highest priority threats,” says Richard Breaux, Manager Security Operations Quanta Services.
As a result, the company’s security team spends less time getting to the core information they need to address these incidents.
“What used to take us 15 minutes or more, we can now accomplish in seconds,” says Breaux.
7 Common Myths About Cloud Computing That Must Be Banished
4268
Of your peers have already read this article.
3:15 Minutes
The most insightful time you'll spend today!
No discussion about the emerging technologies that are shaping our future is complete without a mention of cloud computing. Unfortunately, too many of these discussions are filled with misinformation and fear-mongering. And while it’s always advisable to approach new technologies with a degree of healthy skepticism, this should always be backed by facts.
Here are seven of the most common myths about cloud computing that you shouldn’t take at face value:
An On-Premises Data Centre is More Secure Than the Cloud
One of the first concerns cloud skeptics cite as the case against it is the possibility of a data breach or cyber attack. However, the overwhelming majority of professionals agree that cloud computing offers much more security than an on-premises data centre. Not only do cloud solutions come with the requisite infrastructure, but they’re also maintained by some of the finest security personnel in the world, who work for cloud service providers.
Cloud Computing is Too Expensive
The concerns surrounding the cost of cloud computing have been blown vastly out of proportion. While the cost of implementation varies from one business to another, cloud computing’s ‘pay-as-you-go’ model helps prevent unnecessary spending on infrastructure. In fact, some providers offer users their service without any up-front costs or termination fees. However, it is up to an enterprise to evaluate whether this model will be more cost-effective for their requirements, as opposed to building a data centre.
Migration to the Cloud Will Result in Extended Downtime
There’s a reason that the old adage ‘Time is money’ is as popular as it is. Irrespective of its industry, geography, size or nature, no organisation can afford more than the bare minimum downtime. However, cloud migration doesn’t have to be synonymous with lost time and money. In fact, most popular cloud service providers offer seamless, live migration, and may only result in negligible downtime where the existing servers are extremely outdated.
My Business is Too Small for Cloud Computing
Cloud computing was once thought of as the prerogative of big companies that have massive computing power and infrastructure needs. However, a quick glance through the pros and cons of the technology shows that this isn’t the case. In fact, small and medium organisations might just find it to be more useful. Not only does cloud adoption save the cost of building a data centre, but also the human resource expenditure that comes with having to maintain it. Moreover, offerings like Google Apps for Business consolidate a multitude of services into one platform, making them easier to implement and maintain.
Cloud Computing is Only Good for Storage and Analytics
Data storage and analytics are among the cloud’s most popular uses. However, this doesn’t mean that the technology is only of use to technology and data science departments. Figures from 2011 showed that the bulk of cloud spending in the Asia Pacific region is concentrated in the functions of customer service, marketing, sales, manufacturing and human resource management. Seamless flow of accurate and up-to-date information is department-agnostic, and if leveraged in the right way, can be a game-changer for every organisation.
I Won’t Have Enough Control on the Cloud
Cloud solutions aren’t as rigid as most people assume. Between the public, private and hybrid cloud, there are a host of options that businesses can choose from when it comes to cloud migration. Each of these offers a different degree of customisation and flexibility, which companies can pick from, on the basis of their requirements and skill level.
Cloud Computing is Digital Transformation
Deploying cloud solutions can put organisations on the fast track to success in the digital age. However, it is far from the only component that is needed to make the big transition. It’s not unheard of for vendors and IT professionals to make cloud computing seem like the be-all and end-all solution to all of a business’ digital woes. This is why, senior management should have a clearly defined purpose and realistic expectations in mind before migrating to the cloud.
Quick Recap on Google Cloud: Latest News, Launches, Updates, Events and More
12888
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more.
Tip: Not sure where to find what you’re looking for on the Google Cloud blog? Start here: Google Cloud blog 101: Full list of topics, links, and resources.
Week of May 24-May 28 2021
- Google Cloud for financial services: driving your transformation cloud journey–As we welcome the industry to our Financial Services Summit, we’re sharing more on how Google Cloud accelerates a financial organization’s digital transformation through app and infrastructure modernization, data democratization, people connections, and trusted transactions. Read more or watch the summit on demand.
- Introducing Datashare solution for financial services–We announced the general availability of Datashare for financial services, a new Google Cloud solution that brings together the entire capital markets ecosystem—data publishers and data consumers—to exchange market data securely and easily. Read more.
- Announcing Datastream in Preview–Datastream, a serverless change data capture (CDC) and replication service, allows enterprises to synchronize data across heterogeneous databases, storage systems, and applications reliably and with minimal latency to support real-time analytics, database replication, and event-driven architectures. Read more.
- Introducing Dataplex: An intelligent data fabric for analytics at scale–Dataplex provides a way to centrally manage, monitor, and govern your data across data lakes, data warehouses and data marts, and make this data securely accessible to a variety of analytics and data science tools. Read more.
- Announcing Dataflow Prime–Available in Preview in Q3 2021, Dataflow Prime is a new platform based on a serverless, no-ops, auto-tuning architecture built to bring unparalleled resource utilization and radical operational simplicity to big data processing. Dataflow Prime builds on Dataflow and brings new user benefits with innovations in resource utilization and distributed diagnostics. The new capabilities in Dataflow significantly reduce the time spent on infrastructure sizing and tuning tasks, as well as time spent diagnosing data freshness problems. Read more.
- Secure and scalable sharing for data and analytics with Analytics Hub–With Analytics Hub, available in Preview in Q3, organizations get a rich data ecosystem by publishing and subscribing to analytics-ready datasets; control and monitoring over how their data is being used; a self-service way to access valuable and trusted data assets; and an easy way to monetize their data assets without the overhead of building and managing the infrastructure. Read more.
- Cloud Spanner trims entry cost by 90%–Coming soon to Preview, granular instance sizing in Spanner lets organizations run workloads at as low as 1/10th the cost of regular instances, equating to approximately $65/month. Read more.
- Cloud Bigtable lifts SLA and adds new security features for regulated industries–Bigtable instances with a multi-cluster routing policy across 3 or more regions are now covered by a 99.999% monthly uptime percentage under the new SLA. In addition, new Data Access audit logs can help determine whether sensitive customer information has been accessed in the event of a security incident, and if so, when, and by whom. Read more.
- Build a no-code journaling app–In honor of Mental Health Awareness Month, Google Cloud’s no-code application development platform, AppSheet, demonstrates how you can build a journaling app complete with titles, time stamps, mood entries, and more. Learn how with this blog and video here.
- New features in Security Command Center—On May 24th, Security Command Center Premium launched the general availability of granular access controls at project- and folder-level and Center for Internet Security (CIS) 1.1 benchmarks for Google Cloud Platform Foundation. These new capabilities enable organizations to improve their security posture and efficiently manage risk for their Google Cloud environment. Learn more.
- Simplified API operations with AI–Google Cloud’s API management platform Apigee applies Google’s industry leading ML and AI to your API metadata. Understand how it works with anomaly detection here.
- This week: Data Cloud and Financial Services Summits–Our Google Cloud Summit series begins this week with the Data Cloud Summit on Wednesday May 26 (Global). At this half-day event, you’ll learn how leading companies like PayPal, Workday, Equifax, and many others are driving competitive differentiation using Google Cloud technologies to build their data clouds and transform data into value that drives innovation. The following day, Thursday May 27 (Global & EMEA) at the Financial Services Summit, discover how Google Cloud is helping financial institutions such as PayPal, Global Payments, HSBC, Credit Suisse, AXA Switzerland and more unlock new possibilities and accelerate business through innovation. Read more and explore the entire summit series.
- Announcing the Google for Games Developer Summit 2021 on July 12th-13th–With a surge of new gamers and an increase in time spent playing games in the last year, it’s more important than ever for game developers to delight and engage players. To help developers with this opportunity, the games teams at Google are back to announce the return of the Google for Games Developer Summit 2021 on July 12th-13th. Hear from experts across Google about new game solutions they’re building to make it easier for you to continue creating great games, connecting with players and scaling your business. Registration is free and open to all game developers. Register for the free online event at g.co/gamedevsummit to get more details in the coming weeks. We can’t wait to share our latest innovations with the developer community. Learn more.
Modernize your Windows Server Workloads using Google Cloud Platform
DOWNLOAD RESEARCH REPORTS6246
Of your peers have already downloaded this article
1:30 Minutes
The most insightful time you'll spend today!
Application Modernization is an important enabler of Digital Transformations (DX), which fuel competitive advantage through increased productivity and business agility. Public cloud infrastructure proves to be a solid foundation for application modernization by providing Self-Service Provisioning capabilities, cloud-based & cloud-native technologies, and easier access to technology innovations such as AI/ML.
Windows Server-based enterprise applications rely on the underlying infrastructure for platform performance, security, and availability. A better performing cloud platform enables them to perform better and hence prove to be more resource-optimized and cost-effective.
Download this IDC report to understand why you should move your Windows Server workloads to Google Cloud and the benefits you can derive.
3911
Of your peers have already downloaded this article
1:30 Minutes
The most insightful time you'll spend today!
Migrating to the cloud can be complex, time consuming, and risky, especially when you have hundreds or thousands of existing workloads to move. Make your journey fast and smooth by planning ahead and using tried-and-true best practices. To help you get started, here’s a handy guide that outlines four basic phases of a successful cloud migration:
ASSESS: Identify your team, get an overview of your IT landscape, and decide which applications to move first.
PLAN: Choose one or more migration strategies, consider a streaming-based solution, and test your applications’ performance in the cloud.
MIGRATE: Use a phased, agile approach that allows you to revert to the on-premises configuration if necessary.
OPTIMIZE: Fine-tune your cloud environment to align usage with demand and implement capabilities like cost controls and governance tools.
Download this handy guide to get started and get a detailed checklist of key milestones on your journey to the cloud, ensuring that you complete every step and always know what’s next.
More Relevant Stories for Your Company
Google Cloud’s High-performance Compute Speeds Up the Chip Design Process
Cloud offers a proven way to accelerate end-to-end chip design flows. In a previous blog, we demonstrated the inherent elasticity of the cloud, showcasing how front-end simulation workloads can scale with access to more compute resources. Another benefit of the cloud is access to a powerful, modern and global infrastructure. On-prem
This Chart, from Home Depot, Dramatically Demonstrates the Power of a Cloud Data Warehouse
The Home Depot (THD) is the world’s largest home-improvement chain, growing to more than 2,200 stores and 700,000 products in four decades. Much of that success was driven through the analysis of data. This included developing sales forecasts, replenishing inventory through the supply chain network, and providing timely performance scorecards. However,
Giving Customers More Choice: Google Cloud’s New Product and Pricing Options
Over the past several years, Google Cloud has made significant investments in our infrastructure product portfolio. We launched new Tau T2D VMs, which deliver 42% better price-performance vs. other leading cloud providers. We upgraded Cloud Storage to offer more flexibility to support customers’ enterprise and analytics workloads, with dual-region buckets
Moving Flock Freight to Google Cloud for a more efficient, resilient and environmentally sustainable shipping supply chain
Commercial trucks often travel partially empty because many shippers don’t have enough cargo to fill an entire container or trailer. Although offering available space to other shippers helps minimize carbon emissions and reduce operating costs, most trucking companies can’t efficiently schedule, track, or deliver multiple freight loads. Companies have always
