BigQuery ML for Sentiment Analysis: How to Make the Most of Your Data
1804
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
Introduction
We recently announced BigQuery support for sparse features which help users to store and process the sparse features efficiently while working with them. That functionality enables users to represent sparse tensors and train machine learning models directly in the BigQuery environment. Being able to represent sparse tensors is a useful feature because sparse tensors are used extensively in encoding schemes like TF-IDF as part of data pre-processing in NLP applications and for pre-processing images with a lot of dark pixels in computer vision applications.
There are numerous applications of sparse features such as text generation and sentiment analysis. In this blog, we’ll demonstrate how to perform sentiment analysis with the space features in BigQuery ML by training and inferencing machine learning models using a public dataset. This blog also highlights how easy it is to work with unstructured text data on BigQuery, an environment traditionally used for structured data.
Using sample IMDb dataset
Let’s say you want to conduct a sentiment analysis on movie reviews from the IMDb website. For the benefit of readers who want to follow along, we will be using the IMDb reviews dataset from BigQuery public datasets. Let’s look at the top 2 rows of the dataset.
Although the reviews table has 7 columns, we only use reviews and label columns to perform sentiment analysis for this case. Also, we are only considering negative and positive values in the label columns. The following query can be used to select only the required information from the dataset.
SELECT
review,
label,
FROM
`bigquery-public-data.imdb.reviews`
WHERE
label IN ('Negative', 'Positive')The top 2 rows of the result is as follows:
Methodology
Based on the dataset that we have, the following steps will be carried out:
- Build a vocabulary list using the review column
- Convert the review column into sparse tensors
- Train a classification model using the sparse tensors to predict the label (“positive” or “negative”)
- Make predictions on new test data to classify reviews as positive or negative.
Feature engineering
In this section, we will convert the text from the reviews column to numerical features so that we can feed them into a machine learning model. One of the ways is the bag-of-words approach where we build a vocabulary using the words from the reviews and select the most common words to build numerical features for model training. But first, we must extract the words from each review. The following code creates a dataset and a table with row numbers and extracted words from reviews.
-- Create a dataset named `sparse_features_demo` if doesn’t exist
CREATE SCHEMA IF NOT EXISTS sparse_features_demo;
-- Select unique reviews with only negative and positive labels
CREATE OR REPLACE TABLE sparse_features_demo.processed_reviews AS (
SELECT
ROW_NUMBER() OVER () AS review_number,
review,
REGEXP_EXTRACT_ALL(LOWER(review), '[a-z]{2,}') AS words,
label,
split
FROM (
SELECT
DISTINCT review,
label,
split
FROM
`bigquery-public-data.imdb.reviews`
WHERE
label IN ('Negative', 'Positive')
)
);The output table from the query above should look like this:
The next step is to build a vocabulary using the extracted words. The following code creates a vocabulary including word frequency and word index from reviews. For this case, we are going to select only the top 20,000 words to reduce the computation time.
-- Create a vocabulary using train dataset and select only top 20,000 words based on frequency
CREATE OR REPLACE TABLE sparse_features_demo.vocabulary AS (
SELECT
word,
word_frequency,
word_index
FROM (
SELECT
word,
word_frequency,
ROW_NUMBER() OVER (ORDER BY word_frequency DESC) - 1 AS word_index
FROM (
SELECT
word,
COUNT(word) AS word_frequency
FROM
sparse_features_demo.processed_reviews,
UNNEST(words) AS word
WHERE
split = "train"
GROUP BY
word
)
)
WHERE
word_index < 20000 # Select top 20,000 words based on word count
);The following shows the top 10 words based on frequency and their respective index from the resulting table of the query above.
Creating a sparse feature
Now we will use the newly added feature to create a sparse feature in BigQuery. For this case, we aggregate word_index and word_frequency in each review, which generates a column as ARRAY[STRUCT] type. Now, each review is represented as ARRAY[(word_index, word_frequency)].
-- Generate a sparse feature by aggregating word_index and word_frequency in each review.
CREATE OR REPLACE TABLE sparse_features_demo.sparse_feature AS (
SELECT
review_number,
review,
ARRAY_AGG(STRUCT(word_index, word_frequency)) AS feature,
label,
split
FROM (
SELECT
DISTINCT review_number,
review,
word,
label,
split
FROM
sparse_features_demo.processed_reviews,
UNNEST(words) AS word
WHERE
word IN (SELECT word FROM sparse_features_demo.vocabulary)
) AS word_list
LEFT JOIN
sparse_features_demo.vocabulary AS topk_words
ON
word_list.word = topk_words.word
GROUP BY
review_number,
review,
label,
split
);Once the query is executed, a sparse feature named `feature` will be created. That `feature` column is an `ARRAY of STRUCT` column which is made of `word_index` and `word_frequency` columns. The picture below displays the resulting table at a glance.
Training a BigQuery ML model
We just created a dataset with a sparse feature in BigQuery. Let’s see how we can use that dataset to train with a machine learning model with BigQuery ML. In the following query, we will train a logistic regression model using the review_number, review, and feature to predict the label:
-- Train a logistic regression classifier using the data with sparse feature
CREATE OR REPLACE MODEL sparse_features_demo.logistic_reg_classifier
TRANSFORM (
* EXCEPT (
review_number,
review
)
)
OPTIONS(
MODEL_TYPE='LOGISTIC_REG',
INPUT_LABEL_COLS = ['label']
) AS
SELECT
review_number,
review,
feature,
label
FROM
sparse_features_demo.sparse_feature
WHERE
split = "train"
;Now that we have trained a BigQuery ML Model using a sparse feature, we evaluate the model and tune it as needed.
-- Evaluate the trained logistic regression classifier
SELECT * FROM ML.EVALUATE(MODEL sparse_features_demo.logistic_reg_classifier);The score looks like a decent starting point, so let’s go ahead and test the model with the test dataset.
-- Evaluate the trained logistic regression classifier using test data
SELECT * FROM ML.EVALUATE(MODEL sparse_features_demo.logistic_reg_classifier,
(
SELECT
review_number,
review,
feature,
label
FROM
sparse_features_demo.sparse_feature
WHERE
split = "test"
)
);
The model performance for the test dataset looks satisfactory and it can now be used for inference. One thing to note here is that since the model is trained on the numerical features, the model will only accept numeral features as input. Hence, the new reviews have to go through the same transformation steps before they can be used for inference. The next step shows how the transformation can be applied to a user-defined dataset.
Sentiment predictions from the BigQuery ML model
All we have left to do now is to create a user-defined dataset, apply the same transformations to the reviews, and use the user-defined sparse features to perform model inference. It can be achieved using a WITH statement as shown below.
WITH
-- Create a user defined reviews
user_defined_reviews AS (
SELECT
ROW_NUMBER() OVER () AS review_number,
review,
REGEXP_EXTRACT_ALL(LOWER(review), '[a-z]{2,}') AS words
FROM (
SELECT "What a boring movie" AS review UNION ALL
SELECT "I don't like this movie" AS review UNION ALL
SELECT "The best movie ever" AS review
)
),
-- Create a sparse feature from user defined reviews
user_defined_sparse_feature AS (
SELECT
review_number,
review,
ARRAY_AGG(STRUCT(word_index, word_frequency)) AS feature
FROM (
SELECT
DISTINCT review_number,
review,
word
FROM
user_defined_reviews,
UNNEST(words) as word
WHERE
word IN (SELECT word FROM sparse_features_demo.vocabulary)
) AS word_list
LEFT JOIN
sparse_features_demo.vocabulary AS topk_words
ON
word_list.word = topk_words.word
GROUP BY
review_number,
review
)
-- Evaluate the trained model using user defined data
SELECT review, predicted_label FROM ML.PREDICT(MODEL sparse_features_demo.logistic_reg_classifier,
(
SELECT
*
FROM
user_defined_sparse_feature
)
);Here is what you would get for executing the query above:
And that’s it! We just performed a sentiment analysis on the IMDb dataset from a BigQuery Public Dataset using only SQL statements and BigQuery ML. Now that we have demonstrated how sparse features can be used with BigQuery ML models, we can’t wait to see all the amazing projects that you would create by harnessing this functionality.
If you’re just getting started with BigQuery, check out our interactive tutorial to begin exploring.
HarbourBridge Schema Assistant Allows Quick, Bulk Migration to Cloud Spanner
5459
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Today we’re announcing the HarbourBridge Schema Assistant, which provides a guided schema-design workflow for migrating from MySQL or PostgreSQL to Spanner. HarbourBridge imports dump files (from mysqldump or pg_dump) or directly connects to your source database, and converts the source database schema to an equivalent Spanner schema. The new Schema Assistant capability displays the source schema and Spanner schema side-by-side, highlights errors and walks you through a series of steps to validate and optimize your Spanner schema. It also produces a browsable assessment report with an overall migration-fitness score for Spanner, a table-by-table detailed analysis of type mappings and a list of features used in the source database that aren’t supported by Spanner. It supports editing of table and column names, column types, primary keys and constraints, as well as dropping of tables, columns, foreign keys and secondary indexes.
The new Schema Assistant complements HarbourBridge’s existing data and schema migration capabilities and is a critical step towards our goal of building a complete open-source migration toolkit. HarbourBridge continues to support command-line schema and data migration and turn-key Spanner evaluation.
Complementing the bulk data migration capabilities of HarbourBridge, we are also announcing the ability to migrate change events from MySQL to Cloud Spanner.
Supported Features in Schema Assistant
- Global type mapping. Users can customize the global mapping for how types should be mapped to Spanner consistently across the schema. For example, mapping large integers in source schema to Spanner’s NUMERIC.
- Local type mapping. Users can override the custom type mapping for a given table/column.
- Session management. A session keeps track of all the changes made to the schema mapping.
- Customization of secondary indexes. Users can add, edit and delete secondary indexes to optimize their Spanner performance.
- Customization of foreign keys and interleaved tables. Table interleaving is an important design consideration when migrating to Cloud Spanner as explained in more detail in this blog post.
Features in the pipeline
We are already working to further expand the supported set of schema editing features and welcome your feedback. We are particularly excited to expand the Schema Assistant’s design recommendations for optimizing Spanner schemas e.g. in-depth recommendations for primary key design.
HarbourBridge is open source and we gladly accept contributions from the wider community.
Daffodil Software Saves Hundreds of Hours Managing Databases with Google Cloud SQL
4714
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
Daffodil Software is an India-based information technology services company with 180 employees and satellite offices in the US, Singapore, and the United Arab Emirates.
Its product: A suite of web-based applications, including a business customer relationship management (CRM) program and an enterprise resource planning (ERP) app for schools.
Daffodil’s Challenge
Daffodil needed a cloud-based hosting platform for its applications that would save money and time over in-house hosting. The company tried Amazon Elastic Cloud Compute (EC2) with MySQL to manage the databases that its applications depended on. The potential was there, but building, testing and deploying the applications took too long, and scalability was also an issue.
“The old system provided the option to scale up if many users accessed our applications at the same time, but we had to do this manually,” says Gaurav Sharma, assistant head of Daffodil’s product team.
Soon, Daffodil switched to Google App Engine, which allows businesses to build and host web apps on the same infrastructure that powers Google applications.
Integration with App Engine lets Sharma and his team quickly build, test and deploy applications and allows them to automatically accommodate fluctuations in the number of users so they don’t have to manually manage the system when demand increases.
Although App Engine’s built-in Datastore was useful, they found that they wanted more SQL-like functionality for their data-driven programs.
Google’s Solution
At Google’s suggestion, Daffodil migrated to Cloud SQL three months later to take advantage of the system’s database indexing and dynamic filtering abilities, which speed up data sorting for Daffodil’s end users.
“Using Google App Engine and Google Cloud SQL make our applications go live in half the time and have provided us with hassle-free control over all processes.”
—Yogesh Agarwal, CEO, Daffodil Software
Besides indexing and dynamic filtering, Cloud SQL has simplified database management for the Daffodil team. Sharma can easily import their databases and administer them through an intuitive interface, while Google handles backend maintenance and administration.
Google Cloud SQL replicates data among multiple geographic regions, giving Daffodil CEO Yogesh Agarwal peace of mind that users’ data held by the applications is protected.
“The safety of our customers’ data is a major concern, so we appreciate that we don’t need to worry about data loss,” he says.
The Results
With App Engine and Cloud SQL, the Daffodil team is saving about 80 hours of development time every month because of easier deployment, no active management for scaling, and simplified database management. They have also slashed the time it takes to bring new products to market.
“The shift to Google Cloud SQL has allowed us to focus on making our applications even better.”
—Yogesh Agarwal, CEO, Daffodil Software
“Using Google App Engine and Google Cloud SQL make our applications go live in half the time and have provided us with hassle-free control over all processes, such as development, deployment and monitoring,” Agarwal says. “We’re able to deploy our applications much faster and spend less time managing them.”
Going forward, the Daffodil team plans to add other Google services to their applications, including Prediction API, which allows CRM application users to better predict the probability of a lead becoming a customer, and App Engine’s Full Text Search API, which helps users retrieve customer data more easily.
Agarwal is happy with the move to App Engine and Cloud SQL. “The shift has allowed us to focus on making our applications even better,” he says.
4228
Of your peers have already watched this video.
54:00 Minutes
The most insightful time you'll spend today!
Behind the Scenes: How eBay Provides its Customers New Shopping Experiences
“If it exists in the world, you are likely to find it on eBay.” So they say. With 180 million buyers and a global presence in over 190 markets, it’s probably true. A company that emerged out of the ashes of the dot-com bubble, eBay today is one of the most renowned organizations in the world.
eBay catalog has over 1 billion listings available worldwide at any given point in time and over 15 million listings are added on a daily basis. That essentially means many datasets from different sources. And that was a challenge.
The company needed a scalable and flexible platform to efficiently enable new shopping experiences globally, showcasing unique eBay inventory. It needed to iterate quickly and integrate data-heavy AI technologies.
eBay turned to Cloud Bigtable from Google. Today, Cloud Bigtable handles eBay’s global catalog with billions of listings, scaling to hundreds of terabytes with many millions of reads and writes, and gigabytes of data transferred in and out every second.
This helped eBay synchronize its entire catalog over GCP in near real-time with low latency and high consistency. It also enabled the company to serve catalog data to other services on GCP with high read volume and low latency and keep and serve all listing images in GCP along with AI vision signatures.
Find out how.
BigQuery Now Supports Multi-statement Transactions
3759
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Mercury, the Roman god of commerce, is often depicted carrying a purse, symbolic of business transactions, wearing winged sandals, illustrating his abilities to move at great speeds. Transactions power the world’s business systems today, ranging from millions of packages moving worldwide tracked in real time by logistics companies to global payments from personal loans to securities trading to intergovernmental transactions, keeping goods and services flowing worldwide. Today, we are very pleased to announce the public preview of multi-statement transactions in BigQuery.
BigQuery has long supported single-statement transactions through DML statements such as INSERT, UPDATE, DELETE, MERGE and TRUNCATE, applied to one table per transaction. Multi-statement transactions expand on this scope by supporting multiple SQL statements, including DML, spanning multiple tables in a single transaction. This means that the changes to data across multiple tables associated with all statements in a given transaction are committed atomically (all at once) if successful or all rolled back atomically in the event of a failure. These new multi-statement transaction capabilities now bridge the gap between online transaction processing (OLTP) systems and BigQuery through tighter and faster integration, just as Mercury’s speed enabled him to travel rapidly between the mortal and divine worlds.
Overview
A transaction is a set of tasks carried out against a database or a data warehouse representing a business unit of work. For example, an order expedite action may result in a series of changes to multiple tables, e.g., updating delivery dates on the order, adding new shipment lines on a shipment to create expedited delivery shipments and canceling (updating) prior normal delivery shipment lines. Correspondingly, when a batch job execute an ETL process in a data warehouse, it may perform data wrangling and data cleansing operations by transforming and saving the data in multiple tables. In both of these cases, if a failure occurs, then the entire transaction must be rolled back so that the system of record does not have partial data saved putting the dataset in an inconsistent state.
Transactions are known for their atomicity, consistency, isolation, and durability (ACID) properties. Atomicity indicates that all data changes (including metadata changes) are applied or reverted in a single operation, e.g. when a set of rows are added to a table with enrichment and the corresponding set of rows deleted, both operations happen together. Consistency refers to the consistency of data before and after the transaction has completed. When data is added to or updated in a table, the changes to data in tables are consistent with the constraints specified on the table or its columns, e.g. ensuring that the NOT NULL constraint is complied with when data gets added or modified in a table. Isolation indicates that multiple transactions can make changes to the table concurrently as long as the changes don’t conflict with each other, e.g. two transactions loading data into tables from different business units can proceed concurrently. Durability is the last and possibly the most important property of transactions which guarantees that in the event of a failure (after completing the transaction), all the changes committed by the transaction are saved for future transactions or queries.
Isolation level
All transactions in BigQuery including multi-statement transactions support snapshot isolation. At this isolation level, all statements in a transaction see a consistent snapshot of the database, as of the start of the transaction. BigQuery achieves this due to its multi-version property where it maintains the history of mutations to the database for a certain period. Transactions do not see any changes, committed or uncommitted, from other concurrent transactions while the transaction is in progress. BigQuery supports read-your-own-writes, where statements in a transaction can see all the changes applied by prior statements in the same transaction.
Example
Let’s run through an example to demonstrate how BigQuery supports the concept of multi-statement transactions with multiple sets of changes across multiple tables in one operation. In this example, we will set up this demo by creating 10 tables each with 100 partitions
Language: SQL
DECLARE num_tables DEFAULT 10;DECLARE x INT64 DEFAULT 0;# replace "demo" with your preferred dataset nameDECLARE ds STRING default "`00jathreya`";DECLARE prefix STRING DEFAULT "mst_demo";DECLARE table_name STRING;DECLARE pre_table_name STRING;/*Sets up 10 tables, with schema (partition_id:integer,value:timestamp).Each table has 100 partitions, with partition key 0, 10, 20, ..., 990.*/SET x = 0;WHILE x < num_tables DOSET table_name = format("%s.%s_%d", ds, prefix, x);EXECUTE IMMEDIATE format("""DROP TABLE IF EXISTS %s""", table_name);EXECUTE IMMEDIATE format("""CREATE OR REPLACE TABLE %s (partition_id INT64, value TIMESTAMP)PARTITION BY RANGE_BUCKET(partition_id, GENERATE_ARRAY(0, 1000, 10))AS SELECT partition_id, CURRENT_TIMESTAMP() AS value FROM UNNEST(GENERATE_ARRAY(0, 990, 10)) AS partition_id""", table_name);SET x = x + 1;END WHILE;# Verify the partitioned tables created and the timestampexecute immediate format("""select table_schema, table_name, count(partition_id), sum(total_rows), max(last_modified_time) from %s.INFORMATION_SCHEMA.PARTITIONSwhere starts_with(table_name,'%s')group by table_schema, table_name""", ds, prefix);# Verify data in table 9 based on the timestampEXECUTE IMMEDIATE format("""SELECT * FROM %s ORDER BY partition_id""", table_name);
Next, we will show how the new COMMIT and ROLLBACK commands in BigQuery allow you to save or reverse the changes in a transaction spanning multiple tables.
Language: SQL
DECLARE num_tables DEFAULT 10;DECLARE x INT64 DEFAULT 0;# Replace `mydataset` with your preferred dataset nameDECLARE ds STRING DEFAULT "`mydataset`";DECLARE prefix STRING DEFAULT "mst_demo";DECLARE table_name STRING;DECLARE pre_table_name STRING;/*Steps of the transaction:1. For table 0, updates the timestamp to CURRENT_TIMESTAMP.2. FOR table 1 to 9, merge table x using data from table x-1. All MERGE shouldsee the updated data in table x-1, so as a result the change in table 0 should bepropagated all the way to table 9.3. A total of 1000 partition changes on 10 tables.*/BEGIN TRANSACTION;SET x = 0;WHILE x < num_tables DOSET table_name = format("%s.%s_%d", ds, prefix, x);IF x = 0 THENEXECUTE IMMEDIATE format("""UPDATE %s SET value = CURRENT_TIMESTAMP WHERE true""", table_name);ELSESET pre_table_name = format("%s.%s_%d", ds, prefix, x-1);EXECUTE IMMEDIATE format("""MERGE %s t2 USING %s t1 ON t2.partition_id = t1.partition_idWHEN MATCHED THEN UPDATE SET value = t1.value""", table_name, pre_table_name);END IF;SET x = x + 1;END WHILE;# Verify data in table 9 inside transaction shows recently updated dataSET table_name = format("%s.%s_%d", ds, prefix, num_tables-1);EXECUTE IMMEDIATE format("""SELECT * FROM %s ORDER BY partition_id""", table_name);IF x = 9 THENCOMMIT TRANSACTION;ELSEROLLBACK TRANSACTION;END IF;# Verify data in table 9 outside transaction shows timestamp of original dataEXECUTE IMMEDIATE format("""SELECT * FROM %s ORDER BY partition_id""", table_name);
Monitoring and Management
Unlike single-statement transactions which map to a single DML command, multi-statement transactions are complex spanning multiple DML operations across multiple tables. To monitor these complex jobs, the BigQuery JOBS INFORMATION_SCHEMA views have been expanded to add a new column, TRANSACTION_ID, which contains a unique identifier for each transaction in BigQuery. You can query transactions that are in progress or the ones that have completed by querying these views.
Language: SQL
#Returns the list of transactions running under a parent job ID .;SELECTDISTINCT transaction_idFROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECTWHERE parent_job_id = "job_id" AND state = "RUNNING";# Returns the active transactions and parent job id that are running and affect a named table.;WITHrunning_transactions AS (SELECT DISTINCT transaction_idFROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECTWHERE state = "RUNNING")SELECTtransaction_id, parent_job_id, queryFROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT, running_transactionsWHERE destination_table = "mytable"AND transaction_id = running_transactions.transaction_id;# Returns all successfully committed transactionsSELECTtransaction_id, parent_job_id, queryFROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECTWHERE statement_type = "COMMIT_TRANSACTION" AND error_result IS NULL;# Returns all rolled back transactionsSELECTtransaction_id, parent_job_id, queryFROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECTWHERE statement_type = "ROLLBACK_TRANSACTION" AND error_result IS NULL;
To terminate a transaction, you will need to terminate the parent job under which the transaction is running by passing the parent job ID of the script containing the transaction to the BQ.JOBS.CANCEL system procedure.
Conclusion
As enterprises accelerate towards more real time insight-driven decisions, multi-statement transactions are a key enabler towards bringing transaction data into analytical data warehouses, such as BigQuery. With strong ACID properties, full transaction support for commit and rollback, and rich monitoring and management APIs built on proven transaction processing backbone, we are very pleased to help our BigQuery customers unlock the value of transactions through these new capabilities.
Insurer Uses Google Cloud AI to Battle Slow Growth: It Improves Sales by 5% in 8 Weeks
8827
Of your peers have already read this article.
7:30 Minutes
The most insightful time you'll spend today!
For a business to succeed in the long term, it needs to learn not just to adapt to inevitable change, but to harness it. South Africa-based PPS has been an insurance company since 1941 and today is the biggest mutual insurance provider in the country.
As a mutual company, PPS is owned by more than 200,000 members, making them shareholders. In recent years, PPS and other companies like it have been affected by a number of external factors.
“For one thing, technology platforms have brought in a new gig economy that has all kinds of implications for insurance,” says Avsharn Bachoo, CTO at PPS. “What we’ve been seeing is basically a disruption of the South African insurance industry. We chose to see that as an opportunity.”
“Our servers were at the end of their life cycle and we had to decide whether to refresh them or switch completely. To embrace the world of AI and machine learning effectively, we knew we needed a cloud-based infrastructure. We’ve found the answer in Google Cloud Platform.”
—Avsharn Bachoo, CTO, PPS
In early 2018, faced with an uncertain economic environment that was squeezing growth and profitability, PPS decided to transform itself from a traditional broker-based business into a digital insurance provider. A key pillar of this new strategy was to overhaul the company’s technology infrastructure. To turn the strategy into reality, Avsharn and his team chose Google Cloud Platform (GCP).
“Our servers were at the end of their life cycle and we had to decide whether to refresh them or switch completely,” says Avsharn. “To embrace the world of AI and machine learning (ML) effectively, we knew we needed a cloud-based infrastructure. We’ve found the answer in Google Cloud Platform.”
Power, speed, flexibility with Google Cloud Platform
Previously, PPS maintained an on-premises IT infrastructure, which worked for its traditional business but was unsuited for its new way of working. In early 2018, the company started working on new products for its members but this required large amounts of compute power that proved prohibitively expensive with on-premises servers. Even existing products were starting to require more than the infrastructure could deliver. Aging equipment meant that it’s testing and quality assurance environments bore little resemblance to the actual production environment.
“We had no pre-production environments at all,” says Avsharn, resulting in more work for developers after products had been released. Meanwhile, the capital required to buy and configure more servers for new projects meant fewer resources available for innovation, and left the company less able to react to changes in the market. PPS knew it had to find a cloud-based alternative.
Shortly after devising a new digital strategy, PPS engineers attended a training session on cloud infrastructure given by leading South African Google Cloud Partner Siatik. Impressed with the presentation, PPS engaged Siatik to help run a proof of concept for a cloud-based infrastructure, running on GCP. With on-site engineers and constant communication, Siatik formed a very close working relationship with PPS. “The team at Siatik was exemplary,” recalls Avsharn. “They were well-organized, with cutting-edge technical acumen and very creative solutions to our problems. They were real game-changers.”
“We wanted the platform to retrain its models in response to new data and improve its recommendations with more information. Normally this would be a manual process but Google Cloud ML Engine lets the models do this automatically.”
—Kimoon Kim, Lead Solution Architect and Data Engineer, Siatik
The proof of concept was successful, with GCP outperforming the existing infrastructure in terms of how it handled compute demands, databases, and storage.
“It’s the speed of GCP that really impresses us,” says Avsharn. PPS saw that GCP wasn’t just an opportunity to migrate its existing infrastructure to the cloud. With Siatik’s help, it redesigned its monolithic core architecture to one based around microservices using Google Kubernetes Engine (GKE). For data processing and storage, Cloud Dataflow and Cloud Datastore proved invaluable, while Stackdriver helped the IT team stay on top of logging and monitoring the system.
“Google Cloud makes migrations very easy,” says Brett St. Clair, CEO at Siatik. “It takes care of all the hard work with configurations and replications, so when we switch the machines on, everything is ready and working.”
The ease with which PPS migrated to GCP means that it can now tackle strategic goals much more quickly than before. The most ambitious of these is an AI-powered product recommendation platform. Information is collected from customers who opt in at a defined point in their journey, this database is queried using BigQuery, and the information is fed into the platform. The AI model then calculates the most appropriate products for each member, according to their personal history.
“Most of the product recommendation engines out there are based on clustering, where you’re offered products based on your peer groups,” explains Avsharn. “For the first time, we can make recommendations to members based on their individual preferences and historical behavior. That’s really powerful for us.”
Siatik helped PPS use TensorFlow and Cloud Machine Learning Engine to build the AI platform. For the engineers, these easy-to-use tools helped speed up the process considerably, allowing them to host the models locally without any fuss. Previously, it took one to three months to manually build the model and match an offer to a customer. With the AI platform, a match takes just a few minutes. Cloud ML Engine, in particular, helped the platform adapt to new information on the fly and easily make adjustments to its hyperparameters, that is, preset variables which define the model-training process.
“We wanted the platform to retrain its models in response to new data and improve its recommendations with more information,” says Kimoon Kim, Lead Solution Architect and Data Engineer at Siatik. “Normally this would be a manual process but Google Cloud ML Engine lets the models do this automatically.”
“Google Cloud helped us cancel out a lot of the noise around machine learning and AI. We don’t have to build new complicated algorithms or hire huge teams of data scientists to benefit. We just bring our data and use the right tools to focus on what’s really important.”
—Avsharn Bachoo, CTO, PPS
Harnessing artificial intelligence for real-world results
PPS deployed its new AI recommendation platform in December, 2018. Just a couple of months later, its impact was clear. “In around eight weeks, we saw a 5 percent growth in sales,” says Avsharn. “It’s been a direct result of building our recommendation platform with Google Cloud. We can offer the right products to the right members.”
For developers and engineers at PPS, working with Google Cloud gives them access to high performance technology and automation options with GKE. As a result, the infrastructure runs 70 percent faster than before with fewer cores and less memory. Developers can also work in mature testing environments, and for the first time, are able to build pre-production environments, leading to better quality products. More strategically, moving to a serverless, cloud-based infrastructure has helped PPS take control of its budget, moving away from intermittent, large capital spends to more manageable, project-to-project flows of operational expenditure. The company expects to see savings of around 50 percent, or $695,000.
“We have a lot more flexibility with our resources thanks to Google Cloud,” says Avsharn. “When we have a new idea, we don’t have to outlay new capital such as servers before we can even start working on it. We just spin up instances when we want and spin them back down when we’re done.”
With the AI platform deployed and working well, PPS is already looking at ways to improve it, including real-time updates and further automation. Soon, the company will integrate the platform with more sales campaigns for more effective targeting to boost sales even further. Meanwhile, it’s also experimenting with machine learning to spot patterns in data at scale for fraud analytics and risk assessment.
For PPS, working with Google Cloud has helped it transform quickly and effectively from disrupted to disruptor. The company is now looking to gain the same transformative effects by implementing G Suite for increased productivity and collaboration.
“Google Cloud helped us cancel out a lot of the noise around machine learning and AI,” says Avsharn. “We don’t have to build new complicated algorithms or hire huge teams of data scientists to benefit. We just bring our data and use the right tools to focus on what’s really important.”
More Relevant Stories for Your Company
Lending DocAI Shortens Borrowers’ Journey on Roostify
The home lending journey entails processing an immense number of documents daily from hundreds of thousands of borrowers. Currently, home lending document processing relies on some outdated digital models and a high dependency on manual labor, resulting in slow processing times and higher origination costs. Scaling a business that sorts
What’s Google Cloud Firestore Database and What are its Benefits for Business and Developers?
Cloud Firestore is a NoSQL document database that simplifies storing, syncing, and querying data for your mobile and web apps at global scale. Cloud Firestore is a fast, fully managed, serverless, cloud-native NoSQL document database that simplifies storing, syncing, and querying data for your mobile, web, and IoT apps at
The Power of Personalization: Ocado Retail’s Strategy to Boost Revenue and Lower Churn
Retailers are becoming more skilled at making individual customers feel heard and valued. This is a necessity given the fact that 66% of respondents to a McKinsey survey stated that they expect email marketing messages to be tailored to their needs. While marketing personalization expertise is growing, it’s still difficult to manage,
The True Story of How HotStar Broke a World-Record–Thanks to Firebase and Google BigQuery
Hotstar, India’s largest video streaming platform with 150 million monthly active users around the world, provides live-streaming of TV shows, movies, sports, and news on the go. By using a combination of Firebase products together, Hotstar safely rolled out new features to its watch screen during a major live-streaming event
