Founded in 2006, PaGaLGuY started as a forum that enabled students, typically aged between 20 and 30 to discuss and seek advice on academic issues. By 2011, PaGaLGuY had increased its traffic to about 250,000 page views per month. The business is now one of India’s largest education networks and provides an app that users can download to their Android and iOS devices.

Over the past six years, PaGaLGuY has extended its service to include video advice from experts on education topics and grown the number of views of its pages to 1.5 million per day. As Head of Technology for the business, Sandeep Kalidindi has played a key role in ensuring PaGaLGuY is as engaging as possible to users. “Because the product is advertising based, the greater the user engagement, the greater the advertising revenue,” Kalidindi explains.

Google Cloud Platform Results

• Supported growth to 1.5 million page views per day and demand spikes that see requests increase from about 90 per second to about 1,200 per second
• Reduced API latency from about 1 second to about 40 milliseconds
• Reduced system administration time from three to four days per week to 30 minutes every two weeks

In 2015, PaGaLGuY’s senior management team decided to deliver an even more relevant experience for users of the education network. “The core thing we had to do was personalise the experience for each and every student that visited PaGaLGuY,” Kalidindi says. “So we had to capture each student’s data to customise what they see when they open the site.”

The business also found traffic to the network was straining its infrastructure. During demand peaks, created by exams involving as many as 5 million students, PaGaLGuY would be inaccessible for periods of 30 minutes to one hour. Furthermore, average API latency had climbed to an unacceptable 1 second, compromising performance.

PaGaLGuY needed to access extensive compute resources to undertake its planned change. Had the business relied on a physical technology architecture to undertake the transformation, it would have had to purchase capacity equivalent to 16 new servers. “There was no way with a small team we could grow to that extent in a short time,” Kalidindi says. “This was the right moment for us to explore cloud services.”

The business established two primary requirements the selected cloud service needed to meet. First, PaGaLGuY had to be able to scale the platform with costs rising only in proportion to the increase in resources consumed. Accordingly, the business would have to minimise the number of employees required to manage the cloud environment. Second, the platform had to give PaGaLGuY easy access to student data and the ability to undertake prompt, granular analysis.

PaGaLGuY reviewed available public cloud services and determined that Google Cloud Platform (GCP) was the best fit for its business. “Google Cloud Platform was considerably more mature than the alternatives, with a high degree of automation and a suite of managed services,” Kalidindi says. PaGaLGuY management then discussed with Google how to optimise cost, performance and availability of its personalised education network on GCP.

With assistance from Google and business transformation specialists Searce, PaGaLGuY was able to deliver the platform into production on GCP in 10 months. “Searce was very proactive in ensuring the environment met our needs and allowing us to gain priority access to Google services in development,” Kalidindi says. “Their team was integral to the success of the migration.”

PaGaLGuY has been running in production in GCP for two years. The education network’s GCP architecture comprises a scalable back-end built on Google App Engine; a managed environment for its containerised applications in Google Kubernetes Engine; messaging-oriented middleware through Google Cloud Pub/Sub; a relational database in Google Cloud SQL; a managed data analytics warehouse running in Google BigQuery; stream and batch data processing through Google Cloud Dataflow; and object storage in Google Cloud Storage.

PaGaLGuY has leveraged GCP services to break down its platform application from a monolithic build to a series of microservices running in Google App Engine that enable independent deployment cycles, minimise test and quality assurance overheads and provide clearer monitoring and logging.

Running on GCP has enabled PaGaLGuY to add new personalisation features and grow fourfold without having to add any new engineers or administrators to accommodate the increased traffic. The business has also used the platform to seamlessly collect and aggregate students’ data for analysis, reporting and delivering a more targeted user experience. Furthermore, PaGaLGuY has been able to provide its management team with direct access to Google BigQuery to scrutinise data rather than require them to wait at least a day to view reports created by the product or technology teams.

Support demand peaks of 1,200 requests per second

“Thanks to Google Cloud Platform, we can easily support demand peaks that see requests per second rise from an average 90 per second to about 1,200 per second for as long as 45 minutes,” Kalidindi says. Due to GCP’s scalability, PaGaLGuY can ensure its education network remains available and performance remains consistent during those periods.

Latency cut to 40 milliseconds

The business has also reduced average API latency from 1 second to about 40 milliseconds. Furthermore, using GCP has enabled PaGaLGuY to automate most of its processes and reduce system administration requirements from three to four days a week across its team members to about half an hour per week.

The performance of GCP has transformed PaGaLGuY’s culture and processes. “Once our team was exposed to Google Cloud Platform and understood the superiority of the platform, our mindset changed from ‘let us do everything on our own’ to ‘let us do what we do best’ and delegate the remainder,” Kalidindi says. The quality of the service provided by GCP means PaGaLGuY effectively considers the cloud provider as part of its team. “We are always eager to see what new services are being launched and are extremely excited about what Google Cloud Platform can provide as part of its roadmap.” he concludes.

Organizations rely on data warehouses to aggregate data from disparate sources, process it, and make it available for data analysis in support of strategic decision-making. BigQuery is the Google Cloud enterprise data warehouse designed to help organizations to run large scale analytics with ease and quickly unlock actionable insights. You can ingest data into BigQuery either through batch uploading or by streaming data directly to unlock real-time insights. As a fully-managed data warehouse, BigQuery takes care of the infrastructure so you can focus on analyzing your data up to petabyte-scale. BigQuery supports SQL (Structured Query Language), which you’re likely already familiar with if you’ve worked with ANSI-compliant relational databases. 

BigQuery unique features

BI Engine – BigQuery BI Engine is a fast, in-memory analysis service that provides subsecond query response times with high concurrency. BI Engine integrates with Google Data Studio and Looker for visualizing query results and enables integration with other popular business intelligence (BI) tools. 

BigQuery ML: BigQuery ML is unlocking machine learning for millions of data analysts. It  enables data analysts and data scientists to build and operationalize machine learning models directly within BigQuery, using simple SQL.

BigQuery Omni – BigQuery Omni is a flexible, multi-cloud analytics solution powered by Anthos that lets you cost-effectively access and securely analyze data across Google Cloud, Amazon Web Services (AWS), and Azure, without leaving the BigQuery user interface (UI). Using standard SQL and familiar BigQuery APIs, you can break down data silos and gain critical business insights from a single pane of glass. 

Data QnA: Data QnA enables self-service analytics for business users on BigQuery data as well as federated data from Cloud Storage, Bigtable, Cloud SQL, or Google Drive. It uses Dialogflow and enables users to formulate free-form text analytical questions, with auto-suggested entities while users type a question.

Connected Sheets -The native integration between Sheets and BigQuery makes it possible for all business stakeholders, who are already quite familiar with spreadsheet tools, to get their own up-to-date insights at any time.

Geospatial data – BigQuery offers accurate and scalable geospatial analysis with geography data types. It supports core GIS functions – measurements, transforms, constructors, and more – using standard SQL.

How does it work?

Here’s how it works: You ingest your own data into BigQuery or use data from the public datasets. Storage and compute are decoupled and can scale independently on demand. This offers immense flexibility and cost control for your business as you don’t need to keep expensive compute resources up and running all the time. As a result, BigQuery is much more cost-effective than traditional node-based cloud data warehouse solutions or on-premises systems. BigQuery also provides automatic backup and restore of your data. 

You can ingest data into BigQuery in batches or stream real-time data from web, IoT, or mobile devices via Pub/Sub. You can also use Data Transfer Service to ingest data from other clouds, on-premises systems or third-party services. BigQuery also supports ODBC and JDBC drivers to connect with existing tools and infrastructure. 

Interacting with BigQuery to load data, run queries, or create ML models can be done in three different ways. You can use the UI in the Cloud Console, the BigQuery command-line tool, or the API via client libraries available in several languages.

When it comes time to visualize your data, BigQuery integrates with Looker as well as several other business intelligence tools across the Google partner ecosystem.

What about security?

BigQuery offers built-in data protection at scale. It provides security and governance tools to efficiently govern data and democratize insights within your organization. 

• Within BigQuery, users can assign dataset-level and project-level permissions to help govern data access. Secure data sharing ensures you can collaborate and operate your business with trust.
• Data is automatically encrypted both while in transit and at rest, ensuring that your data is protected from intrusions, theft, and attacks. 
• Cloud DLP helps you discover and classify sensitive data assets.
• Cloud IAM provides access control and visibility into security policies.
• Data Catalog helps you discover and manage data. 

How much does it cost?

The BigQuery sandbox lets you explore BigQuery capabilities at no cost and confirm that BigQuery fits your needs. With BigQuery you get predictable price-performance: you pay for storing and querying data, and for streaming inserts. Loading and exporting data are free of charge. Storage costs are based on the amount of data stored, and have two rates based on how often the data is changing. Query costs can be either:

• On-demand – you are charged per query by the amount of data processed
• Flat-rate – if you prefer to purchase dedicated resources 

You can start with the pay-as-you-go, on-demand option and later move to flat-rate if that better suits your usage. Or, start with flat-rate, get a better understanding of your usage and move to the pay-as-you-go models for additional workloads. 

To explore BigQuery and its capabilities a bit more, check out the sandbox; and when you’re ready to modernize your data warehouse with BigQuery then check out the documentation to streamline your migration process here. https://www.youtube.com/embed/So-tVyBQt8E?enablejsapi=1&

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.

We’re pleased to announce that Gartner has named Google Cloud a Leader in its 2019 Magic Quadrant report for Operational Database Management Systems (OPDBMS). This news reflects what we hear from our customers: that Google Cloud databases are flexible, open, and easy to use.

These include our fully compatible managed services for popular database engines like MySQL, PostgreSQL, SQL Server and Redis, and scalable cloud-native relational and non-relational databases like Cloud Spanner, Cloud Bigtable, and Cloud Firestore, plus fully managed partner services like MongoDB Atlas, Elastic, and Redis Enterprise. You can also run proprietary database workloads on Google Compute Engine or via our Bare Metal Solution.

Enterprises databases in production
We’ve heard great stories from our customers about their use of Google Cloud databases to run their businesses with ease and flexibility. Our database products meet varying needs for scalability and power.

Gaming company Bandai Namco Entertainment needed fast scalability, a global network, and real-time analytics to serve users its Dragon Ball Legends game. They were initially considering sharded MySQL to handle the scale, but opted for Cloud Spanner. Because it’s strongly consistent, fully managed, and scales seamlessly, Cloud Spanner supported the game’s rollout and allowed millions of worldwide players to compete without downtime. 

And media leader The New York Times found our Cloud Firestore database service as they built a truly real-time collaboration tool that lets multiple writers and editors make changes in docs at the same time, keeping track of what’s newest. Cloud Firestore is designed for just this type of task, since it supports offline and real-time sync. 

E-commerce brand analytics and protection company 3PM Solutions empowers global brands to manage, protect, and grow revenue by using Google Cloud Platform services such as Cloud Bigtable. Using Google Cloud, they’ve been able to analyze 160 million customer reviews of more than 2 million sellers in less than four hours. 

Leanplum built its cloud-native marketing platform on Google Cloud to help marketers personalize and orchestrate cross-channel customer engagement campaigns at scale. “

We were looking for cost-effective cloud services that can reduce engineers’ overhead associated with managing infrastructure,” says Athena Koutsonikolos, Leanplum’s VP of marketing. “Using fully managed services like Cloud SQL, developers can now focus on building what matters to our customers.”

Gartner 2019 Magic Quadrant for Operational Database Management Systems – November 25, 2019, Merv Adrian, Donald Feinberg, Henry Cook. 

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.