GCP for Bioinformatics - Build What's Next

2857

Of your peers have already watched this video.

17:30 Minutes

The most insightful time you'll spend today!

Explainer

GCP for Bioinformatics

Join Cloud GDE and developer Lynn Langit in this fast-paced session to get resources you can use to learn how to use the Google Cloud Platform for bioinformatics.

Lynn has created an open source course (on GitHub) to introduce researchers to using GCP to scale their analysis jobs. In this short talk, she’ll guide you through her course materials, so that you can get started learning using examples from genomics.

She talks is divided into four parts:

  • What is needed?
  • Why do we need to have patterns?
  • How can you use pattern information,
  • How can you learn more.
How-to

Build Open Data Platform on GCP with Delta Lake, Presto and Dataproc

3410

Of your peers have already read this article.

5:00 Minutes

The most insightful time you'll spend today!

Many organizations deal with data from on-prem sources and on cloud. To build and deploy open data lake architecture on GCP with Presto, Delta Lake and Dataproc here is a step-by-step guide.

Organizations today build data lakes to process, manage and store large amounts of data that  originate from different sources both on-premise and on cloud. As part of their data lake strategy, organizations want to leverage some of the leading OSS frameworks such as Apache Spark for data processing, Presto as a query engine and Open Formats for storing data such as Delta Lake for the flexibility to run anywhere and avoiding lock-ins.

Traditionally, some of the major challenges with building and deploying such an architecture were:

  • Object Storage was not well suited for handling mutating data and engineering teams spent a lot of time in building workarounds for this
  • Google Cloud provided the benefit of running Spark, Presto and other varieties of clusters with the Dataproc service, but one of the challenges with such deployments was the lack of a central Hive Metastore service which allowed for sharing of metadata across multiple clusters.
  • Lack of integration and interoperability across different Open Source projects

To solve for these problems, Google Cloud and the Open Source community now offers:

  • Native Delta Lake support in Dataproc, a managed OSS Big Data stack for building a data lake with Google Cloud Storage, an object storage that can handle mutations
  • A managed Hive Metastore service called Dataproc Metastore which is natively integrated with Dataproc for common metadata management and discovery across different types of Dataproc clusters
  • Spark 3.0 and Delta 0.7.0 now allows for registering Delta tables with the Hive Metastore which allows for a common metastore repository that can be accessed by different clusters.

Architecture

Here’s what a standard Open Cloud Datalake deployment on GCP might consist of:

  1. Apache Spark running on Dataproc with native Delta Lake Support
  2. Google Cloud Storage as the central data lake repository which stores data in Delta format
  3. Dataproc Metastore service acting as the central catalog that can be integrated with different Dataproc clusters
  4. Presto running on Dataproc for interactive queries
Architecture

Such an integration provides several benefits:

  1. Managed Hive Metastore service
  2. Integration with Data Catalog for data governance
  3. Multiple ephemeral clusters with shared metadata
  4. Out of the box integration with open file formats and standards

Reference implementation

Below is a step by step guide for a reference implementation of setting up the infrastructure and running a sample application

Setup

The first thing we would need to do is set up 4 things:

  1. Google Cloud Storage bucket for storing our data
  2. Dataproc Metastore Service
  3. Delta Cluster to run a Spark Application that stores data in Delta format
  4. Presto Cluster which will be leveraged for interactive queries

Create a Google Cloud Storage bucket

Create a Google Cloud Storage bucket with the following command using a unique name.

  gsutil mb gs://<your-bucket-name>

Create a Dataproc Metastore service

Create a Dataproc Metastore service with the name “demo-service” and with version 3.1.2. Choose a region such as us-central1. Set this and your project id as environment variables.

  REGION=<your-region>
PROJECT_ID=<your-project-id>
gcloud metastore services create demo-service \
    --hive-metastore-version=3.1.2 \
    --location=${REGION}

Create a Dataproc cluster with Delta Lake

Create a Dataproc cluster which is connected to the Dataproc Metastore service created in the previous step and is in the same region. This cluster will be used to populate the data lake. The jars needed to use Delta Lake are available by default on Dataproc image version 1.5+

  gcloud dataproc clusters create delta-cluster \
    --dataproc-metastore=projects/${PROJECT_ID}/locations/us-central1/services/demo-service \
    --region=${REGION} \
    --image-version=2.0.0-RC22-debian10

Create a  Dataproc cluster with Presto 

Create a Dataproc cluster in us-central1 region with the Presto Optional Component and connected to the Dataproc Metastore service.

  gcloud dataproc clusters create presto-cluster \
    --dataproc-metastore=projects/${PROJECT_ID}/locations/us-central1/services/demo-service \
    --region=${REGION} \
    --image-version=2.0-debian10 \
    --optional-components=PRESTO \
    --enable-component-gateway

Spark Application

Once the clusters are created we can log into the Spark Shell by SSHing into the master node of our Dataproc cluster “delta-cluster”.. Once logged into the master node the next step is to start the Spark Shell with the delta jar files which are already available in the Dataproc cluster. The below command needs to be executed to start the Spark Shell. Then, generate some data.

  spark-shell --jars /usr/lib/delta/jars/delta-core.jar
import io.delta.tables._
import org.apache.spark.sql.functions._
// Simulate application data
val orig_df = Seq(
  (1L, 3.0), (2L, -1.0), (3L, 0.0)
).toDF("x", "y")

# Write Initial Delta format to GCS

Write the data to GCS with the following command, replacing the project ID.

  orig_df.write.mode("append").format("delta").save("gs://<your-bucket-name>/first-delta-table")

# Ensure that data is read properly from Spark

Confirm the data is written to GCS with the following command, replacing the project ID.

  spark.read.format("delta").load("gs://<your-bucket-name>/first-delta-table").show()

Once the data has been written we need to generate the manifest files so that Presto can read the data once the table is created via the metastore service.

# Generate manifest files

  val deltaTable = DeltaTable.forPath("gs://<your-bucket-name>/first-delta-table")
deltaTable.generate("symlink_format_manifest")

With Spark 3.0 and Delta 0.7.0 we now have the ability to create a Delta table in Hive metastore. To create the table below command can be used. More details can be found here 

# Create Table in Hive metastore

  spark.sql("CREATE TABLE my_first_table (x bigint,y double) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'gs://<your-bucket-name>/first-delta-table/_symlink_format_manifest'")

Once the table is created in Spark, log into the Presto cluster in a new window and verify the data. The steps to log into the Presto cluster and start the Presto shell can be found here.

#Verify Data in Presto

  presto:default> select * from hive.default.my_first_table;
 x  |  y  
----+------
  2 | -1.0
  3 |  0.0
  1 |  3.0

Once we verify that the data can be read via Presto the next step is to look at schema evolution. To test this feature out we create a new dataframe with an extra column called “z” as shown below:

# Schema Evolution in Spark

Switch back to your Delta cluster’s Spark shell and enable the automatic schema evolution flag

  spark.sql("SET spark.databricks.delta.schema.autoMerge.enabled = true")

Once this flag has been enabled  create a new dataframe that has a new set of rows to be inserted along with a new column 

  val merge_df = Seq(
  (10L, 30.0, "a"), (20L, -10.0, "b"), (30L, 100.0, "c")
).toDF("x", "y", "z")

Once the dataframe has been created we leverage the Delta Merge function to UPDATE existing data and INSERT new data 

# Use Delta Merge Statement to handle automatic schema evolution and add new rows

  deltaTable.alias("o").merge(merge_df.as("n"),"o.x = n.x").whenMatched.updateAll().whenNotMatched.insertAll().execute()

As a next step we would need to do two things for the data to reflect in Presto:

  1. Generate updated schema manifest files so that Presto is aware of the updated data
  2. Modify the table schema so that Presto is aware of the new column.

When the data in a Delta table is updated you must regenerate the manifests using either of the following approaches:

  • Update explicitly: After all the data updates, you can run the generate operation to update the manifests.
  • Update automatically: You can configure a Delta table so that all write operations on the table automatically update the manifests. To enable this automatic mode, you can set the corresponding table property using the following SQL command.
  ALTER TABLE delta.<path-to-delta-table> SET TBLPROPERTIES(delta.compatibility.symlinkFormatManifest.enabled=true)

However, in this particular case we will use the explicit method to generate the manifest files again

  deltaTable.generate("symlink_format_manifest")

Once the manifest file has been re-created the next step is to update the schema in Hive metastore for Presto to be aware of the new column. This can be done in multiple ways, one of the ways to do this is shown below:

# Promote Schema Changes via Delta to Presto

  val schema_evolution = "ALTER TABLE my_first_table ADD COLUMN ( " + merge_df.schema.toDDL.replace(orig_df.schema.toDDL,"").substring(1) + ")"
spark.sql(s"$schema_evolution")

Once these changes are done we can now verify the new data and new column in Presto as shown below:

# Verify changes in Presto

  presto:default> select * hive.default.from my_first_table;
 x  |   y   |  z  
----+-------+------
 20 | -10.0 | b    
  2 |  -1.0 | NULL
  3 |   0.0 | NULL
 30 | 100.0 | c    
 10 |  30.0 | a    
  1 |   3.0 | NULL

In summary, this article demonstrated:

  1. Set up the Hive metastore service using Dataproc Metastore, spin up Spark with Delta lake and Presto clusters using Dataproc
  2. Integrate the Hive metastore service with the different Dataproc clusters
  3. Build an end to end application that can run on an OSS Datalake platform powered by different GCP services

Next steps

If you are interested in building an Open Data Platform on GCP please look at the Dataproc Metastore service for which the details are available here and for details around the Dataproc service please refer to the documentation available here. In addition, refer to this blog which explains in detail the different open storage formats such as Delta & Iceberg that are natively supported within the Dataproc service.

Whitepaper

Modernize the Data Stack to Transform the Data Experience

DOWNLOAD WHITEPAPER

3472

Of your peers have already downloaded this article

10:30 Minutes

The most insightful time you'll spend today!

As organizations look to revolutionize how they analyze and utilize data, modernizing the data-centric technology stack is critical to success.

Today, the traditional stack poses several challenges—too many steps, too many tools, and too many integrations—all leading to operational complexity, time delays, and high cost. Simplifying the data pipeline, data lifecycle, and data stack offers organizations improved efficiency as well as cost savings, and provides them more value by freeing up resources to focus on deriving insight through the analysis of data.

As organizations begin to transform their approach to analytics, modernizing the analytics stack is a top priority.

To address both the data supply and data demand, data teams must look for ways to simplify and optimize the data pipeline. That means transitioning traditional solutions, like data warehouses and business intelligence tools, to modern architectures built with scalability, agility, and availability in mind.

And what follows with a modernized stack is an ideal data experience rich in actionable insight, API-driven application integration, and the ability to address the real-time needs of a dynamic business.

To find out the advantages to modernizing data warehouses and how this can be accomplished, download, ESG’s report: Modernize the Data Stack to Transform the Data Experience.

Case Study

Making Mothers’ Day Special: How Google Cloud Migration for 1-800-FLOWERS.COM, Inc Impacts CX

6849

Of your peers have already read this article.

2:30 Minutes

The most insightful time you'll spend today!

Leading gift company, 1-800-FLOWERS.COM, Inc., has an interesting way of making mothers' day special! By migrating e-Commerce and other services to Google Cloud, they digitally transformed workflows, launched new brands and improved CX.

Editor’s note: In honor of Mother’s Day, we look at how 1-800-FLOWERS.COM, Inc. migrated to Google Cloud as part of its digital transformation to quickly deploy seamless and convenient customer experiences across multiple brands on Mother’s Day and every day. 

As a leading provider of gifts designed to help customers express, connect, and celebrate,  1-800-FLOWERS.COM, Inc. has embraced cloud technologies to grow and transform its business through constant innovation. As part of our digital transformation, we recently completed the migration of our ecommerce platform and other services to Google Cloud. We’ve transitioned from a monolithic to a microservices platform, moved many workloads from our on-premises data centers to our Google Cloud environment, and scaled both horizontally and vertically. 

Since our migration, we’ve developed efficient processes to launch new brands, improved the customer experience across all brands, and seen significantly increased site traffic. 

Nurturing a more delightful customer journey

Customer delight is at the core of everything we do. Whether it be with a flower bouquet, a sweet treat, or a personalized keepsake, our mission is to deliver smiles. With the rise of the COVID-19 pandemic, we’ve all been challenged to find unique and safe ways to continue honoring the special connections in our lives and celebrating occasions with loved ones. Our customers have adapted by doing things such as sending gifts to isolated loved ones, sharing the same meal together virtually, or using video to engage with others through group activities like flower arranging and building charcuterie boards. 

The customer experience is a top priority for us, and we constantly look for innovative ways to enhance the customer journey across our ecommerce platform of more than a dozen brands. As a result, we’ve continued to see a rise in demand as customers enjoy the ease and convenience of our site and discover our full family of brands. 

Migrating to a cloud-first mindset

As we’ve continued to innovate and iterate on the customer experience, we knew we wanted to evolve our platform. We wanted to shift to a microservices platform, which would allow our team the opportunity to release updates to our site more often and set up the right continuous integration/continuous deployment (CI/CD) practices. 

Working with Google Cloud, we were able to move our ecommerce platform to the cloud and standardize our site and brand deployment by building one release that could then be repeated across all of our brands. We built everything in a modular fashion, including microservices and      code libraries, so that sites could be easily constructed and replicated for each brand. And because of this, we were able to launch Shari’s Berries extremely quickly after we acquired the brand in 2019.   

Moving our platform to a completely homegrown solution of microservices was a daunting task. But our team handled it beautifully through load-testing, stress-testing, and building new monitoring tools. And with Google Cloud supporting us all along the way, managing the migration process was simple and easy from start to finish. 

Arranging a better bouquet of services

Currently, we’ve migrated every customer-facing touchpoint for all of our brands to Google Cloud—whether it’s on the web or mobile, our AI bots, or our chat interfaces. 

  • We run on Google Kubernetes Engine and Istio.
  • We have nearly 200 microservices built to help power our entire ecommerce stack across several cloud services running on Google Cloud. 
  • We’re utilizing BigQuery for our offline intelligence.

Results are coming up roses

Our new stack on Google Cloud has benefits for both us and our customers. We moved from a session-based to a token-based system, which provides enhanced security as well as a consistent, convenient experience across all our brands. Using service workers and a single-page app, we are able to download all the relevant site content to the browser in under two seconds to create an instant-click experience for each and every customer. We also use Google Analytics to measure our user interactions and provide personalized results to each customer. Our hope is that with this new system, we can learn from customer behavior to offer gift givers a more personalized shopping experience during each visit. 

The benefits of our new tech stack have not only helped us enhance the solutions we offer to customers today, they’ve also enabled us to offer new ones at lightning speed. With our legacy system, we used to release new code once a week or once a month. Now, even during our peak periods, we’re able to release 10 to 15 times a day and can deploy and pivot quickly to create new microservices and microsites on the fly—often without having to touch any code. 

Efficiencies abound     

The benefits of moving our platform to Google Cloud have extended to our internal teams as well. Before the migration, we had only two environments for developing and testing, which made it time-consuming to test updates before they went into production. Now with Google Cloud, we have several different journey teams—which are made up of developers, product owners, and technical owners—all working in several different environments, solving problems, and creating new solutions together. 

Everyone is now empowered to be self-sufficient, developing and releasing microservices on their own when they’re ready. This has given our developers more time to take part in continued development and learning opportunities. For example, we offer lunch-and-learn sessions as well as other resources for everyone to take advantage of so they can continue to learn and refine their skills.

Planting the seeds for future growth

As we look to the future and think about how we help our customers express, connect, and celebrate, we’ll continue to collaborate across teams to deliver solutions that spread smiles. Specifically, we’re exploring additional use of AI to help us better serve our customers across all our brands. 

We’ve enjoyed the ongoing support we’ve received from the Google Cloud team as they help us build new solutions and design a road map for the future. Their support has helped the 1-800-FLOWERS.COM, Inc. team to realize the power of the cloud and bring the very best experience to our customers.

Learn more about 1-800-FLOWERS.COM, Inc., or check out our recent blog about cloud migration for the real world.

Blog

Cart.com to Transform e-Commerce for Brands Globally

8794

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

Cart.com supported by the Startup Program by Google Cloud and Google Cloud solutions is set out to democratize e-commerce by empowering brands of all sizes with its unified platform to unlock customer data and business value. Read now!

The ecommerce playing field has been hard to navigate for most retailers, and Cart.com is on a mission to change that. Traditionally, retailers needing to run their online store, order fulfillment, customer service, marketing, and other essential activities have had to cobble together systems to get the capabilities they need – much less having access to analytics across these functions. The result is costly, siloed ecommerce operations that are difficult to manage and scale.

It’s clearly not a formula for success, yet that’s the reality facing most retailers. Cart.com, in contrast, has set out to democratize ecommerce by giving brands of all sizes the full capabilities they need to take on the world’s largest online retailers. Our end-to-end environment empowers retailers to keep more of their revenue, set up proven strategies for managing all aspects of their business, and act on valuable insights from customer data every step of the way.

Together with our talented team, we’re building a unified ecommerce platform that already provides value to many leading or up and coming brands including Whataburger, GUESS, Dr. Scholl’s, Rowing Blazers, and Howler Bros. 

We’re excited about the opportunity ahead as we reimagine traditional approaches to online sales, fulfillment, marketing, accessing growth capital, providing a unified view of all ecommerce and marketing analytics, and other activities. Expectations for Cart.com are high, and we are building a company that can scale to $100B in revenue and beyond. Supported by the Startup Program by Google Cloud and Google Cloud solutions, we’re establishing a technology platform to transform all aspects of ecommerce for brands worldwide. 

Partner in disruption

At Cart.com, we’re currently targeting an underserved market. Our ideal customer is beyond demonstrating product-market-fit and is now at an inflection point seeking a growth opportunity. Typically, those companies are generating between $1M and $100M in annual revenue. We’ve seen an enthusiastic response from brands and retailers as well as investors, with backing from investors in just over a year totaling $143 million in three funding rounds.

Our strategy is to build an integrated ecommerce model that combines best-of-breed solutions, many of which we gain through acquisitions and then build upon to provide a streamlined and fully integrated experience for our brands. We’ve made seven acquisitions so far to round out our online store, order fulfillment, marketing services, customer service, and we have launched some integral partnerships including easy access to growth capital through our relationship with Clearco and product protection for customers on every purchase with Extend. Instead of acquiring a data company, we’re building our data platform on Google Cloud, across each operating function for a single-view for brands to harness actionable data. We see Google Cloud as the leader for data management, analytics, machine learning (ML) and artificial intelligence (AI).

Other reasons why we’re building our business on Google Cloud include scalability, excellence, security, reach, and data analytics that are far superior to other environments.

We also feel a cultural and mission alignment with Google Cloud and envision leaning into a long-term partnership of marketing, selling, and disrupting the disruptors together. Equally important to us are the investments Google Cloud is willing to make in early-stage companies like ours. The support through the Google Cloud for Startups program has been outstanding.

Built on Google Cloud

A wide range of Google Cloud solutions provide the foundation for our platform. For instance, Cloud Pub/Sub keeps our services communicating with one another. We rely on fully managed relational databases, like Cloud SQL and Cloud Spanner, to securely handle the huge volume of brand and shopper data generated every day.

Cloud Run allowed us to develop inside of containers before our Kubernetes infrastructure was ready to go. Now, we are taking advantage of all the capabilities in Google Kubernetes Engine. BigQuery integrates with all Google Cloud solutions and offers true data streaming natively out of the box, along with Dataflow for advanced analytics. We also use Container Registry to store and manage our Docker container images. Right now, we’re testing Cloud Composer to evaluate using it for data workflow orchestration instead of Apache Airflow.

The openness of the Google Cloud environment is further enabled by Anthos, which we may deploy soon to perform data integrations quickly as we acquire more companies over the next year. For example, if we acquire a company using Azure, we can easily align it with our Google Cloud ecosystem.

Enabling ecommerce 2.0

Recently, our team has been experimenting with Google Cloud Vertex AI and the fully managed services of AI deployment and ML operations. The capabilities would save us substantial time in the management of the ML lifecycle which allows us to focus more on developing proprietary AI that will transform commerce at scale.

Because Google Cloud is so far ahead in data science, our teams benefit from deep Google Cloud expertise as we look to provide brands with unmatched insights into customers to improve services and revenue. We’re also planning to test Recommendations AI among other tools to deploy customer product recommendations and personalization as turnkey productized offerings. Moving forward, we will likely use Bigtable to aid in serving machine learning to hundreds of thousands of brands due to its low latency and scalability.

Fanatical about brand success

We know that our work with Google Cloud for Startups and use of Google Cloud solutions for best-in-class data management, analytics, ML, and AI will enable us to offer even more transformative services to brands.

We also see the opportunity to use our platform and customer insights to break down barriers between brands, enabling retailers to share information and work better together when it’s in their best interests. What we’re building today on Google Cloud is fundamentally changing what’s possible for retailers of any size everywhere. 

As a startup, when recruiting talent or working with prospective customers, it helps to share our success with Google Cloud. We view them as an extension of the Cart.com team. It also validates our business as we continue building a more integrated, holistic approach to commerce that opens new opportunities and drives growth for brands worldwide.

For more details about Cart.com’s vision for unified ecommerce, check out our video.

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

Case Study

Google Cloud Platform Gives Us 5x the Processing Power to Analyze Physician Performance at 75% Lower Cost

6741

Of your peers have already read this article.

6:30 Minutes

The most insightful time you'll spend today!

MD Insider is using ML to help patients figure out which doctors have the best outcomes for specific procedures by analyzing data from thousands of institutions and doctor-patient interactions. That wouldn’t have been possible without Google Cloud.

Patients about to undergo a healthcare procedure understandably want the best medical professionals they can get. But how can they know which doctors have had the most experience and the best outcomes with that particular procedure? How can they make an informed decision about which doctor to select when the information they have is limited to the doctor’s practice area and subjective reviews from other patients?

MD Insider is working to solve that problem using machine learning (ML) to objectively analyze doctor performance. By analyzing data from thousands of institutions and millions of doctor-patient interactions and medical events, MD Insider identifies physician performance insights based on their experience and outcomes. Insights are then integrated into a triage engine, that enables consumers to search for and schedule appointments with providers who meet their clinical criteria and convenience preferences, such as insurances accepted, office hours, locations, and language.

MD Insider also offers robust data APIs to help health systems, health plans, and employers reduce costs and improve quality of care. Payers use the APIs to curate high-quality provider networks and manage provider directories. Examples of MD Insider’s data APIs include Provider Experience and Share of Practice metrics, Provider Quality and Outcomes, Network Modeling, Expert Clinical Search Taxonomy, Find a Provider, Acute-Care Hospital Quality, and Provider and Facility Metadata.

“We use Google Cloud Platform the way the cloud was supposed to be used. By comparison, other cloud providers feel like you’re renting somebody else’s data center.”

Ed Holsinger, Lead Data Engineer and Head of Data Science, MD Insider

MD Insider is continuously ingesting the latest performance data about physicians and analyzing billions of rows of data. Requiring constant scalability, the company was born in the cloud; however, it had difficulty configuring server instances for the optimal balance of memory and CPU, and its Hadoop cluster had to be kept running 24/7. Network performance was often slow for no apparent reason. As a result, failure rates from node timeouts increased, and costs grew along with the data. MD Insider had to estimate its usage and pay up front, and received little financial benefit from sustained use commitments.

Knowing that data would continue to grow, MD Insider decided to move its data services — the most demanding and complex portion of its infrastructure — to Google Cloud Platform (GCP), and took advantage of GCP managed services for container management and big data analytics.

“One of the reasons we decided to move to Google Cloud Platform is because it feels like a unified, well-designed cloud architecture and pricing model,” says Ed Holsinger, Lead Data Engineer and Head of Data Science at MD Insider. “We use Google Cloud Platform the way the cloud was supposed to be used. By comparison, other cloud providers feel like you’re renting somebody else’s data center.”

“Moving to Google Cloud Platform and using Kubernetes Engine gave us 5x the processing power for analyzing physician performance at 75% less cost. Our data scientists have more power than ever before to generate insights for our customers.”

Ed Holsinger, Lead Data Engineer and Head of Data Science, MD Insider

5x the performance, 75% less cost

MD Insider now uses Kubernetes Engine to automate container management and deploy clusters in minutes with just a few clicks. When hundreds of machines are required to analyze a large dataset, automation in Kubernetes Engine deploys ML models as containers, each of which manages the full lifecycle of its task, including scaling up resources, deploying results, and scaling back down when the task is finished. It’s easy for MD Insider to specify exactly how much CPU and memory each container needs, helping maximize performance while reducing costs.

“Moving to Google Cloud Platform and Kubernetes Engine gave us 5x the processing power for analyzing physician performance at 75% less cost,” says Ed. “Our data scientists have more power than ever before to generate insights for our customers. Data scoring jobs that used to take three business days now take four hours.”

Adds Galen Meurer, Senior Software Engineer at MD Insider: “Even if all Google Cloud Platform had to offer was Kubernetes Engine, I would still want to use it. Previously we spent up to 30% of our time managing our container infrastructure, which we can now use for product development.”

A foundation for data science

MD Insider was happy to find that GCP offers a wide variety of managed services. For example, the company is supplementing its Kubernetes Engine clusters with BigQuery for its big data masters and selection jobs, enabling scientists to analyze new and different types of data as well as analyze larger datasets in less time. MD Insider also uses Cloud Storage for big data staging and Cloud Dataproc to run managed Apache Spark clusters for data processing.

“Google Cloud Platform gives us an incredibly powerful cloud architecture for data engineering and data science,” says Eric Wilson, CEO of MD Insider. “That gives our scientists independence, they can do what they need to do without waiting and with no contention between them.”

A developer-friendly platform

Migrating its data services was such a success that MD Insider decided to move the rest of its infrastructure to GCP, including the front end for its web application. Since the migration, MD Insider has experienced no unplanned downtime on GCP, allowing it to easily meet the 99.5% uptime SLA it promises to customers. It’s also taking advantage of Build Triggers in Kubernetes Engine to automate container builds and reduce build times by more than 40%. Production code can be updated in seconds, with no impact to end users other than making new features available.

“GCP has simplified our workflow in so many ways, from intelligent load balancing to content delivery and automating builds,” says Matthew Frey, Software Engineer. “Everything on GCP is cohesive and developer friendly, with a superior UI and better network performance than other cloud providers.”

Ryan Beaini, Senior Software Engineer at MD Insider, agrees: “Since we moved to GCP, our developers are definitely happier. The pain and the headaches we experienced because of the limitations of our previous toolset all went away.”

“We’re a small company, but what we’re doing is incredibly important. We’re helping people make decisions about healthcare providers that could impact their lives and even be life-saving. Google Cloud Platform is helping us make our mission bigger, better, and brighter.”

Eric Wilson, CEO, MD Insider

Securing billions of rows of clinical and non-clinical healthcare data

As a healthcare technology company, MD Insider processes billions of rows of clinical and non-clinical healthcare data. To control user access to GCP, it uses Identity & Access Management (IAM) along with Yubico YubiKeys for hardware-based two-factor authentication when logging into Google Workspace. MD Insider takes comfort that GCP encrypts data at rest by default, and encrypts and authenticates data in transit when data moves outside physical boundaries not controlled by Google or on behalf of Google.

“On GCP, everything that we need to be encrypted for compliance purposes is encrypted, which is fantastic,” says Eric. “When I tell our potential clients and partners about the resources that Google has dedicated to security, it gives them the confidence that their data will be protected.”

Transforming how teams work

As a growing company, MD Insider must collaborate seamlessly between offices in California, Colorado, and Illinois. It relies on Google Workspace for communication and productivity, using DocsSheets, and Slides to drive the business. Employee and team files are stored in Drive, and meetings are conducted via Google Meet with Chromebox for Meetings videoconferencing hardware kits. Google Workspace also helps MD Insider maintain information security by authenticating email domains with digital signatures in Gmail and scanning outgoing email using Gmail Data Loss Prevention (DLP).

“I use Google Workspace every day, and everyone else here does too,” says Eric. “Team Drives are a big time saver for us. We’ve let our previous office software expire, because there’s no need to pay for those licenses anymore.”

Promoting healthcare transparency

With GCP helping MD Insider increase velocity and momentum, the company is making exciting progress. For example, it has entered into a strategic partnership with Zelis Healthcare, which will use MD Insider’s API to provide insights for a next-gen analytics platform that will give health plans unprecedented transparency around physician performance.

“We’re a small company, but what we’re doing is incredibly important,” says Eric. “We’re helping people make decisions about healthcare providers that could impact their lives and even be life-saving. Google Cloud Platform is helping us make our mission bigger, better, and brighter.”

*Google Workspace was formerly known as G Suite prior to Oct. 6, 2020.

More Relevant Stories for Your Company

Blog

Introduction to Cloud Shell Editor

Watch the video to understand how Google's Cloud Shell Editor and its powerful features-packed environment can streamline your development workflows.

Explainer

Accelerating Insights with the Health Analytics Platform

The healthcare industry is beset with a data challenge, specifically clinical data, and challenges around the proliferation of EHR instances and ancillary clinical systems. In this video, Vivian Neilley, Solution Architect, Google Cloud and Marianne Slight, Product Manager, Google Cloud discuss how some of these challenges can be overcome with

Webinar

L’Oréal: Managing Big-data Complexity with Google Cloud

L’Oreal is a global company with a presence in 150 countries worldwide. Between managing all of its brands and requirements for different countries, L’Oreal looks to data to make insightful business decisions. How does L’Oreal unify its data across all its systems and databases? How does L’Oreal make the data

Podcast

Setting up Data Pipelines Easily for Streaming and Non-Streaming Data

We live in a world of incessant data. Our phones, factory equipment, smart watches, health devices, and other IoT devices are constantly streaming data. Some of the most popular uses of machine learning and AI are also based on streaming data. Think real-time credit card fraud detection, or software that

SHOW MORE STORIES