Editor’s note: Today we hear from ForgeRock, a multinational identity and access management software company with more than 1,100 enterprise customers, including a major public broadcaster. In total, customers use the ForgeRock Identity Platform to authenticate and log in over 45 million users daily, helping them manage identity, governance, and access management across all platforms, including on-premises and multicloud environments. 

Operating at that kind of scale isn’t easy. In this blog post, ForgeRock Engineering Director, Warren Strange discusses the three things that help make their developers efficient and productive, and the Google Cloud tools they use along the way. 

At ForgeRock, we’ve been an early adopter of Kubernetes, viewing it as a strategic platform. Running on Kubernetes allows us to drive multicloud support across Google Kubernetes Engine (GKE), Amazon (EKS), and Azure (AKS). So no matter which cloud our customers are running on, we are able to seamlessly integrate our products into customers’ environments. 

Making it easier for ForgeRock’s developers and operators to build, deploy and manage applications has been crucial in our ability to continually provide high quality solutions for our customers. We’re always looking for tools to improve productivity and keep our developers focused on coding instead of configuration. Google Cloud’s suite of DevOps tools have streamlined three specific practices to help keep our developers productive: 

1. Make developers productive within IDEs

Developer productivity is core to the success of any organization, including ForgeRock. Since developers spend most of their time within their IDE of choice, our goal at ForgeRock has been to make it easier for our developers to write Kubernetes applications within the IDEs they know and love. Cloud Code helps us precisely with that: it makes the process of building, deploying, scaling, and managing Kubernetes infrastructure and applications a breeze. 

In particular, working with the Kubernetes YAML syntax and schema takes time, and a lot of trial and error to master. Thanks to YAML authoring support within Cloud Code, we can easily avoid the complicated and time consuming task of writing YAML files at ForgeRock. With YAML authoring support, developers save time on every bug. Cloud Code’s inline  snippets, completions, and schema validation, a.k.a. “linting,” further streamline working with YAML files. 

The benefits of Cloud Code extend to local development as well. Iterating locally on Kubernetes applications often requires multiple manual steps, including building container images, updating Kubernetes manifests, and redeploying applications. Doing these steps over and over again can be a chore. Cloud Code supports Skaffold under the hood, which tracks changes as they come and automatically rebuilds and redeploys—reducing repetitive development tasks. 

Finally, developing for Kubernetes usually involves jumping between the IDE, documentation, samples etc. Cloud Code reduces this context switching with Kubernetes code samples. With samples, we can get new developers up and running quickly. They spend less time learning about configuration and management of the application—and spend more time on writing and evolving the code.

2. Drive end-to-end automation

To further improve developer productivity, we’ve focused on end-to-end automation: from writing code within IDEs, to automatically triggering CI/CD pipelines and running the code in production. In particular, Tekton, Cloud Build, Container Registry, and GKE have been critical to Forgerock as we streamline the flow of code, feedback and remediation through the build and deployment processes. The process looks something like this:

We begin by developing Kubernetes manifests and dockerfiles using Cloud Code. Then we use Skaffold to build containers locally, while Cloud Build helps with continuous integration (CI). The Cloud Build GitHub app allows us to automate builds and tests as part of our GitHub workflow. Cloud Build is differentiated from other continuous integration tools since it is fully serverless. It scales up and scales down in response to load, with no need for us to pre-provision servers or pay in advance for additional capacity. We pay for the exact resources we use. 

Once the image is built by Cloud Build, it is stored, managed, and secured in Google’s Container Registry. Just like Cloud Build, Container Registry is serverless, so we only pay for what we  use. Additionally, since Container Registry comes with automatic vulnerability scanning, every time we upload a new image to Container Registry, we can also scan it for vulnerabilities. 

Next, a Tekton pipeline is triggered, which deploys the docker images stored in Container Registry and Kubernetes manifests to a running GKE cluster. Along with Cloud Build, Tekton is a critical part of our CI/CD process at ForgeRock. Most importantly, since Tekton comes with standardized Kubernetes-native primitives, we can create continuous delivery workflows very quickly.

After deployment, Tekton triggers a functional test suite to ensure that the applications we deploy perform as expected. The test results are posted to our team Slack channel so all developers have instant access and insights about each cluster. From there, we are able to provide our customers with their finished product request.

3.  Leverage multicloud patterns and practices

The industry has seen a shift towards multicloud. Organizations have adopted multicloud strategies to minimize vendor lock-in, take advantage of best-in-class solutions, improve cost-efficiencies, and increase flexibility through choice. 

At ForgeRock, we’re big proponents of multicloud. Part of that comes from the fact that our identity and access management product works across Google Cloud, AWS, and Azure. Developing products using open-source technologies such as Kubernetes has been particularly helpful in driving this interoperability. 

Tekton has been another critical project that has allowed us to prevent vendor lock-in. Thanks to Tekton, our continuous delivery pipelines can deploy across any Kubernetes cluster. Most importantly, since Tekton pipelines run on Kubernetes, these pipelines can be decoupled from the runtime. Like Tekton and Kubernetes, both Cloud Build and Container Registry are based on open technologies. Community-contributed builders and official builder images allow us to connect to a variety of tools as a part of the build process. And finally, with support for open technologies like Google Cloud buildpacks within Cloud Build, we can build containers without even knowing Docker. 

Making it easier for developers and operators to build, deploy and manage applications is critical for the success of any organization. Driving developer productivity within IDEs, leveraging end-to-end automation, and support for multi-cloud patterns and practices are just some of the ways we are trying to achieve this at ForgeRock. To learn more about ForgeRock, and to deploy the ForgeRock Identity Platform into your Kubernetes cluster, check out our open-source ForgeOps repository on GitHub.

Vodafone India had a manual, labor-intensive process for determining network capabilities and reach. The company sent field operatives to every customer location to conduct a feasibility study. These feasibility studies help Vodafone determine whether they can provide connectivity and services to customers based on the infrastructure at that location.

Everyone from the IT team and end users to the field operatives doing the work recognized the need to adopt a new solution to automate the measurements. They needed a technology that was easy to use and maintain.

“Vodafone used to manually perform physical surveys for each feasibility, which is a time-consuming and labor-intensive process. Often, feasibility studies were delayed, and we missed out on opportunities to serve additional customers. With SmartFeasibility, we’ve increased our capacity 15 fold, which positively impacts our bottom line and allows us to provide better and smarter customer service.”

—Rajneesh Asthana, IT Planning and Delivery, Vodafone India

Partnering with Lepton Software (a leading global provider of location-based analytics solution) Vodafone introduced SmartFeasibility—a solution that changed the feasibility testing from a manual to an automated process. This involved a full Google geo platform solution – leveraging world class technology like maps, roads and directions.

Google Maps Platform Results

• Employees are able to access information faster with SmartFeasibility—they have data at their fingertips, rather than waiting for an employee to collect it
• Field operatives have increased their conversion rates by providing more accurate readings on feasibilities and closing more customer business
• Addresses are now easy to find with a click of a button. The Vodafone India team can search feasibilities that have been loaded into the database, so if there’s an issue or if they need to reference a past action, they have that information at their fingertips
• 2 day turnaround versus 5 before the solution was implemented
• 400+ new customers added per day

With the new solution, Vodafone India no longer needs field operatives to manually calculate these measurements. With Google Maps, users can search customers’ addresses, calculate the distance between Vodafone’s location and the customer’s location and research building data such as height.

The old system of having field operatives collect data was unreliable. With Google Maps Platform, the Vodafone India team knows that the measurements are accurate and reliable.

Picking the right database for your application is not easy. The choice depends heavily on your use case—transactional processing, analytical processing, in-memory database, and so on—but it also depends on other factors. This post covers the different database options available within Google Cloud across relational (SQL) and non-relational (NoSQL) databases and explains which use cases are best suited for each database option. 

Relational databases 

In relational databases information is stored in tables, rows and columns, which typically works best for structured data. As a result they are used for applications in which the structure of the data does not change often. SQL (Structured Query Language) is used when interacting with most relational databases. They offer ACID consistency mode for the data, which means:

• Atomic: All operations in a transaction succeed or the operation is rolled back.
• Consistent: On the completion of a transaction, the database is structurally sound.
• Isolated: Transactions do not contend with one another. Contentious access to data is moderated by the database so that transactions appear to run sequentially.
• Durable: The results of applying a transaction are permanent, even in the presence of failures.

Because of these properties, relational databases are used in applications that require high accuracy and for transactional queries such as financial and retail transactions. For example: In banking when a customer makes a funds transfer request, you want to make sure the transaction is possible and it actually happens on the most up-to-date account balance, in this case an error or resubmit request is likely fine.

There are three relational database options in Google Cloud: Cloud SQL, Cloud Spanner, and Bare Metal Solution.

Cloud SQL: Provides managed MySQL, PostgreSQL and SQL Server databases on Google Cloud. It reduces maintenance cost and automates database provisioning, storage capacity management, back ups, and out-of-the-box high availability and disaster recovery/failover. For these reasons it is best for general-purpose web frameworks, CRM, ERP, SaaS and e-commerce applications.

Cloud Spanner: Cloud Spanner is an enterprise-grade, globally-distributed, and strongly-consistent database that offers up to 99.999% availability, built specifically to combine the benefits of relational database structure with non-relational horizontal scale. It is a unique database that combines ACID transactions, SQL queries, and relational structure with the scalability that you typically associate with non-relational or NoSQL databases. As a result, Spanner is best used for applications such as gaming, payment solutions, global financial ledgers, retail banking and inventory management that require ability to scale limitlessly with strong-consistency and high-availability. 

Bare Metal Solution: Provides hardware to run specialized workloads with low latency on Google Cloud. This is specifically useful if there is an Oracle database that you want to lift and shift into Google Cloud. This enables data center retirements and paves a path to modernize legacy applications. 

Non-relational databases

Non-relational databases (or NoSQL databases) store compex, unstructured data in a non-tabular form such as documents. Non-relational databases are often used when large quantities of complex and diverse data need to be organized. Unlike relational databases, they perform faster because a query doesn’t have to access several tables to deliver an answer, making them ideal for storing data that may change frequently or for applications that handle many different kinds of data. 

For example, an apparel store might have a database in which shirts have their own document containing all of their information, including size, brand, and color with room for adding more parameters later such as sleeve size, collars, and so on.

Qualities that make NoSQL databases fast:

• Eventual consistency: stores usually exhibit consistency at some later point (e.g., lazily at read time)
• Horizontal scaling, usually using hashed distributions
• Typically, they are optimized for a specific workload pattern (i.e., key-value, graph, wide-column)
• Typically, they don’t support cross shard transactions or flexible isolation modes.

Because of these properties, non-relational databases are used in applications that require large scale, reliability, availability, and frequent data changes.They can easily scale horizontally by adding more servers, unlike some relational databases, which scale vertically by increasing the machine size as the data grows. Although, some relations databases such as Cloud Spanner support scale-out and strict consistency.

Non-relational databases can store a variety of unstructured data such as documents, key-value, graphs, wide columns, and more. Here are your non-relational database options in Google Cloud: 

• Document databases: Store information as documents (in formats such as JSON and XML). For example: Firestore
• Key-value stores: Group associated data in collections with records that are identified with unique keys for easy retrieval. Key-value stores have just enough structure to mirror the value of relational databases while still preserving the benefits of NoSQL. For example: Datastore, Bigtable, Memorystore
• In-memory database: Purpose-built database that relies primarily on memory for data storage. These are designed to attain minimal response time by eliminating the need to access disks. They are ideal for applications that require microsecond response times and can have large spikes in traffic. For example: Memorystore
• Wide-column databases: Use the tabular format but allow a wide variance in how data is named and formatted in each row, even in the same table. They have some basic structure while preserving a lot of flexibility. For example: Bigtable
• Graph databases: Use graph structures to define the relationships between stored data points; useful for identifying patterns in unstructured and semi-structured information. For example: JanusGraph

There are three non-relational databases in Google Cloud:

• Firestore: Is a serverless document database which scales on demand and acts as a backend-as-a-service. It is DBaaS that increases the speed of building applications. It is perfect for all general purpose uses cases such as ecommerce, gaming, IoT and real time dashboards. With Firestore users can interact with and collaborate on live and offline data making it great for real-time application and mobile apps.  
• Cloud Bigtable: Cloud Bigtable is a sparsely populated table that can scale to billions of rows and thousands of columns, enabling you to store terabytes or even petabytes of data. It is ideal for storing very large amounts of single-keyed data with very low latency. It supports high read and write throughput at sub-millisecond latency, and it is an ideal data source for MapReduce operations. It also supports the open-source HBase API standard to easily integrate with the Apache ecosystem including HBase, Beam, Hadoop and Spark along with Google Cloud ecosystem.
• Memorystore: Memorystore is a fully managed in-memory data store service for Redis and Memcached at Google Cloud. It is best for in-memory and transient data stores and automates the complex tasks of provisioning, replication, failover, and patching so you can spend more time coding. Because it offers extremely low latency and high performance, Memorystore is great for web and mobile, gaming, leaderboard, social, chat, and news feed applications.

Conclusion

Choosing a relational or a non-relational database largely depends on the use case. Broadly, if your application requires ACID transactions and your data structure is not going to change much, select a relational database. 

In Google Cloud use Cloud SQL for any general-purpose SQL database and Cloud Spanner for large-scale globally scalable, strongly consistent use cases. In general, if your data structure may change later and if scale and availability is a bigger requirement than consistency then a non-relational database is a preferable choice.  Google Cloud offers Firestore, Memorystore, and Cloud Bigtable to support a variety of use cases across the document, key-value, and wide column database spectrum.

For more comparison resources on each database check out the overview. For more hands-on experience with Bigtable, check out our on-demand training here and learn about migrating databases to managed services check out this whitepaper.  

https://youtube.com/watch?v=2TZXSnCTd7E%3Fenablejsapi%3D1%26

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.

As we become increasingly reliant on technology to make decisions, geospatial data is becoming more critical than ever. It is a powerful resource that can be used to solve a variety of problems, from tracking the movement of goods, identifying interest areas and potential areas of disaster.

Geospatial data incorporates data with a geographic component, such as latitude and longitude coordinates, addresses, postal codes, or place names, and can be obtained from a variety of sources, including satellites, sensors, and surveys, making it an incredibly powerful tool with a broad range of applications.

One of the key features of BigQuery, Google Cloud’s serverless enterprise data warehouse, is its ability to analyze geospatial data. However, oftentimes geospatial data sits in a variety of public clouds, not just Google Cloud. To access it effectively, you need a multi-cloud analytics solution that lets you capitalize on the distinct capabilities of each cloud platform, while extracting insights and value from data sitting across multiple cloud platforms.

BigQuery Omni is a multi-cloud analytics solution that enables the analysis of data stored across public cloud environments, including Google Cloud, Amazon Web Services (AWS) and Microsoft Azure, without the need to transfer the data. With BigQuery Omni, users can employ the same SQL queries and tools used to analyze data in Google Cloud to analyze data in other clouds, making it easier to gain insights from all data, regardless of the storage location. For businesses using multiple clouds, BigQuery Omni is an excellent tool to unify analytics and optimize the value of data.

With BigQuery Omni, organizations can analyze location-based information or geographic components, such as latitude and longitude coordinates, addresses, postal codes, or place names without even copying their data to Google Cloud. For example, you could use BigQuery Omni to analyze data from a fleet of delivery vehicles to track their location and identify potential problems.

BigQuery Omni and geospatial data analysis

If you are working with geospatial data, BigQuery Omni is a powerful tool that can help you to get insights from your data. It is scalable, reliable, and secure, making it a great option for unifying your analytics and getting the most out of your data. 

Here are a few examples of ways organizations might use BigQuery Omni for geospatial data:

• A transportation company could use BigQuery Omni to analyze data from GPS sensors in its vehicles to track the movement of its fleet and identify potential problems.
• A retail company could use BigQuery Omni to analyze data from its point-of-sale systems to track customer behavior and identify trends.
• A government agency could use BigQuery Omni to analyze data from its weather sensors to track the movement of storms and identify areas at risk of flooding.

BigQuery Omni and geospatial data can be used together to gain insights into a variety of business problems. Specifically, some of the advantages of using BigQuery Omni and geospatial data include:

• Access to quality geospatial data: BigQuery supports loading of newline-delimited GeoJSON files and provides built-in support for loading and querying geospatial data. Data from public data sources like BigQuery public datasets, the Earth Engine catalog, and the United States Geological Survey (USGS) can be easily integrated into your BigQuery environment. Earth Engine has an integrated data catalog with a comprehensive collection of analysis-ready datasets, including satellite imagery and climate data. This data can be combined with proprietary data sources such as SAP, Oracle, Esri ArcGIS Server, Carto, and QGIS.
• Loading and preprocessing of geospatial data: BigQuery has built-in support for loading and querying geospatial data types, and you can use partner solutions such as FME Spatial ETL to load data.
• Working with different geospatial data types and formats: BigQuery supports a variety of file types and formats including WKT, WKB, CSV and GeoJSON.
• Coordinate reference systems: BigQuery’s geography data type is globally consistent. That means that your data is registered to the WGS84 reference system and your analyses can span a city block or multiple continents.

Overall, geospatial analytics with BigQuery Omni provides a wide range of technical capabilities for processing and analyzing geospatial data, making it a powerful tool for businesses that need to work with location-based data.

Analyzing geospatial data with BigQuery Omni

Imagine a retailer who has a large chain of department stores with locations all over the country. They are looking to expand their business and want to identify areas with high sales potential. They want a way to get a better understanding of their sales volume within specific geographic boundaries. To achieve this goal, the retailer turns to the GIS (Geographic Information System) functions built into BigQuery. Here are the steps what the retailer takes to analyze this dataset:

Step 1 : An initial orders dataset on AWS S3 contains 5.54 million rows, and with separate locations (300 rows) and zipcode (33144 rows) metadata files on AWS S3.

Orders Parquet files:

Location and Zipcode files:

Step  2 : The retailer uses BigQuery Omni to establish a connection between the data stored in AWS and BigQuery, enabling them to access the S3 datasets externally.

External Connection for AWS

External Table for orders

External Table for locations

External Table for zipcode

Step 3 : They combine the orders and locations datasets using BigQuery Omni, and remotely aggregate the data on AWS. Joining this dataset with geospatial datasets helps them derive geospatial coordinates.

The final aggregated dataset is reduced to 23 rows. Subsequently, they bring back the result dataset, which contains just 23 rows. This helps them reduce the extraction of millions of rows to just 23 rows for their geospatial analytics.

Select sales.store_city store_city,
sales.number_of_sales_last_10_mins number_of_sales_last_10_mins,
sales.store_zip store_zip,
 ST_GeogPoint(zip_lat_lng.longitude ,
   zip_lat_lng.latitude  ) geo
FROM (
 SELECT
   FORMAT_DATETIME("%X",
     CURRENT_DATETIME("America/Los_Angeles")) current_time,
   MAX(DATETIME(time_of_sale,
       "America/Los_Angeles")) time_of_last_sale,
   COUNT(1) number_of_sales_last_10_mins,
   locations.city store_city,
   locations.zip store_zip
 FROM
   `bqomni-blog.aws_locations.orders_small`  sales
 JOIN
   `bqomni-blog.aws_locations.locations` locations
 ON
   sales.store_id = locations.id
 GROUP BY
   locations.city,
   locations.zip ) sales
JOIN
   `bqomni-blog.aws_locations.zipcode` zip_lat_lng
ON
 cast(sales.store_zip as INT) = cast(zip_lat_lng.zipcode as INT)
 WHERE ST_WITHIN( ST_GeogPoint(zip_lat_lng.longitude , zip_lat_lng.latitude  ) ,ST_GeogFromText(zip_lat_lng.zipcode_geom ) )
 AND zip_lat_lng.state_name  = "New York"
 ORDER BY number_of_sales_last_10_mins

Aggregated Sales data by region

To build richer views of their sales volume data, the retailer uses BigQuery GeoViz integration, a powerful tool that allows for visualizing geographic data on maps.BigQuery Geo Viz is a web tool for visualization of geospatial data in BigQuery using Google Maps APIs. You can run a SQL query and display the results on an interactive map

BigQuery Geo view for sales data analysis

With the geo-tagged data in place, the retailer can now see regional sales volume, sales density, distribution by department by time, and distribution within department, all powered through BigQuery.

Satellite view

Benefits of using BigQuery Omni 

Beyond geospatial analysis, BigQuery Omni offers a number of benefits, including:

Reduced costs: BigQuery Omni’s ability to eliminate data transfers between clouds can help organizations reduce costs and simplify data management, making it a valuable tool for multi-cloud analytics. Also, the ability to access and analyze data across multiple clouds can reduce the need for data replication and synchronization, which can further simplify the ETL process and improve data consistency.

Unified governance: BigQuery Omni uses the same security controls as BigQuery, which include features such as encryption, access controls, and audit logs, to help protect data from unauthorized access.

Single pane for analytics: BigQuery Omni provides a single interface for querying data across all three clouds, which can simplify the process of analyzing data and reduce the need for organizations to use multiple analytics tools.

Flexibility: Analyze data stored in any of the supported cloud storage services, giving organizations the flexibility to work with the data they have regardless of where it’s located.

BigQuery Omni is a valuable tool for geospatial analysis because it allows you to analyze data from multiple sources without having to move the data. This can save you time and money, and it can also help you to get more accurate insights from your data.If you are looking for a way to improve the accuracy, efficiency, and decision-making of your business, using BigQuery Omni to analyze geospatial data can be a powerful tool.

References

• Omni-introduction 
• Omni-benefits 
• Omni-aws-create-connection
• Omni-azure-create-connection 
• BigQuery Omni Retail Demo on Looker
• Omni-pricing 
• Working with Geospatial Data
• Geospatial Analytics – Intro

Learn more about how BigQuery Omni can help your organization.