Earlier this year, we launched the Google Cloud skills challenge, which provides 30 days of free access to training to build your cloud knowledge and an opportunity to earn skill badges that showcase your Google Cloud competencies. Today, we’re adding a Native App Development track to the skills challenge, joining the Getting Started, Data Analytics, Kubernetes, Machine Learning (ML) and Artificial Intelligence (AI) tracks. 

The Native App Development track is designed for cloud developers who want to learn to build serverless web apps and Google Assistant applications on Google Cloud using Cloud Run and Firebase. Specifically, you’ll have an opportunity to earn three skill badges in the Native App Dev track: Serverless Firebase Development, Serverless Cloud Run Development, and Build Interactive Apps with Google Assistant. To earn a skill badge, you complete a series of hands-on labs and take a final assessment challenge lab to test your skills.

Here’s an overview of each badge.

Serverless Firebase Development

To earn this skill badge, you’ll learn how to build serverless web apps, import data into a serverless database, and build Google Assistant applications using Firebase, Google’s backend-as-service platform for creating mobile and web applications.

Serverless Cloud Run Development

For this badge, you’ll discover how to use Cloud Run, a fully managed serverless platform, to connect and leverage data stored in Cloud Storage. You’ll learn how to use Cloud Run to build a resilient, asynchronous system with Pub/Sub, build a REST API gateway as well as build and expose services. 

Build Interactive Apps with Google Assistant

To earn the final skills badge, you’ll build Google Assistant applications by creating a project in the Actions console, integrating Dialogflow, testing your action in the Actions simulator, and adding Cloud Translation API to your assistant application. 

Ready to jump into the skills challenge? Sign up here. 

You can also check out this quick video below to learn how to join the skills challenge.

While many organizations are driving digital transformation by migrating to the cloud, there are some industries, geographies, and use cases that require a different approach to cloud modernization. Regulated industries such as healthcare, insurance, pharmaceutical, energy, telecommunication, and banking have stringent data residency and sovereignty requirements. Other industries need to meet local data processing requirements, while others require real-time data processing with sub-millisecond latencies, for example to detect defects on manufacturing lines. These use cases demand a combination of edge, on-premises and cloud services for their infrastructure.

With these requirements in mind, Google Cloud launched Google Distributed Cloud powered by Anthos to extend the power of Google Cloud infrastructure and services to the edge (or closer). The underlying infrastructure for this service comes in two variants: a 42U rack filled with compute, storage, and networking devices called Google Distributed Cloud Edge Rack and a 1U appliance called Google Distributed Cloud Edge Appliance.

In this blog post, we discuss the Google Distributed Cloud Edge Appliance and how manufacturing, retail, and automotive industry verticals can use it to address common use cases.

How the appliance works

But first, let’s talk about the Google Distributed Cloud Edge Appliance itself.

Google Distributed Cloud Edge Appliances comprises two components: (1) Distributed Cloud Edge infrastructure and (2) the Distributed Cloud Edge service.

The Google Distributed Cloud Edge service runs on Google Cloud and serves as a control plane for the nodes and clusters running on your appliance. In order to perform remote management of the appliance and to collect metrics, the Distributed Cloud Edge service must be connected to Google Cloud at all times, allowing you to manage your workloads on the edge hardware through the Google Cloud Console. For customers who can’t be connected at all times for data residency or sovereignty reasons, we highly recommend that the appliance be connected to the cloud at least once a month to allow for needed security patches and updates.

Google Distributed Cloud Edge Appliances come with built-in network ports that provide connectivity to the control plane via the internet, Cloud VPN, or Dedicated Interconnect, and to your on-prem network. Each Google Distributed Cloud Edge Appliance is homed to a specific Google Cloud region but it is designed to also use any public Google Cloud endpoint to communicate with the control plane in Google Cloud, allowing you to move these appliances between different geographic locations.

There are two NFS shares on each appliance; one is offline, meaning it does not transfer data to Google Cloud, and the other is online, meaning data saved to that share is synced to Cloud Storage on Google Cloud for further processing. The appliance supports Server Message Block (SMB) and Secure File Transfer Protocols (SFTP) for communication.

Each Google Distributed Cloud Edge Appliance runs Google Distributed Cloud Virtual, enabling you to build a single-node Kubernetes cluster with access to the underlying file system of the appliance. This allows you to build containerized applications on the underlying appliance hardware to address use cases in the following verticals.

Vertical use cases

Now that you understand how Google Cloud Edge Appliance is configured, let’s consider some of the industry use cases where it can provide unique value.

Manufacturing

In the manufacturing industry, quality control and safety is a crucial factor. Businesses need to ensure products are manufactured to the highest standards to remain competitive in their markets, to retain customers, and to keep factory workers safe. To do this, manufacturers need real-time data about the products being manufactured on the production lines, ensuring quality control and gaining a real-time view of where people are on the factory floor.

In manufacturing environments, Google Distributed Cloud Edge Appliance can be used to detect hazards or manufacturing defects in real-time. Figure 1 is a reference architecture for a hazard detection solution running off a Google Distributed Cloud Edge Appliance on a factory floor.

In this architecture, cameras on the factory floor stream live video into the Google Distributed Cloud Edge Appliance. Depending on the number of cameras and appliances, cameras could be split or mapped to different appliances. This architecture makes it possible to initially transfer video data to Google Cloud using an online NFS share. Once in Google Cloud, you can use AutoML to train and build models that can be used as part of the hazard detection solution.

With these trained models, the cameras can stream video data into the appliance using the real-time streaming protocol (RTSP). You can then use AutoML inference to analyze the real-time video streaming data.

For example, in this reference architecture, if an individual comes too close to the fork lift, a function is triggered by the microservices running on the edge appliance that pushes a notification either to a messaging service, or to an enterprise resource planning tool. This alerts factory managers to factory floor hazards in real time so they can take corrective action.

You can also review messages and videos later on for preventive planning purposes, or push streamed videos to Cloud Storage for archive, to use the appliance’s storage space more efficiently.

Data transfers to Google Cloud can be done over Google Cloud Dedicated Interconnect, or VPN between the region and your site. This connectivity also allows you to send the appliance’s control-plane network traffic to the region.

You could also use the reference architecture in figure 2 for a product anomaly detection solution running off a Google Distributed Cloud Edge Appliance on a factory floor or manufacturing line. In this instance, machine learning models are trained to detect anomalies on finished products before final packaging.

Retail

In the retail industry, the Google Distributed Cloud Edge Appliance reference architecture in Figure 2 enables a number of transformative capabilities for retail operations, including:

• contactless checkout
• product scans
• mobile-scan-bag
• cashierless checkout
• unattended retail shops
• visual check-out monitoring

It does all this within a retailer’s facilities with the low latency and high throughput you need to process data locally, so you can obtain actionable insights from your data.

Or, you could use Google Distributed Cloud Edge Appliance at the edge to overhaul store management operations, for example, monitoring store occupancy, queue depth and wait times, detecting slips and falls and out-of-stock items, or monitoring inventory compliance.

Automotive

Advanced Driver Assistance Systems (ADAS) are becoming standard in modern automobiles. To successfully build and roll out continued improvements around ADAS, the automotive industry continues to run extensive tests on ADAS systems that are built into the vehicles they manufacture. Automotive companies can use Google Distributed Cloud Edge Appliance to modernize and transform how they collect data for the ADAS systems they’re developing. For example, test vehicles contain several different sensors that generate data, which can be quickly offloaded to an in-vehicle edge appliance.

Then, within the appliance, you can deploy containerized workloads to transform sensor data, infer videos and images and detect events. This alleviates the need for operators to label all events and allows development teams to quickly gather insights from the tests.

If you want to focus on a subset of information, you can transfer specific data or the entire data payload into Transfer Appliances when vehicles return to the development center. All these systems, i.e., transfer appliances and edge appliances, work in tandem to reduce local system administration and operational costs through a cloud-based control plane.

This approach allows you to deploy, track, monitor and configure services that are running in data centers or at edge locations from the cloud. From the factories, the data can be moved offline or online into Google Cloud where you can use different storage classes and processing capabilities to further process or store the data. You can also deploy newly trained models and business rules back to the edge appliances. In all this, data transfers between the cloud and the appliance are performed using end-to-end encryption, to give you control over your data.

The reference architecture in Figure 3 shows an ADAS implementation where Google Distributed Cloud Edge Appliance is being used to gather, process and transform data at the edge in the automotive industry. It could also be applied to data capture and processing use cases in manned and unmanned vehicles. Notice how the Distributed Edge Appliance extends to the cloud by sending data there, or using other cloud-based services.

We’re just getting started

These are just a few of the use cases where organizations in the manufacturing, retail and automotive industries are using Google Distributed Cloud Edge Appliance with modern and containerized applications that are powered by Google Cloud. If you’re interested in bringing the power of Google Cloud to the edge using Google Distributed Cloud Edge Appliances to transform your business, reach out to us or any of our accredited partners.

Editor’s note: Today we hear from Jon White and Angela Donohoe from BPAY Group. BPAY Group is best known for BPAY, the leading electronic bill payment system in Australia, handling one-third of the market. Learn how BPAY Group is positioning the organization for the future by using APIs to streamline workflows for existing customers and new businesses.

BPAY has been a leader in the bill payment industry in Australia for 22 years and provides a secure, fast, and convenient way to connect individuals, businesses, and banks to help people stay on top of their bills.

One of the reasons that BPAY is the preferred bill payment service for so many Australians is our commitment to human-centered design. We’re continuously talking with customers and looking at ways that we can deliver better experiences, products, and services, such as peer-to-peer payments. During these conversations, we noticed some ways that our processes were causing friction for existing or potential customers.

For example, we traditionally used a batch processing system to handle requests between billing companies and banks. But that could cause headaches for some customers, as an error in even one request could cause the whole batch to be rejected. Plus, many “neobanks” (new types of digital-only banks) wanted to work with real-time transactions instead of batch processes, which take longer to complete.

We realized that APIs had the potential to solve many of the challenges impacting customers while opening the doors for future product and business development. We developed a few customer-facing APIs and tested them in closed betas. This experiment went far better than we expected, and we realized that there was a huge appetite for APIs among our biller customers.

While we had developed many APIs for internal systems, developing APIs that were easy to consume by our customers was a new challenge. We needed to move away from our home-grown API development approach and make our API environment more powerful, versatile, and easier to use. The API experts at The Singularity worked with us to develop a strong API strategy. We decided that we would need to support our new strategy with a scalable API management platform.

After a rigorous search, we landed on the Apigee API Management Platform. Apigee was the only solution that met all our technology and business requirements. With Apigee, we have a solid foundation for APIs that will help us deliver more value for all customers.

Creating a custom development environment

BPAY is a trusted brand in Australia, so it was very important to us that we maintain our reputation for excellent customer experiences. When setting up our developer portal, we started with a closed pilot and used developer feedback to make the portal as convenient and simple to use as possible. The Apigee developer portal has many built-in features to help us customize experiences, and if we run into roadblocks, the Apigee team at Google listens and helps us create the custom experience we want.

The developer portal has already proven to be extremely popular. In its first month live, we registered 104 developers in the sandbox environment and 10 developers in the production environment. That was before we even started marketing our developer portal, so we expect those numbers to rise quickly.

Breaking new ground with APIs

We’ve already released four foundational APIs, with a goal of eventually releasing dozens. Our APIs are helping us create smoother experiences for customers. We mentioned that when processing a batch of payment files, one mistake could cause the entire batch to get rejected. Our APIs now enable businesses to validate all payment information before submitting a batch file, dramatically reducing the chances of errors. They can even use our APIs to automatically generate batch files in the right format for different banks.

While APIs improve service for current customers, they also open the doors for new areas of business. Buy now, pay later (BNPL) services, which enable customers to spread out payments across weeks or months, are already popular in retail spaces. After releasing our first APIs, we connected with two BNPL billing services. These companies use our APIs to validate customers’ bill payment information and then pay the bill in full on behalf of customers. This was a completely new use case for us, one that could not have been implemented without our APIs.

The payment service NoahPay also adopted our APIs to validate payment information and let customers pay bills using funds from their WeChat accounts. This is an exciting new market for us, as it’s one of the first examples of how we can connect to international digital wallets through our new APIs. It’s also a great way to introduce users of WeChat, a messaging app used by more than 1 billion people in China, to the BPAY brand.

Planning for the future of bill pay

We have big plans for APIs in the future, and Apigee helps make these plans a reality. We plan to establish a generous freemium monetization model that will allow customers to make up to 200,000 API calls for free each month with tiered payment plans above that. This will enable us to open the doors for smaller organizations while providing optimal support for larger businesses and banks that might need to make millions of calls. Having powerful end-to-end monetization features built in to Apigee means that we can process monetized transactions with ease.

Built-in reporting functionality will also help us make sure that we’re understanding the market’s need for APIs and always providing our customers with valuable services and support.

Apigee greatly streamlines creating self-service API environments. Even as we grow our business, our internal teams will be able to continue providing excellent customer service without needing extra staff to answer questions, help with integration support, and constantly check API security. APIs are the way of the future, and Apigee prepares us meet the challenges that come along with it.

Editor’s note: Today we hear from the Lowe’s SRE team. They share about how they have been able to increase the number of releases they can support by adopting Google’s Site Reliability Engineering (SRE) framework and leveraging their partnership with Google Cloud. 

At Lowe’s, we’ve made significant progress in our multiyear technology transformation. To modernize our systems and build new capabilities for our customers and associates, we leverage Google’s SRE framework and Google Cloud, which helps us meet their needs faster and more effectively. With these efforts, we’ve been able to go from one release every two weeks to 20+ releases daily—about 20X more releases per month. 

Our SRE transformation didn’t happen overnight, though. Every step along the way brought some challenges. But looking back, we are excited to see how much we have accomplished for our customers as a result. 

Back in 2018, before adopting SRE practices, we were more reactive than proactive, following an “eyes on glass” approach. On-call structures and incident management efficiency were not at optimal levels with too many repetitive and manual tasks, resulting in operational toil. Production concerns were not surfaced into the product roadmap, which resulted in delays in making fixes.

Bootstrapping SRE at Lowe’s

As we moved from on-prem to Google Cloud, we decided to move from a monolithic- to microservices-based architecture. And to better manage this new architecture, we embarked on an SRE journey. 

Then as COVID-19 hit, we really had to accelerate this journey as customers increasingly moved to online ordering and delivery to meet their Total Home Improvement needs. To do so, we followed four key principles that allowed us to meet changing customer needs quickly and release fast and reliably.

1. Automate away toil 
   As we moved from traditional Ops to an SRE ecosystem, our biggest opportunity was reducing toil, so that engineers can spend time on activities that drive business impact and customer outcomes. We think of toil as work that is manual, repetitive,  tactical, devoid of enduring value—but automatable. So, to tackle toil, we focused on automating away the need for manual intervention. As an example, we made sure engineers were not the first point of contact for any alert. Any triage or resolution that an engineer can perform, a machine can be trained to do the same. We used supervised and unsupervised learning techniques to automate our toil. With a long-term goal of “no toil,” our SREs work on identifying and reducing toil to a manageable level across the organization.
2. Engineer alignment through roadmaps
   Our goal is to maximize the engineering velocity of developer teams while keeping products reliable. We want an engagement model where product, SRE and development teams are closely aligned. A key way we’ve been able to create this alignment is by having our SREs embedded into domain and product teams. Each domain has an SRE, who is  involved at the beginning stages of product development to ensure that the domain stakeholders are in alignment with the SRE initiatives. As such, SREs are able to improve the reliability, performance, scalability and launch velocity of the services throughout all phases of the service lifecycle. 
3. Adopt one-touch releases
   Our path to production used to contain many manual steps and validations, slowing the rate at which we released features. Additionally, we used to bulk all our releases together to deploy at once, which increased the risk of failure and created a longer feedback loop from production. To tackle this with an SRE mindset, we created a one-touch release process in which SREs review the product team’s pull requests. When approved, this triggers a DevSecOps pipeline that deploys the approved changes to production securely. This process created a safe, reliable and sustainable continuous delivery pipeline with quick feedback loops. Striking the right balance between speed, innovation and stability, we were able to increase our releases exponentially for the year, taking less than 30 minutes per release to deliver quality code, including various automated quality checks and processes, all in just one click. 
4. Embrace capacity planning
   To ensure our services have enough spare capacity to handle any surge in traffic patterns, our SREs emphasize capacity planning, making recommended capacity changes in the continuous delivery (CD) pipeline. They constantly monitor performance to make sure the service is robust, stable and available. And when there’s a sudden surge beyond the forecasted volume, SREs change the capacity on demand and document changes for the performance and domain teams. 

Capacity planning is especially important for us during peak holiday times such as Black Friday and Cyber Monday (BFCM). We lay out our SRE stability plan three months in advance and surface into the domain team’s product roadmap. This way development teams are able to allocate sufficient engineering time to reliability. We do performance testing to ensure the environment is able to sustain increased load over long periods of time and also handle sudden surges in traffic. We also do region failover testing at a global scale to validate the automatic failover duration of service level agreements  (SLAs), SRE and domain readiness. Additionally, we conduct Black Friday and Cyber Monday-specific destructive testing to validate customer experience, reliability and more.

Google Cloud’s Black Friday and Cyber Monday white-glove service played a key role in ensuring our success in both BFCM 2019 and BFCM 2020. This service included on-site visits from Google’s Customer Reliability Engineering (CRE) team who reviewed Lowe’s web architecture, capacity planning, operations practices for event risks, and presented workshops on topics such as incident response best practices. 

Looking ahead

There is always room for improvement, and at Lowe’s we aim to continuously improve our SRE practices. One thing that has worked well for us, which we plan to continue, has been our road shows, where senior SRE leads present to other SREs and application domain teams on the latest SRE principles and best practices, and to get input in real-time from them. 

Google’s tools and methodology have played an instrumental role in helping reshape our SRE practices and better serve our customers. We look forward to building on the momentum and partnership as we continue our SRE journey at Lowe’s. 

If you want to learn more about how to adopt SRE best practices on Google Cloud, check out our documentation. If you want to learn more about Google SRE, visit our website. Stay tuned for the next blogs with Lowe’s discussing how they trained their engineering talent to adopt SRE practices and tooling, and how they improved MTTR using SRE principles.

Picking the right database for your application is not easy. The choice depends heavily on your use case—transactional processing, analytical processing, in-memory database, and so on—but it also depends on other factors. This post covers the different database options available within Google Cloud across relational (SQL) and non-relational (NoSQL) databases and explains which use cases are best suited for each database option. 

Relational databases 

In relational databases information is stored in tables, rows and columns, which typically works best for structured data. As a result they are used for applications in which the structure of the data does not change often. SQL (Structured Query Language) is used when interacting with most relational databases. They offer ACID consistency mode for the data, which means:

• Atomic: All operations in a transaction succeed or the operation is rolled back.
• Consistent: On the completion of a transaction, the database is structurally sound.
• Isolated: Transactions do not contend with one another. Contentious access to data is moderated by the database so that transactions appear to run sequentially.
• Durable: The results of applying a transaction are permanent, even in the presence of failures.

Because of these properties, relational databases are used in applications that require high accuracy and for transactional queries such as financial and retail transactions. For example: In banking when a customer makes a funds transfer request, you want to make sure the transaction is possible and it actually happens on the most up-to-date account balance, in this case an error or resubmit request is likely fine.

There are three relational database options in Google Cloud: Cloud SQL, Cloud Spanner, and Bare Metal Solution.

Cloud SQL: Provides managed MySQL, PostgreSQL and SQL Server databases on Google Cloud. It reduces maintenance cost and automates database provisioning, storage capacity management, back ups, and out-of-the-box high availability and disaster recovery/failover. For these reasons it is best for general-purpose web frameworks, CRM, ERP, SaaS and e-commerce applications.

Cloud Spanner: Cloud Spanner is an enterprise-grade, globally-distributed, and strongly-consistent database that offers up to 99.999% availability, built specifically to combine the benefits of relational database structure with non-relational horizontal scale. It is a unique database that combines ACID transactions, SQL queries, and relational structure with the scalability that you typically associate with non-relational or NoSQL databases. As a result, Spanner is best used for applications such as gaming, payment solutions, global financial ledgers, retail banking and inventory management that require ability to scale limitlessly with strong-consistency and high-availability. 

Bare Metal Solution: Provides hardware to run specialized workloads with low latency on Google Cloud. This is specifically useful if there is an Oracle database that you want to lift and shift into Google Cloud. This enables data center retirements and paves a path to modernize legacy applications. 

Non-relational databases

Non-relational databases (or NoSQL databases) store compex, unstructured data in a non-tabular form such as documents. Non-relational databases are often used when large quantities of complex and diverse data need to be organized. Unlike relational databases, they perform faster because a query doesn’t have to access several tables to deliver an answer, making them ideal for storing data that may change frequently or for applications that handle many different kinds of data. 

For example, an apparel store might have a database in which shirts have their own document containing all of their information, including size, brand, and color with room for adding more parameters later such as sleeve size, collars, and so on.

Qualities that make NoSQL databases fast:

• Eventual consistency: stores usually exhibit consistency at some later point (e.g., lazily at read time)
• Horizontal scaling, usually using hashed distributions
• Typically, they are optimized for a specific workload pattern (i.e., key-value, graph, wide-column)
• Typically, they don’t support cross shard transactions or flexible isolation modes.

Because of these properties, non-relational databases are used in applications that require large scale, reliability, availability, and frequent data changes.They can easily scale horizontally by adding more servers, unlike some relational databases, which scale vertically by increasing the machine size as the data grows. Although, some relations databases such as Cloud Spanner support scale-out and strict consistency.

Non-relational databases can store a variety of unstructured data such as documents, key-value, graphs, wide columns, and more. Here are your non-relational database options in Google Cloud: 

• Document databases: Store information as documents (in formats such as JSON and XML). For example: Firestore
• Key-value stores: Group associated data in collections with records that are identified with unique keys for easy retrieval. Key-value stores have just enough structure to mirror the value of relational databases while still preserving the benefits of NoSQL. For example: Datastore, Bigtable, Memorystore
• In-memory database: Purpose-built database that relies primarily on memory for data storage. These are designed to attain minimal response time by eliminating the need to access disks. They are ideal for applications that require microsecond response times and can have large spikes in traffic. For example: Memorystore
• Wide-column databases: Use the tabular format but allow a wide variance in how data is named and formatted in each row, even in the same table. They have some basic structure while preserving a lot of flexibility. For example: Bigtable
• Graph databases: Use graph structures to define the relationships between stored data points; useful for identifying patterns in unstructured and semi-structured information. For example: JanusGraph

There are three non-relational databases in Google Cloud:

• Firestore: Is a serverless document database which scales on demand and acts as a backend-as-a-service. It is DBaaS that increases the speed of building applications. It is perfect for all general purpose uses cases such as ecommerce, gaming, IoT and real time dashboards. With Firestore users can interact with and collaborate on live and offline data making it great for real-time application and mobile apps.  
• Cloud Bigtable: Cloud Bigtable is a sparsely populated table that can scale to billions of rows and thousands of columns, enabling you to store terabytes or even petabytes of data. It is ideal for storing very large amounts of single-keyed data with very low latency. It supports high read and write throughput at sub-millisecond latency, and it is an ideal data source for MapReduce operations. It also supports the open-source HBase API standard to easily integrate with the Apache ecosystem including HBase, Beam, Hadoop and Spark along with Google Cloud ecosystem.
• Memorystore: Memorystore is a fully managed in-memory data store service for Redis and Memcached at Google Cloud. It is best for in-memory and transient data stores and automates the complex tasks of provisioning, replication, failover, and patching so you can spend more time coding. Because it offers extremely low latency and high performance, Memorystore is great for web and mobile, gaming, leaderboard, social, chat, and news feed applications.

Conclusion

Choosing a relational or a non-relational database largely depends on the use case. Broadly, if your application requires ACID transactions and your data structure is not going to change much, select a relational database. 

In Google Cloud use Cloud SQL for any general-purpose SQL database and Cloud Spanner for large-scale globally scalable, strongly consistent use cases. In general, if your data structure may change later and if scale and availability is a bigger requirement than consistency then a non-relational database is a preferable choice.  Google Cloud offers Firestore, Memorystore, and Cloud Bigtable to support a variety of use cases across the document, key-value, and wide column database spectrum.

For more comparison resources on each database check out the overview. For more hands-on experience with Bigtable, check out our on-demand training here and learn about migrating databases to managed services check out this whitepaper.  

https://youtube.com/watch?v=2TZXSnCTd7E%3Fenablejsapi%3D1%26

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.