This week, the Vera C. Rubin Observatory is launching the first preview of its new Rubin Science Platform (RSP) for an initial cohort of astronomers. The observatory, which is located in Chile but managed by the U.S. National Science Foundation’s NOIRLab in Tucson, AZ and SLAC in California, is jointly funded by the NSF and the U.S. Department of Energy. The platform provides an easy-to-use interface to store and analyze the massive datasets of the Legacy Survey of Space and Time (LSST), which will survey a third of the sky each night for ten years, detecting billions of stars and galaxies, and millions of supernovae, variable stars, and small bodies in our Solar System.

The LSST datasets are unprecedented in size and complexity, and will be far too large for scientists to download to their personal computers for analysis. Instead, scientists will use the RSP to process, query, visualize, and analyze the LSST data archives through a mixture of web portal, notebook, and other virtual data analysis services. An initial launch with simulated data, called Data Preview 0, builds on the Rubin Observatory’s three-year partnership with Google to develop an Interim Data Facility (IDF) on Google Cloud to prototype hosting of the massive LSST dataset. This agreement marks the first time a cloud-based data facility has been used for an astronomy application of this magnitude.

Bringing the stars to the cloud

For Data Preview 0, the IDF leverages Cloud Storage, Google Kubernetes Engine (GKE), and Compute Engine to provide the Rubin Observatory user community access to simulated LSST data in an early version of the RSP. The simulated data were developed over several years by the LSST Dark Energy Science Collaboration to imitate five years of an LSST-like survey over 300 square degrees of the sky (about 1,500 times the area of the moon). The resulting images are very realistic: they have the same instrumental characteristics, such as pixel size and sensitivity to photons, that are expected from the Rubin Observatory’s LSST Camera, and they were processed with an early version of the LSST Science Pipelines that will eventually be used to process LSST data. “This will be the first time that these workloads have ever been hosted in a cloud environment. Researchers will have an opportunity to explore an early version of this platform,” says Ranpal Gill, senior manager and head of communications at the Rubin Observatory.

Broadening access for more researchers

Over 200 scientists and students with Rubin Observatory data rights were selected to participate in Data Preview 0 from a pool of applicants that represents a wide range of demographic criteria, regions, and experience level. Participants will be supported with resources such as tutorials, seminars, communication channels, and networking opportunities—and they will be free to pursue their own science at their own pace using the data in the RSP. 

“The revolutionary nature of the future LSST dataset requires a commensurately innovative system for data access and analysis paired with robust support for scientists,” says Melissa Graham, lead community scientist for the Rubin Observatory and research scientist in the astronomy department at the University of Washington. “I’m personally excited to enhance my own skills by using the RSP’s tools for big data analysis, while also helping others to learn and to pursue their LSST-related science goals during Data Preview 0.” 

At the same time, the fact that the RSP is hosted in the cloud provides researchers at smaller institutions access to state-of-the-art astronomy infrastructure that is comparable to that of the largest national research centers.

The launch benefits the observatory too: the development team can learn what researchers are interested in while also testing and debugging the platform. Graham says that “the platform is still in active development so researchers using it will be able to follow along in the progress, and provide feedback on ways that we can optimize the development of the tools.”

Next steps

The LSST aims to begin the ten-year survey in 2023-24 and expects it to include 500 petabytes of data. Through the cloud, Google aims to help make this extraordinary project scalable and accessible to researchers everywhere. To learn more about Data Preview 0, watch this video.

Want to ramp up your own research in the cloud? We offer research credits to academics using Google Cloud for qualifying projects in eligible countries. You can find our application form on Google Cloud’s website or contact our sales team.

Compute Engine is a customizable compute service that lets you create and run virtual machines on Google’s infrastructure. You can create a Virtual Machine (VM) that fits your needs. Predefined machine types are pre-built and ready-to-go configurations of VMs with specific amounts of vCPU and memory to start running apps quickly. With Custom Machine Types, you can create virtual machines with the optimal amount of CPU and memory for your workloads. This allows you to tailor your infrastructure to your workload. If requirements change, using the stop/start feature you can move your workload to a smaller or larger Custom Machine Type instance, or to a predefined configuration.

Machine types

In Compute Engine, machine types are grouped and curated by families for different workloads. You can choose from general-purpose, memory-optimized, compute-optimized and accelerator-optimized families. 

• General-purpose machines are used for Day-to-day computing at a lower cost and for balanced price/performance across a wide range of VM shapes. The use cases that best fit here are web serving, app serving, back office applications, databases, cache, media-streaming, microservices, virtual desktops, development environments.
• Memory-Optimized machine are recommended for ultra high-memory workloads such as in-memory analytics and large in-memory databases such as SAP HANA 
• Compute-Optimized machines are recommended for ultra high performance workloads such as High Performance Computing (HPC), Electronic Design Automation (EDA), gaming, video transcoding, single-threaded applications.
• Accelerator-Optimized machines are optimized for high performance computing workloads such as Machine learning (ML), Massive parallelized computations and High Performance Computing (HPC)

How does it work?

You can create a VM instance using a boot disk image, a boot disk snapshot, or a container image. The image can be a public operating system (OS) image or a custom one. Depending on where your users are you can define the zone you want the virtual machine to be created in. By default all traffic from the internet is blocked by the firewall and you can enable the HTTP(s) traffic if needed. 

Use snapshot schedules (hourly, daily, or weekly) as a best practice to back up your Compute Engine workloads. Compute Engine offers live migration by default to keep your virtual machine instances running even when software or hardware update occurs. Your running instances are migrated to another host in the same zone instead of requiring your VMs to be rebooted. 

Availability

For High Availability (HA) Compute Engine offers automatic failover to other regions or zones in event of a failure. Managed instance groups (MIGs) help keep the instances running by automatically replicating instances from a predefined image. They also provide application based autohealing health checks. If an application is not responding on a VM, the auto healer automatically recreates that VM for you. Regional MIGs let you spread app load across multiple zones. This replication protects against zonal failures. MIGs work with load balancing services to distribute traffic across all of the instances in the group. 

Compute Engine offers autoscaling to automatically add or remove VM instances from a managed instance group based on increases or decreases in load. Autoscaling lets your apps gracefully handle increases in traffic, and it reduces cost when the need for resources is lower. You define the autoscaling policy for automatic scaling based on the measured load, CPU utilization, requests per second or other metrics.

Active Assist’s new feature, predictive autoscaling, helps improve response times for your applications–When you enable predictive autoscaling, Compute Engine forecasts future load based on your Managed Instance Group’s (MIG) history and scales it out in advance of predicted load, so that new instances are ready to serve when the load arrives. Without predictive autoscaling, an autoscaler can only scale a group reactively, based on observed changes in load in real time. With predictive autoscaling enabled, the autoscaler works with real-time data as well as with historical data to cover both the current and forecasted load. That makes predictive autoscaling ideal for those apps with long initialization times and whose workloads vary predictably with daily or weekly cycles. For more information, see How predictive autoscaling works or check if predictive autoscaling is suitable for your workload, and to learn more about other intelligent features, check out Active Assist.

Pricing

You pay for what you use. But you can save cost by taking advantage of some discounts! Sustained use saving are automatic discounts applied for running instances for a significant portion of the month. If you know your usage upfront, you can take advantage of committed use discounts which can lead up to significant savings without any upfront cost. And by using short lived preemptive instances you can save up to 80%, they are great for batch jobs and fault tolerant workloads. You can also optimize resource utilization with automatic recommendations. For example if you are using a bigger instance for a workload that can run on a smaller instance you can save costs applying these recommendations.

Security

Compute Engine provides you default hardware security. Using Identity and Access Management (IAM) you just have to ensure that proper permissions are given to control access to your VM resources. All the other basic security principles apply, if the resources are not related and don’t require network communication amongst themselves, consider hosting them on different VPC networks. By default, users in a project can create persistent disks or copy images using any of the public images or any images that project members can access through IAM roles. You may want to restrict your project members so that they can create boot disks only from images that contain approved software that meet your policy or security requirements. You can define an organization policy that only allows Compute Engine VMs to be created from approved images. This can be done by using the Trusted Images Policy to enforce images that can be used in your organization. 

By default all VM families are Shielded VMs. Shielded VMs are virtual machine instances that are hardened with a set of easily configurable security features to ensure that when your VM boots, it’s running a verified bootloader and kernel — is the default for everyone using Compute Engine, at no additional charge. For more details on Shielded VMs refer to the documentation here.

For additional security, you also have the option to use Confidential VM to encrypt your data in use, while it’s being processed in Compute Engine. For more details on Confidential VM refer to the documentation here.

Use cases

There are many use cases Compute Engine can serve in addition to running websites and databases. You can also migrate your existing systems onto Google Cloud, with Migrate for Compute Engine, enabling you to run stateful workloads in the cloud within minutes rather than days or weeks. Windows, Oracle or VMware applications have solution sets enabling a smooth transition to Google Cloud. To run windows applications either bring your own license leveraging Sole-tenant nodes or using the included licenced images. 

Conclusion

Whatever your application use case may be, from legacy enterprise applications to digital native applications, Compute Engine’s families will fit it. For a more in-depth look into Compute Engine check out the documentation. 

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev

When banks and other financial institutions make their products available via non-banks, we call that Embedded Finance. Think, for example, of a retailer offering financing for people buying goods; or a budgeting app gathering bank transactions to help you categorize and understand your spending patterns. The finance – the money management – has been embedded into the retailer or the budgeting experience.

This last example falls under the Data Exchange category of Embedded Finance , and it’s often also called Open Banking. In many countries this active data exchange is actually mandated by law, with specific Open Banking standards encoded into law for all financial institutions. As of this blog post, there are approximately 37 countries with such laws already in place or currently being worked on.

In an Embedded Finance scheme there are three main players: The financial institution offering their products and services, a third party provider (TPP) making use of those products or services, and an end user who is a customer of the financial institution and is interacting with the TPP.

The most important ingredient for these schemes is consumer trust. If a consumer does not trust that their data will be safely shared with a third party, and that they can easily understand the terms they have agreed to and stop sharing their data whenever they want to, consumers will simply not participate. When a consumer agrees for their data to be shared, this is called consent.

What components do I need for a successful Embedded Finance solution?

Banks’ products and services are made available to other parties via APIs, thus a fundamental component is an API Management platform. An API Management Platform lets you design, secure, analyze, and scale APIs anywhere with visibility and control.

Also, we need a component to manage consents given by end users. Consent Management refers not only to the capability of checking whether a certain action by a third party (eg: access an end user’s account balance) is valid according to the consent granted by the end user to that TPP,  but also the availability of tools for end users to check existing consents, modify them or revoke them, as well as tools for financial institutions to perform similar actions on behalf of their customers.

Finally, we need an Identity Platform capable of authenticating an end user and establishing their identity.

A Google Cloud based embedded finance solution uses Apigee as the API management platform, Identity Platform or your own identity platform of choice, and a consent management service (CMS).  Google Cloud has partnered with two leading providers in the consent management space: Clarip and CloudEntity.

How do these components work together?

The main roles and responsibilities of each of the three components are quite clear, but there is a certain degree of flexibility on how to orchestrate the interactions between the components, and which component performs which function when there is an overlap of functionality between them.

Google Cloud has tested the following interaction patterns:

• Interaction Pattern #1: Apigee as the main orchestrator

In this interaction pattern, Apigee interacts with both the consent management service and the identity platform, coordinating end user authentication with the identity platform, and authorization and consent with the consent management service. Apigee acts as the “broker” between users, their identity, and their consent.

• Interaction Pattern #2: The consent management service abstracts the identity platform.

In this case, the consent management service manages all the interactions with the identity management platform, and the identity platform acts as the “broker”. Apigee only interacts with the consent service.

The three components share a common pool of client apps and end users. One of the components is typically designated as the source of truth for these entities, while the other components that need to know about these entities synchronize that information from the source of truth. 

The identity platform is typically the source of truth for end user identities and the consent management component will synchronize that information by pulling it from the identity platform. 

The consent management service is the source of truth for user opt-in and current consent status. Apigee will either check with the service the validity of a consent every time a request comes through, or to make it more efficient, it will validate the main attributes of the consent based on information stored in Apigee. To maintain consistency, the consent management service should tell Apigee to invalidate any tokens associated with a consent when it is revoked via other means (for example, if an end user revoked consent from a consent dashboard).

For client apps, either Apigee or the consent management component can act as the source of truth, with the other component synchronizing the information. In the context of Open Banking in particular, standards often mandate that financial institutions support Dynamic Client Registration. This means having endpoints that new clients can use to register themselves as clients for the Open Banking APIs. In this case, the synchronization of clients can easily be incorporated into Dynamic Client Registration.

Let’s look now into more detail about how the three components interact. There are two user journeys that are particularly relevant:

Authentication/Authorization/Consent

This is the beginning of the typical consumer client journey, when an end user starts using a third party app that requires access to that user’s data. The third party app needs to authenticate itself, and request authorization to access the user’s data. This will trigger the authentication of the end user and consent granting: Consent granting being the process of, the user authorizing the operations the app will execute on their behalf, and establishing the list of accounts on which these operations will be performed. The authorization is represented by an access token, which the app will need to include in all subsequent requests.

This is quite a complex journey with many steps involved in order to make this as secure as possible. To make it easier to understand, we will simplify some of the steps (apologies in advance to you OIDC/FAPI experts out there!).

At a high level, these are the steps involved in this user journey:

1. End user starts using an app
2. The app requests permission to access user’s data
3. The identity platform (IdP) authenticates the end user
4. The consent management service obtains consent from end user
5. An access token is issued for the app, encapsulating the authorized permissions, accounts, etc. A refresh token is also issued – the access token normally has a short life (measured in minutes), but the app can use the refresh token, which typically lasts much longer (days or months), in order to obtain a new access token without the need of going through the end user authentication/authorization/consent process again.

The way these steps are orchestrated varies depending on the interaction pattern among the components.

Pattern #1: Apigee orchestrates all interactions
When the client requests permission, Apigee will record the request and then redirect the client app to the IdP login page. Once the end user logs in, the IdP will return an authorization code to the callback URL that Apigee registered in the IdP configuration.

Apigee will then redirect the client app to the Consent Management service, so that the consent form is displayed. Once the user grants consent, selects the desired permissions, and selects the accounts it will apply to, the Consent Management Service will update Apigee with the status of the consent, via a callback Apigee registered in the consent configuration.

Now that the end user has authenticated and granted consent, Apigee can now issue an authorization code, and associate all the necessary information, for instance, details about the consent, the authorization code issued by the Identity Platform (IdP), the identification of the end user in the IdP, etc. Apigee will return this authorization code to the client app via a callback endpoint that the app provided when registering with Apigee.

The app will then exchange the authorization code for an access token. Apigee will obtain an access token from the IdP, abd update the status of the consent before issuing an access token to the client.

Pattern #2: The Consent Management Service is the center of the Architecture
In Pattern #2, Apigee will only communicate with the Consent Management Service, which acts as an authorization server for the app and will also coordinate all interaction with the IdP.

When the client requests permission, Apigee will forward the request to the CMS. The CMS will record the request and then redirect the client app to the IdP login page. Once the end user logs in, the IdP will return an authorization code to the callback the CMS registered in the IdP configuration.

The CMS then displays the consent form. Once the user approves the consent and selects the desired permissions and accounts it will apply to, the Consent Management Service will return an authorization code to Apigee, via a callback Apigee registered in the CMS configuration.

From now on, the flow is the same as explained above. Apigee will issue an authorization code, associate all the necessary information and will return this authorization code to the client app via a callback endpoint. The app will then exchange the authorization code for an access token. Apigee will obtain an access token from the CMS, before issuing an access token to the client.

Let’s now look at the other relevant user journey:

Access shared data 

Once the third party app has been authorized, it will then start requesting that end user’s data. For instance, it may request the list of accounts or the balance of a particular account. 

When that request goes through, a series of checks need to be performed: Is this third party app still authorized? Has it been granted the right permission for the operation being requested? Is the account involved included in the list of accounts authorized by the end user? Apigee can perform all these checks by inspecting the access token presented by the third party app. 

There is, however, one other check which cannot be done simply by inspecting the token. Imagine for a moment that in-between the user granting consent and the app requesting the user’s data, the end user decided to revoke that consent. This could be done by other means than the app itself, for instance by checking all consents granted to all apps in a consent dashboard accessible via other channels such as an “Internet Banking” site. When the app requests the user data, the token is still valid, however the consent has already been revoked. Therefore, in this user journey, Apigee will check with the Consent Management service to check if the consent is still valid. This adds a bit of overhead, which can be minimized by other means. For instance, one popular optimization is to make sure that the Consent Management service asks Apigee to revoke any access or refresh tokens associated with a given consent when that consent is revoked via other channels. If that guardrail is in place, then Apigee can determine if a request should be allowed simply by validating the access token and using the consent information stored alongside the token.

The following diagram summarizes the interactions required between Apigee and the Consent Management component. The interaction is essentially the same in both interaction patterns.

What’s next?

In this article we have described how Apigee, a Consent Management service, and an Identity Platform are the fundamental ingredients of a successful and secure Embedded Finance solution. We have also looked at how these components interact with each other to keep user consent consistent and available. Both interaction patterns described are valid and have been proven to work. So which one may be right for you? The answer ultimately depends on your particular requirements, the choice of consent management service, and your choice of identity platform. As a rule of thumb, if you have multiple application experiences, identity systems, and sources of consent, an Apigee-centric architecture is a great solution. If you are looking at a Consent Management service in the context of an Open Banking implementation mandated by a centralized authority (eg: Open Banking Brazil, Consumer Data Standards in Australia), CloudEntity is a good choice for consent management, as their product is also Certified Financial-grade API (FAPI) OpenID Provider, with profiles for the various Open Banking standards. CloudEntity integrates with Apigee using Pattern #2. 

For more details on both Clarip and CloudEntity’s offerings and how their particular solution works with GCP please check:

• Clarip: Universal Consent Management Platform – GDPR
• CloudEntity: Certified OpenBanking  & Consent (UK, Brazil and Australia) at scale

For more details on Apigee API Management, see https://cloud.google.com/apigee

You can also learn about our solutions for retail banking or contact sales.

Editor’s note: Today’s post is written by Hatim M, Chief Commercial Officer at BungkusIT. The on-demand delivery service delivers packages within the hour for one million customers across Malaysia and uses Google Maps Platform to create a seamless end-to-end delivery experience for its customers.

Imagine it’s the end of a long day at work, and you’re heading home for dinner as you look forward to a restful evening. Then just as you arrive at the doorstep, your phone buzzes with a message from a family member, asking if you could grab a carton of milk from the supermarket.

With moments like these in mind, we founded BungkusIT to alleviate the stress of everyday chores and give people a break from having to run seemingly mundane errands that can often be time-consuming. With so much to do and so few hours in a day, we want to make life simpler by providing our customers with an on-demand delivery service.

As a social enterprise, our mission is to create jobs and increase earning potential for BungkusIT roadies (those who run deliveries), while also increasing online visibility for  local small and medium enterprises through an e-commerce platform. As such, we don’t charge merchants a fee for being featured on our app. 

Our roadies help connect users with products and services that they need. Whether it’s picking up the keys you forgot, or buying a birthday cake for a loved one, our on-demand roadies can help accomplish your task quickly and efficiently. In fact, we’ve helped many seniors—a large and often underserved population—to run errands and provide last mile support.

Providing reliable service, 24/7

From the day we launched, our promise has always been to complete every task given to us within one hour. Google Maps Platform empowers us to fulfil this promise. We understand the importance of selecting a reliable mapping service from the get-go. After exploring different map products, we made the decision to go with Google Maps Platform due to its detailed and accurate location mapping system. With the support of our partner Searce, we’ve been able to quickly integrate all the relevant products on our platform and optimize our API calls.

On a day-to-day basis, our roadies rely on the Places API to navigate their journeys. It pinpoints the exact location set by customers so that our roadies can be sure they arrive at the correct destination. With so many small roads and alleys in Malaysia, it can be challenging to find the exact place based on an address. Dropping a pin on a specific location removes any confusion.

Because we price our services based on distance, it’s business critical to determine the exact distance of a location and the most efficient route to reach it. By automating that process with the Distance Matrix API, we can guarantee that customers don’t get overcharged, while making sure that we’re adequately compensated for our services.

Improving the user experience

At the end of the day, we want to make people’s lives easier. That’s why having an app that is intuitive is key. More than that, we want customers to be able to create a request as quickly as possible. By automatically completing the location a customer is typing after they’ve keyed in the first few characters, Place Autocomplete does that job for us. 

As a fairly young company, we’ve got a long way to go. I believe that we’ll continue evolving alongside Google Maps Platform and exploring the new features it has to offer. We’re already talking to global brands to expand our services and I’m confident that with Google Maps Platform as our maps partner, we will go a long way.

For more information on Google Maps Platform, visit our website.

During uncertain times, ecosystem partnerships that leverage APIs have proven to help companies scale and address gaps in their businesses. Apigee customers, like CHAMP Cargosystems, have pursued API-first ecosystem models to enter adjacent markets, create new customer interaction models, and rapidly grow their brand reach and partner ecosystems. As a result of building their API ecosystem, they are transacting 300 million electronic exchanges and 20 million shipments per year.

CHAMP selected Apigee to provide an API platform and developer self-service portal option to all of its SaaS customers. Google Cloud’s Apigee API management platform and portal allows CHAMP and its customers to quickly connect to a variety of backend systems, including in-house, third party apps, marketplace portals and more– thereby accelerating their digitization strategies and opening up new markets through an API ecosystem.

Join this webcast and hear from this leading Enterprise company on how to:

• Identify new revenue sources and markets using an API management platform
• Improve time to market while still complying with all industry requirements
• How to create a proof of concept to grow API adoption throughout your organization
• Align internal business leaders and partners to see the importance of an API-first platform vision