Datashare for Financial Services: Securing the Publishers and Consumers’ Access to Market Data
8587
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
Access to the cloud has advanced the distribution and consumption of financial information on a global scale. In parallel, the global financial data landscape has been transformed by an influx of alternative data sources, including social media, meteorological data, satellite imagery, and other data. Exchanges and market data providers now find they need to include these new datasets to enrich their products and compete, which has meant they now must consider cloud-based models to keep up with the demands of their customers who expect easy, quick, flexible and cost-efficient ways to consume market data.
To address these needs, today we’re announcing the general availability of Datashare for financial services, a new Google Cloud solution that brings together the entire capital markets ecosystem—data publishers, and data consumers—to exchange market data securely and easily.
Datashare helps organize third-party financial information, making it accessible and useful to market data publishers and data consumers. We open-sourced the entire Datashare solution so market data publishers can now onboard their licensed datasets to Google Cloud securely, quickly and easily, while data consumers can consume that data as a service in tools of their preference, such as BigQuery.
Three ways to distribute and consume your data
Batch data delivery
Datashare provides a batch data delivery mechanism for data publishers to deliver their reference data, historical tick data, alternative market data sources and more via BigQuery, reducing the administrative burden on data consumers to extract insights from data.
Real-time data streaming delivery
By using this event-based data delivery channel for rapidly changing instrument prices, tick data, orders, news and others via Pub/Sub, data consumers can reliably process individual messages or rewind to a point in time to replay a prior market scenario and test model changes.
Monetizing licensed datasets
Market data publishers can onboard their licensed datasets to Google Cloud and make them available via a one-stop-shop on Google Cloud Marketplace, enabling a new sales channel to expand market reach.
Reference architecture
Check out the diagram below to see how you can share your batch and real-time data directly to your Google Cloud customers with BigQuery and Pub/Sub.
As you can see in the above reference architecture, both publishers and consumers can derive several benefits from the solution:
Benefits for data publishers
- You no longer have to maintain your own delivery and licensing infrastructure.
- You can easily package and deliver granular data products and experiments with SQL.
- You can have a solution that scales with your business as data volumes and number of customers grow.
Benefits for data consumers
- Your data is ready for analysis and machine learning (ML)— you no longer have to maintain extract, transform, and load (ETL) pipelines to load files and transform data.
- You can avoid the expense and burden of maintaining multiple copies of large data files.
- You can be more targeted with consumption of data using BigQuery queries, improving performance, and compliance, and reducing cost.
Accessing the datasets
Google Cloud has been working with multiple industry firms on innovating in the market data space. By using Datashare for publishing, data publishers can make their entire datasets available on Google Cloud. Early adopters of Datashare include firms such as OneTick and Accern. OneTick’s datasets include reference and historical futures data (that can be accessed in our console with your login). Accern’s datasets include alternative data such as market sentiment and credit analysis data (that can be accessed in our console with your login).
To make it more helpful, we partnered with Accern to create a hypothetical scenario to describe the data acquisition and analytics process step-by-step.
Accern use case
As a sustainability analyst, you require an economic, social and governance (ESG) dataset to determine which sector is the most widely covered ESG sector by analysts, and to also identify the sector with the lowest ESG sentiment score. Now, you can discover and acquire an ESG dataset in Google Cloud.
Step 1. Navigate to the Financial Services solutions page in the Google Cloud console:
Step 2. Click a dataset, for example Accern AI-Generated ESG Insights, then review the overview details, plans and pricing, documentation and support information. To view the available pricing tiers, click ‘View All Plans’. Once you’ve decided on a tier that you would like to subscribe to, click ‘Select’, choose a billing account and review and accept the terms of service to complete the subscription. Once the steps are complete, click ‘Subscribe’ at the bottom. An overlay window will appear, click ‘Register with Accern’ to activate and complete the subscription.
Step 3. Once activation is complete, you’ll be directed to the Datashare ‘My Products’ screen. Voila! You are now subscribed to Accern’s ESG Scores dataset and can access it in your Google Cloud instance using BigQuery. To access the data, click the hour glass icon on the corresponding ‘My Products’ record that you just purchased. An overlay will present you with the details on the dataset and/or table. Click the ‘Navigate to Table’ button to navigate through to the BigQuery console.
Step 4. Now that you have access and are in the BigQuery console, it’s time to generate data insights.
For this example, we’ve eliminated the company identifying information that is included as part of the subscription and aggregated company ESG in a view where each row represents a day, an industry sector, a specific identified ‘ESG Issues’ (event_group and event) and the respective ‘ESG Sentiment’ per issue.
For example, row 1 indicates that within the ‘Healthcare’ sector, there was a ‘Social – Civil Society’ issue identified and it had a negative ESG sentiment score of -15.35.
Step 5. Generate a report by exporting it to Data Studio to build visualizations and conduct additional analysis on the ESG data.
Select ‘Export’ and ‘Explore with Data Studio’.
Step 6. Build a simple/basic report.
Now that the ESG data appears in Data Studio, you can start by building a simple chart to help you understand which industry sectors have the highest volume of discussions around ESG and the overall ESG Sentiment per industry sector.
To build the chart:
- Select the chart type ‘Table’.
- Include
Entity_Sectoras your dimension to aggregate results by ‘Industry Sector.’ - Include
Signal_IDas a measure to count the number of ESG passages identified per ‘Industry Sector.’ - Include
AVG(Event_Sentiment)as a measure to display the overall ESG Sentiment per ‘Industry Sector’ across ESG Issues.
You can see sectors that are most discussed when it comes to ESG related topics and their corresponding ‘ESG Sentiment’ scores.
Step 7. Build your final report in Data Studio.
As a next step you can further drill into the data to understand ESG data specific to each ‘Industry Sector’ and identify positive and negative ESG practices.
Accern has built a more complex sample dashboard and made it available publicly here. You can interact with this report and play around with the data. The dashboard can help to identify material ESG insights for each sector to inform your investment and risk processes. If you have additional questions, you can reach out to Accern directly.
Discovering, accessing and analyzing licensed datasets is quick and easy. Stay tuned for more updates on new licensed datasets.
Publishing your data via Datashare
If you are a publisher of market data, alternative, or exotic data, you can use Datashare to get it published on Google Cloud Marketplace.
Start by joining the Partner Advantage program by registering for the Partner Advantage Portal and applying for the Partner Advantage Build Model engagement. Visit our getting started guide for information to get started on publishing licensed datasets in the Marketplace. Stay tuned for a future blog post about using Datashare to publish datasets in the Marketplace.
More solutions for capital markets
Check out other Google Cloud solutions for capital markets.
Forrester Surveyed Indian Retailers About Digital Transformation. Here’s What They Found
DOWNLOAD WHITEPAPER3998
Of your peers have already downloaded this article
12:30 Minutes
The most insightful time you'll spend today!
As today’s empowered consumers demand more of the retail experience than ever before, leading retailers and brands in India are investing to rethink and reinvent in their customers’ cross-touchpoint experiences.
Our survey results demonstrate that retail decision makers understand that better customer experience can yield financial benefits, including faster revenue growth, and elevate the reach of influence and brand in the market.
Forty percent or more of retail executives are prioritizing revenue growth, improvement of customer experience (CX), and simplification of operations as the top priorities in their business agendas over the next year.
The survey also covers:
- Key Drivers For Retail Organizations To Migrate Application To Public Cloud
- Cloud Investments In The Retail Industry
- The Three Dimensions That The Industry’s Cloud Challenges Are Taking
- The Top Agendas Retailers Want to Accomplish with the Public Cloud
Download Forrester’s Retail Report Now.
Telus Ensures Workers’ Safety Using Edge and 5G
3197
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Editor’s note: In February 2021, Google Cloud and TELUS announced a 10-year strategic alliance to drive innovation of new services and solutions across data analytics, machine learning, and go-to-market strategies that support digital transformation within key industries, including communications technology, healthcare, agriculture, and connected home. By December 2021, TELUS had completed a pilot for a use case that leveraged Google Cloud AI and Machine Learning solutions and Telco Edge Anthos to increase safety in the workplace and save lives in manufacturing facilities. The use case leverages Multi-Access Edge Computing (MEC) to move the processing and management of traffic from a centralized cloud to the edge of TELUS’ 5G network, making it possible to deploy applications and process content closer to its customers, and thus yielding several benefits including better performance, security, and customization. Today, we invite Samer Geissah, Head of Technology Strategy and Architecture at TELUS, to share how the company is delivering on its promise to use this technology to drive meaningful change, starting with workers’ well-being.
Whenever a new technology buzzword comes along I think: what problems does this solve, and for whom is this going to make a real difference? That’s because at TELUS, we see innovation as a means to act on our social purpose to drive meaningful change, from modernizing healthcare and making our food supply more sustainable, to reducing our environmental footprint and connecting Canadians in need. Multi-Access Edge Computing (MEC) is a buzzword that offers an opportunity to do just this. That’s why we want to leverage cloud capabilities and optimize our network’s edge computing potential, tapping into our award-winning high-speed 5G connectivity to help solve some of industry’s most complex challenges.
The reason why this presents such a great opportunity is that companies across industries still rely on maintenance-heavy on-premises systems to manage core computing tasks. But, with cloud capabilities delivered at the edge of our 5G network, we open a new world of possibilities for them. For example, manufacturers who currently rely on IoT-enabled equipment in their facilities can deliver new experiences by running advanced AI-based visual inspections directly from 5G-enabled devices–all without the need for local processing power or extra on-site space. In fact, it’s this example that inspired our new use case, where our Connected Worker Safety solution can be applied across a range of business verticals to help improve safety, prevent injury, and save lives, demonstrating how the perfect combination of skilled people and digital technology can make the world a safer place.
Empowering intelligent decision making at the edge
Be it a farm, manufacturing facility, hospital, or a factory floor, workers should be able to work in environments where their health and safety are held as the highest priority. But how can employers ensure that their remote, frontline, and in-office employees are safe and healthy at all times? We’ve found the answer by combining Google Cloud AI/ML capabilities and Anthos as a platform for delivering workloads, with our network’s infrastructure.
Together with Google Cloud, we have been leveraging solutions with the power of MEC and 5G to develop a workers’ safety application in our Edmonton Data Center that enables on-premise video analytics cameras to screen manufacturing facilities and ensure compliance with safety requirements to operate heavy-duty machinery. The CCTV (closed-circuit television) cameras we used are cost-effective and easier to deploy than RTLS (real time location services) solutions that detect worker proximity and avoid collisions. This is a positive, proactive step to steadily improve workplace safety. For example, if a worker’s hand is close to a drill, that drill press will not bore holes in any surface until the video analytics camera detects that the worker’s hand has been removed from the safety zone area.
A few milliseconds could make all the difference when you are operating heavy equipment without guards in place. So, to power the solution’s predetermined actions with immediate response times, we worked with Accenture and hosted the application on an Anthos bare metal Google Cloud environment running on our TELUS multi-edge access computing.
Because all the conditions in our model are programmable, this solution can be replicated at scale across a variety of practical scenarios other than factory floors. The actions in response to the analysis are also programmable, which means companies can use this technology to look at workers’ conditions and decide the best course of action to educate, assist, and protect them. All this is done through a single pane of glass ecosystem, making it easy to customize this solution to meet various business needs.
Meanwhile, leveraging our existing global networks to process data and compute cycles at the edge eliminates the need to transport data to a central location for real-time computation. This means that we can offer this solution to partners while optimizing latency and lowering costs.
Powering blink-of-an-eye communication with Anthos
To put the importance of lowering speed into perspective, consider that the average latency of blinking your eye is about 300 milliseconds. From a safety point of view, preventative processes need to be much faster than that. For this use case, our machine learning models running on edge are currently processing data at a tenth of the time it takes for you to blink your eyes, and we’re aiming to lower that latency further to help build even safer systems.
Our plan is to deploy Anthos clusters on bare metal to our customers across Canada to take advantage of our existing enterprise infrastructure, making it possible for us to run our solution closer to partners and eventually enable just one millisecond of latency.
At that point, we’ll be able to power new use cases that require near real-time feedback, leaving absolutely no room for error. This could include remote surgery, platooning of fleets on autonomous vehicles, and many other cellular vehicle-to-everything (V2X) solutions that require high-speed communication for platform operators to manage remote edge fleets in far-away places.
Improving workers’ safety while enabling new sources of revenue
Although edge computing and 5G have been around for a while, we believe that use cases like this are only just starting to demonstrate the incredible speed of change and high potential that these models provide. The next step for us is to develop our workers’ safety solution and get it to market, making TELUS an early adopter of new 5G solutions at the edge that can help our business and industry partners make workplaces safer.
It’s a great win to be able to combine efforts with Google Cloud and reduce latency in a context where timing can impact and save lives, and I’m confident that workers’ safety is just the beginning of a series of industry challenges that we’ll address together.
HarbourBridge Schema Assistant Allows Quick, Bulk Migration to Cloud Spanner
5481
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Today we’re announcing the HarbourBridge Schema Assistant, which provides a guided schema-design workflow for migrating from MySQL or PostgreSQL to Spanner. HarbourBridge imports dump files (from mysqldump or pg_dump) or directly connects to your source database, and converts the source database schema to an equivalent Spanner schema. The new Schema Assistant capability displays the source schema and Spanner schema side-by-side, highlights errors and walks you through a series of steps to validate and optimize your Spanner schema. It also produces a browsable assessment report with an overall migration-fitness score for Spanner, a table-by-table detailed analysis of type mappings and a list of features used in the source database that aren’t supported by Spanner. It supports editing of table and column names, column types, primary keys and constraints, as well as dropping of tables, columns, foreign keys and secondary indexes.
The new Schema Assistant complements HarbourBridge’s existing data and schema migration capabilities and is a critical step towards our goal of building a complete open-source migration toolkit. HarbourBridge continues to support command-line schema and data migration and turn-key Spanner evaluation.
Complementing the bulk data migration capabilities of HarbourBridge, we are also announcing the ability to migrate change events from MySQL to Cloud Spanner.
Supported Features in Schema Assistant
- Global type mapping. Users can customize the global mapping for how types should be mapped to Spanner consistently across the schema. For example, mapping large integers in source schema to Spanner’s NUMERIC.
- Local type mapping. Users can override the custom type mapping for a given table/column.
- Session management. A session keeps track of all the changes made to the schema mapping.
- Customization of secondary indexes. Users can add, edit and delete secondary indexes to optimize their Spanner performance.
- Customization of foreign keys and interleaved tables. Table interleaving is an important design consideration when migrating to Cloud Spanner as explained in more detail in this blog post.
Features in the pipeline
We are already working to further expand the supported set of schema editing features and welcome your feedback. We are particularly excited to expand the Schema Assistant’s design recommendations for optimizing Spanner schemas e.g. in-depth recommendations for primary key design.
HarbourBridge is open source and we gladly accept contributions from the wider community.
Indonesia moves towards advanced education with Cloud, ML & Mobile Development
3214
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Indonesia is leading the way for digital transformation in Southeast Asia. According to Google’s e-Conomy South East Asia report, the country’s 2030 Gross Merchandise Value – the value of online retailing to consumers – could be twice the value of the whole of Southeast Asia today.
This growth means that many companies need more qualified IT graduates and employees with digital skills than they have today. Fast-growing tech companies need more qualified IT graduates, and employees with digital skills. According to the World Bank, Indonesia needs an additional nine million people with digital skills by 2030. The shortage of technical talent reiterates the need to invest in a reliable skills pipeline.
Following years of digital talent developments in Indonesia, Google has become a supporter of Bangkit, an academy designed to produce high-caliber technical talent for Indonesian technology companies and startups. Bangkit has facilitated a multi-stakeholder collaboration between Google, government, industry, and universities across Indonesia.
Last year, the President of Indonesia and the Ministry of Education and Culture, Research, and Technology, acknowledged Bangkit’s significant impact, with 3,000 students completing nearly 15,000 courses and specialisations.
Building on last year’s success, Bangkit started its 2022 program in February, offering three learning paths to students:
- Cloud computing with Google Cloud, preparing students for the Google Associate Cloud Engineer certification. Some of the course components are also available online
- Mobile development with Android, preparing students for the Google Associate Android Developer exam. An online version is available here.
- Machine learning with Tensorflow, getting students ready to take the Tensorflow Developer certification. Some of the online courses are available here for others.
Bangkit 2022 has enrolled 3,100 university students who will take a five month study course, obtaining university study credit, as well as industry certifications. The program accepts diverse cohorts of people who are passionate about preparing for a tech career in the near future, with support and encouragement for women, people with disabilities, and students from across Indonesia to apply.
Since its pilot in 2019, Bangkit has been guided by three principles:
- Industry-led: provides curriculum and instructors from industry experts, including Google, GoTo and Traveloka. Instructors include key figures such as Laurence Moroney (Google, Lead AI Advocate), Google Developer Experts, and other committed professionals.
- Immersive: combines online learning methods conducted in both individual and group settings.
- Interdisciplinary: contains knowledge and best practices in tech, soft skills, and English to provide complete career readiness.
The program runs from February to July 2022, and has a 900-hour curriculum throughout the 18-week learning experience.
Benefits for students participating in Bangkit include:
- Study credit conversion
- Job opportunities at our career fair
- Google Cloud, TensorFlow and AAD exam vouchers
- Incubation funds and mentorship support from industry
Towards the end of Bangkit 2022, students will team up for the Capstone Project challenge to propose solutions to some of the nation’s most pressing problems, such as environmentalism, accessibility, and more. The top 15 teams will be selected to receive funding to incubate their capstone projects. These education and career-preparedness offerings are provided at no cost.
Google is partnering with industry, governments, universities, and employers to help meet the skill demands of today. From supporting the State of Ohio to offer tech skills to residents, to working with the University of Minnesota-Rochester to create a customized health sciences degree program, Google is here to help our partners prepare those they serve for a cloud-first world.
How The New York Times Increased Speed of Delivery by Using Kubernetes
DOWNLOAD CASE STUDY6178
Of your peers have already downloaded this article
2:40 Minutes
The most insightful time you'll spend today!
When New York Times decided a few years ago to move out of its data centers, its first deployments on the public cloud were smaller and less critical applications that were being managed on virtual machines.
“We started building more and more tools, and at some point, we realized that we were doing a disservice by treating Amazon as another data center,” says Deep Kapadia, Executive Director, Engineering at The New York Times.
Kapadia was tapped to lead a Delivery Engineering Team that would “design for the abstractions that cloud providers offer us.”
The team decided to use Google Cloud Platform and its Kubernetes-as-a-service offering, GKE (Google Kubernetes Engine). Owing to Google Cloud solution and GKE, The New York Times was able to increase the speed of delivery.
Some of the legacy VM-based deployments took 45 minutes; with Kubernetes, that time was “just a few seconds to a couple of minutes,” says Brian Balser, Engineering Manager at The New York Times.
“Teams that used to deploy on weekly schedules or had to coordinate schedules with the infrastructure team, now deploy their updates independently, and can do it daily when necessary,” says Tony Li, Site Reliability Engineer, The New York Times.
Adopting Cloud Native Computing Foundation technologies allowed The New York Times to have a more unified approach to deployment across the engineering staff, and portability for the company.
More Relevant Stories for Your Company
HSBC Leverages Google Cloud to Deliver Exceptional Cx for 37 Million Customers
HSBC is the world's largest international bank that has been in existence for 150 years. HSBC is present in 70 countries, has 37 million customers and is the number one bank in trade finance or cross-border finance. HSBC is a systemically important financial institution that is heavily regulated. Rapid growth
Explore Google Cloud SQL’s 3 Fault Tolerance Mechanism to Ease Data Pro
If you’re managing a crucial application that has to be fully fault-tolerant, you need your system to be able to handle every fault, no matter the type and scope of failure, with minimal downtime and data loss. Protecting against these faults means juggling numerous variables that can impact performance as
Google Extends Support for Windows Server Containers on Anthos for Faster App Modernization and Consistent Dev Experience
Today, many applications in organizations’ data centers run on Windows Server. Modernizing these traditional Windows apps onto Kubernetes promises a host of benefits: a consistent platform across environments, better portability, scalability, availability, simplified management and speed of deployment, just to name a few. But how? Rewriting traditional .NET applications to
Scope for Tech Adoption and Advancements in Healthcare are Still High: Google Cloud Research
Since the start of the COVID-19 pandemic, there’s been a rapid acceleration of digital transformation across the entire healthcare industry. Telehealth has become a more mainstream and safe way for patients and caregivers to connect. Machine learning modeling has helped speed up innovation and drug discovery. And new levels of
