We’re excited to share the news that leading global research and advisory firm Forrester Research has named Google AppSheet a Leader in the recently released report The Forrester Wave™: Low-code Platforms for Business Developers, Q4 2021. It’s our treasured community of business developers—those closest to the challenges that line-of-business apps can solve—who deserve credit for not only the millions of apps created with AppSheet, but also the collaborative approach to no-code development that has helped further our mission of empowering everyone to build custom solutions to reclaim their time and talent. 

AppSheet received the highest marks possible in the product vision and planned enhancements criteria, with Forrester noting in the report that “AppSheet’s vision for AI-infused-and-supported citizen development is unique and particularly well suited to Google. The tech giant’s deep pockets and ecosystem give AppSheet an advantage in its path to market, market visibility, and product roadmap.”

“Features for process automation and AI are leading,” remarking that, “The platform provides a clean, intuitive modeling environment suitable for both long-running processes and triggered automations, as well as a useful range of pragmatic AI services such as document ingestion.”

Many enterprise customers including Carrefour Property – Carmila, Globe Telecom, American Electric Power, and Singapore Press Holdings (SPH) choose AppSheet as their business developer partner, along with thousands of other organizations in every industry. We are honored to serve these customers and to be a Leader in the Forrester Wave™: Low-code Platforms for Business Developers. We look forward to continuing to innovate and to helping customers on their digital transformation journey. To download the full report, visit here and enter your email address. To learn more about AppSheet, visit our website.

Advances in artificial intelligence (AI) and machine learning (ML) have led to increased adoption in the financial services sector. A prominent use for this technology is to assist in key compliance and risk functions, including the detection of fraud, money laundering, and other financial crimes and illicit finance, as well as trade manipulation — collectively referred to as “Risk AI/ML.” As the use of these models grows, so do questions about managing risks associated with the models. 

In particular, regulators, financial institutions, and technology service providers have been looking into whether existing Model Risk Management (MRM) guidance — which has traditionally been the regulatory regime applicable to managing model risk in the financial services industry — continues to be relevant for AI/ML models. And, if so, how should the guidance be interpreted and applied to this new technology?

As the financial sector increasingly adopts artificial intelligence and machine learning techniques, it is critical for regulators, financial companies and technology providers to work together to assure that there are clear rules of the road,” says Jo Ann Barefoot, AIR CEO and co-founder. “Updated guidelines on the responsible use of these models can help prevent novel technologies from causing harm, and can also open up better ways to combat risk in areas like money laundering, illicit finance, and fraud.

Our new white paper, written in partnership with the Alliance for Innovative Regulation (AIR), seeks to address that question, with the aim of fostering thought and dialogue among agencies, the financial services industry, risk model vendors, and entities interested in the performance, outputs, and compliance of models used to identify, mitigate, and combat risks in financial services. This white paper does not address issues that may arise with other applications of AI/ML in the financial services industry, such as consumer credit underwriting or models using generative AI or Large Language Models, which are better addressed iteratively. 

The paper argues that MRM guidance, given its broad, principles-based approach, continues to provide an appropriate framework for assessing financial institutions’ management of model risk, even for Risk AI/ML models. Working within an existing framework takes advantage of the knowledge and operational capabilities of institutions that already understand this framework, instead of having to create an entirely new approach, which generally takes longer to implement and make effective. Nonetheless, the paper recognizes that AI/ML models have unique traits and characteristics compared to conventional models, including their potential dynamism and pattern recognition capabilities. These distinctions must be in focus when considering how MRM guidance should be applied to Risk AI/ML models. 

Taking into account those unique aspects of AI/ML models, the paper offers specific observations and recommendations regarding the application of MRM guidance to Risk AI/ML models, including:

• Risk assessment: In assessing risk, it is important to recognize that AI/ML models are not inherently more risky than conventional models. A risk-tiering assessment must consider the targeted business application or process for which a model is used, as well as the model’s complexity and materiality. To assist in these assessments, regulators could clarify that the use of AI/ML alone does not place a model into a high-risk tier and publish further guidance to help set expectations regarding the materiality/risk ratings of AI/ML models as applied to common use cases.
• Safety and soundness: Due to the dynamic nature of Risk AI/ML models, reliance on extensive and ongoing testing focused on outcomes throughout the development and implementation stages of such models should be primary in satisfying regulatory expectations of soundness. To that end, the development of technical metrics and related testing benchmarks should be encouraged. Model “explainability,” while useful for purposes of understanding the specific outputs of AI/ML models, may be less effective or insufficient for establishing whether the model as a whole is sound and fit for purpose.
• Model documentation: The touchstone for the sufficiency of documentation should be what is needed for the bank to use and validate the model, and understand its design, theory, and logic. Disclosure of proprietary details, such as model code, is unnecessary and unhelpful in verifying the sufficiency of a model and would deter model builders from sharing best-in-class technology with financial institutions.
• Industry standards and best practices: Regulators should support the development of global standards and their use across the financial services and regulatory landscape by explicitly recognizing such standards as presumptive evidence of compliance with the MRM guidance and sound AI/ML risk mitigation practices. In addition, regulators should foster industry collaboration and training based on such standards.

Governance controls: Regulators should use guidance to advance the use of governance controls, including incremental rollouts and circuit breakers, as essential tools in mitigating risks associated with Risk AI/ML models.

In an era where AI technology has the potential to revolutionize financial services, we acknowledge the foresight of our regulators in setting a solid foundation and blueprint for navigating the labyrinth of potential risks through the MRM guidance,” says Philip Moyer, Global VP, AI and Business Solutions at Google Cloud. “We believe there is room for greater coherence and precision, enhanced risk-mitigation approaches, and refined best practices surrounding AI and ML risk models. Whether it’s in capacity building or information sharing, our call to action is for greater collaboration between regulators and financial institutions. We’re confident that our collective efforts today will help shape a more robust and resilient future for financial services.

We invite a discussion of additional considerations, including the importance of examiner and industry training and collaboration, as well as openness by regulators to continue to refine the MRM guidance as AI/ML technologies develop and standards emerge. 

Implementing our recommendations would advance several goals. It would help regulators, financial institutions, and technology providers work together to better serve their shared purpose of protecting the safety and soundness of the financial system. At the same time, implementing the recommendations and continuing work in this space would promote the adoption of cutting-edge technologies in the industry, including those that combat such scourges as money laundering, illicit finance, and fraud.

You can read the full white paper here.

Background

At IKEA we have multiple places in our customer journey in various channels where different kinds of personalization can deliver a superior customer experience. Product recommendations in the shopping basket, content recommendations in editorial sections, inspirational recommendations on product pages and more. After a while in the broader “recommendations” team there was a decision to split the team to have one sub-team focused on product recommendations. The pandemic altered customer behavior and needs as well. At that inflection point we decided to change our way of working and dive head-first into a more scientific approach to handle the operational complexities of delivering high quality product recommendations at scale. We deemed this necessary to improve our level of personalization and to have a holistic understanding of our customers.

Data Driven Decisions

The first step was to radically improve our ability to get high-quality quantitative information to understand how our ‘recommendation’ solutions affected personalization. We did this through high volume A/B testing on customer behaviour and after initial experimentation, we had a few key learnings:  

1. The mix of both UX and algorithms are really important for a cohesive customer experience. 
2. The quality of personalization can’t be measured in silos. Statistical significance  can be attained  by testing several groups of recommendations at once.

Once we came up with a solid framework for gathering data and acknowledged how little we knew about our customers, we were able to explore an incredible number of creative options – nothing was off the table. This was a very humbling experience, in that it opened up new perspectives for personalization, a more curious and less confined way of thinking. We learned to trust the data because it might show you things you don’t expect. 

Experimentation and Learning Framework

Our teams created ways to quickly deploy experimental modifications to our existing solution. This enabled experimentation in the front-end with the user experience, including details in headings and images. This also covered tweaks in the backend with anything from detailed manual additions or removals of recommendations to mixing and matching of various algorithms both home grown and from Recommendations AI.

This flexibility came with an overhead–more complexity and cost relative to directly retrieving recommendations from Recommendations AI. However, the benefit was that we were no longer dependent on manual evaluation of what made for a good recommendation system. We aligned on a data-driven and qualitative approach to provisioning recommendations and significantly accelerated our experimentation timeline. Together with optimization of the CI/CD pipeline this enabled the team to take an idea or hypothesis from inception to A/B testing with customers in less than half an hour.

Recommendations AI Experiments

Our team’s infrastructure was already running on GCP and when we received early access to Recommendations AI, the requirements to get started were minimal and that allowed us to start with initial tests requiring minimal effort and investment.

We started with a few use-cases and identified places where our existing recommendation algorithms needed improvement or complementary recommendations. We also explored additional ways where more useful information could be presented to the customers through personalized recommendations. 

Recommendations AI Model Combinations

While Recommendations AI might be considered a simple API to get a set of product recommendations, as we dove deeper into the solution it became apparent that it could be tweaked in several different ways to offer many fine tuning configurations to meet business goals. While too much fine tuning and customization could lead to subpar performance, in general we found that it was a great strategy to give us several versions of ML powered recommendations to work with. The further you personalize the experience, the more options you have to likely pick the best one for the customer.

Recommendations AI models like  ‘Recommended for you’, ‘Frequently Bought Together’ and ‘Others you may like’; are coupled with business goals like optimizing for conversion rate, click through rate and revenue. We experimented with many different model combinations and custom rules. All this was easily configurable right in the GCP console. One of the simplest custom configurations we used was to only recommend items that were in stock, and when items were out of stock we looked at similar items that were available to augment the experience. 

Collaboration with Google

Our collaboration with Google Cloud accelerated our learning process during experimentation. We worked closely together early in the product development. Additionally, their model provided flexibility to change direction and allow for more options than we had previously. Ultimately, this provided us a way to drastically improve our time to market with a product that produced tremendous results that we could not have accomplished on our own.

Results and Takeaways

With more personalized and real-time recommendations available we saw great success. We were able to increase the number of relevant recommendations displayed on a page by +400%. To accommodate the wider repertoire of recommendations we had to change the user experience. For example, in some places we had horizontally scrolling displays of product recommendations which were much easier for customers to use.

Another consequence of displaying more personalized recommendations was tangible improvement to conversion rate and average order value. Recommendations AI algorithms helped customers in two ways: 

1. Customers were able to find products that they liked quickly and establish their preferred choice among other options more quickly as well, giving them confidence to make a purchase through much fewer clicks. Even though we previously already had well tuned recommendations of several types, with Recommendations AI we measured +30% improvement in click through rates. 
2. Average order value saw a +2% surge with numerous examples of how Recommendations AI could help customers find both attractive and directly complementary products, expanding the customer purchase from a single product to an entire home furnishing solution.

As a direct effect of having stronger business results, the team started exploring more places in the customer journey where our growing buffet of recommendations could be used. We’d start with an initial experiment to answer if displaying recommendations in the specific context made sense at all. Frequently the data that emerged from these experiments prodded us to iterate further on what additional types of recommendations would be most appropriate to show to the customer as the customer’s behaviour evolved. Today, most of IKEA’s site recommendations are powered by Recommendations AI.

One key takeaway is that for some types of personalized recommendations there are benefits to using advanced algorithms that require a lot of high level data science and engineering competence to build since they outperform simplistic approaches. In some places, simplistic approaches work very well and in others the right decision is to not have product recommendations at all. For an effective use of product recommendations you need to have all the above options and the ability to tell when to use which one.

Next steps

When working with something so tightly related to customer experience, there is a constant change in user behaviour and new learnings to observe and adapt to. Product recommendations are rarely the main stand alone experience and frequently something that is used to help and enhance an experience. We see a lot of value in having a large toolbox of possible options and a team with a relentless focus on collaboration to improve the customer experience. We’re working directly with the Recommendations AI team and experimenting with several new features that we’re excited about. 

In the future we see opportunities of improving the customer journey through a more visual experience that inspires the customer rather than relying on customers to use their imagination to visualize groups of products together. Vision Product Search provides that and is something we’re looking into deploying next. We’ll be sharing more about our journey with Recommendations AI at the Google Cloud Retail Summit session ‘IKEA’s Approach to Building a Powerful Recommendations Engine’ on July 27th 2021.

Best wishes to all developers from the IKEA product recommendations team & the Google Recommendations AI team!

At Google Cloud, we continue to invest in key technologies to progress towards our true north star on invisible security: making strong security pervasive and simple for everyone. Our investments are based on insights from our world-class threat intelligence teams and experience helping customers respond to the most sophisticated cyberattacks. Customers can tap into these capabilities to gain perspective and visibility on the most dangerous threat actors that no one else has. 

Recent advances in artificial intelligence (AI), particularly large language models (LLMs), accelerate our ability to help the people who are responsible for keeping their organizations safe. These new models not only give people a more natural and creative way to understand and manage security, they give people access to AI-powered expertise to go beyond what they could do alone. 

At the RSA Conference 2023, we are excited to announce Google Cloud Security AI Workbench, an industry-first extensible platform powered by a specialized, security LLM, Sec-PaLM. This new security model is fine-tuned for security use cases, incorporating our unsurpassed security intelligence such as Google’s visibility into the threat landscape and Mandiant’s frontline intelligence on vulnerabilities, malware, threat indicators, and behavioral threat actor profiles. 

Google Cloud Security AI Workbench powers new offerings that can now uniquely address three top security challenges: threat overload, toilsome tools, and the talent gap. It will also feature partner plug-in integrations to bring threat intelligence, workflow, and other critical security functionality to customers, with Accenture being the first partner to utilize Security AI Workbench. 

The platform will also let customers make their private data available to the platform at inference time; ensuring we honor all our data privacy commitments to customers. Because Security AI Workbench is built on Google Cloud’s Vertex AI infrastructure, customers control their data with enterprise-grade capabilities such as data isolation, data protection, sovereignty, and compliance support. 

Preventing threats from spreading beyond the first infection

We already provide best-in-class capabilities to help organizations immediately respond to threats. But what if we could not just identify and contain initial infections, but also help prevent them from happening anywhere else? With our AI advances, we can now combine world class threat intelligence with point-in-time incident analysis and novel AI-based detections and analytics to help prevent new infections. These advances are critical to help counter a potential surge in adversarial attacks that use machine learning and generative AI systems. That’s why we’re excited to introduce:  

• VirusTotal Code Insight uses Sec-PaLM to help analyze and explain the behavior of potentially malicious scripts, and will be able to better detect which scripts are actually threats. 
• Mandiant Breach Analytics for Chronicle leverages Google Cloud and Mandiant Threat Intelligence to automatically alert you to active breaches in your environment. It will use Sec-PaLM to help contextualize and respond instantly to these critical findings.

These new updates build on the existing AI in Google’s industry-leading solutions. For example, Chronicle Security Operations already uses frontline intelligence, integrated reasoning, and machine learning to identify initial infections, prioritize impact, and contain threats. Another example is reCAPTCHA Enterprise, which uses image noising capabilities to help protect your site from adversaries that leverage novel AI advances, greatly enhancing our defenses against bots. 

Adding intelligence to reduce toil

At Google Cloud, we help organizations modernize security wherever they are, in part by simplifying their security tools and controls whenever possible. Advances in generative AI can help reduce the number of tools organizations need to secure their vast attack surface areas and ultimately, empower systems to secure themselves. This will minimize the toil it takes to manage multiple environments, to generate security design and capabilities, and to generate security controls. Today, we’re announcing: 

• Assured OSS will use LLMs to help us add even more open-source software (OSS) packages to our OSS vulnerability management solution, which offers the same curated and vulnerability-tested packages that we use at Google.
• Mandiant Threat Intelligence AI, built on top of Mandiant’s massive threat graph, will leverage Sec-PaLM to quickly find, summarize, and act on threats relevant to your organization. 

These announcements build on existing capabilities that help customers centralize visibility and control, detect targets, and improve security across their platform. For example, Security Command Center (SCC) uses always-on machine learning to detect malicious scripts executing in the customer container environment and immediately alert the customer. In addition, Cloud Data Loss Prevention leverages machine learning to find and classify data, and with Confidential Computing you can collaborate on, train, and deploy sensitive and regulated AI models in the cloud, all while preserving confidentiality. 

Evolving how practitioners do security to close the talent gap 

At Google, we believe that to truly democratize security, we need to first acknowledge that AI will soon usher in a new era for security expertise that will profoundly impact how practitioners “do” security. Most people who are responsible for security — developers, system administrators, SRE, even junior analysts — are not security specialists by training. 

Imagine a world where novices and security experts are paired with AI expertise to free themselves from repetition and burnout, and accomplish tasks that seem impossible to us today. To help power this evolution, we’re embedding Sec-PaLM-based features that can make security more understandable while helping to improve effectiveness with exciting new capabilities in two of our solutions: 

• Chronicle AI: Chronicle customers will be able to search billions of security events and interact conversationally with the results, ask follow-up questions, and quickly generate detections, all without learning a new syntax or schema.
• Security Command Center AI: Security Command Center will translate complex attack graphs to human-readable explanations of attack exposure, including impacted assets and recommended mitigations. It will also provide AI-powered risk summaries for security, compliance, and privacy findings for Google Cloud.

These new releases bolster our existing efforts to tackle these issues through capabilities like IAM Recommender, which suggests permissions better suited to actual usage patterns. We will soon be augmenting this capability to cover organizational policies, further enabling the administrator to help improve the security posture of their organization. In addition, Mandiant Automated Defense applies machine learning to help reduce the repetitive Tier 1 alert triage problem and address alert fatigue. 

Offering availability 

VirusTotal Code Insight, available now in Preview, is our first example of putting Security AI Workbench to work for our customers. We will be rolling out other offerings to trusted testers in coming months, and they will be available in Preview more broadly this summer. Click here for the demo. 

Security AI Workbench, including Sec-PaLM and partner integrations, in addition to the product innovations described in our demo, are all building blocks for a larger effort to elevate security across the ecosystem. So far, that effort: 

• Provides assistive functions to rapidly develop IT generalist talent to Tier 1 security operator status in a way that wasn’t previously feasible. Security Command Center now can summarize threat intelligence insights and findings for Google Cloud, and Chronicle can quickly generate YARA-L rules or other detections. 
• Provides advanced functions such as iterative query and multivariate detection generation, conversational filtering and interaction with results, and smart case awareness to empower advanced Tier 2 and 3 security operators to focus on threat analysis instead of struggling with process and toil. Mandiant Threat Intelligence users now can elevate their core competencies to hunt, investigate, and remediate threats — using the same tools our own Mandiant experts use. 
• Fuses threat intelligence and AI-based analytic capabilities, which are unsurpassed in the market. VirusTotal Code Insight enables security teams to help gain insights and identify threats in suspicious code. This can significantly enhance their ability to detect and mitigate potential attacks.

However, this is just an initial step. We’ll continue to iterate and innovate, and we encourage customers and partners to leverage Security AI Workbench in new and exciting ways. Moving forward, we anticipate many new use cases to emerge over time. 

Building a safer future

While generative AI has recently captured the imagination, Sec-PaLM is based on years of foundational AI research by Google and DeepMind, and the deep expertise of our security teams. This work includes new efforts to expand our partner ecosystem to provide businesses with security capabilities at every layer of the cybersecurity stack. We have only just begun to realize the power of applying generative AI to security, and we look forward to continuing to leverage this expertise for our customers and drive advancements across the security community.