Apache Spark has become a popular platform as it can serve all of data engineering, data exploration, and machine learning use cases. However, Spark still requires the on-premises way of managing clusters and tuning infrastructure for each job. Also, end to end use cases require Spark to be used along with technologies like TensorFlow, and programming languages like SQL and Python. Today, these operate in silos, with Spark on unstructured data lakes, SQL on data warehouses, and TensorFlow in completely separate machine learning platforms. This increases costs, reduces agility, and makes governance extremely hard; prohibiting enterprises from making insights available to the right users at the right time.

Announcing Spark on Google Cloud, now serverless and integrated

We are excited to announce Spark on Google Cloud, bringing industry’s first autoscaling serverless Spark, seamlessly integrated with the best of Google Cloud and open source tools, so you can effortlessly power ETL, data science, and data analytics use cases at scale. Google Cloud has been running large scale business critical Spark workloads for enterprise customers for 6+ years, using open source Spark in Dataproc. Today, we are furthering our commitment by enabling customers to:

1. Eliminate time spent managing Spark clusters: With serverless Spark, users submit their Spark jobs, and let them do auto-provision, and autoscale to finish.
2. Enable data users of all levels: Connect, analyze, and execute Spark jobs from the interface of users’ choice including BigQuery, Vertex AI or Dataplex, in 2 clicks, without any custom integrations.
3. Retain flexibility of consumption: No one size fits all. Use Spark as serverless, deploy on Google Kubernetes Engine (GKE), or on compute clusters based on the requirements.

With Spark on Google Cloud, we are providing a way for customers to use Spark in a cloud native manner (serverless), and seamlessly with tools used by data engineers, data analysts, and data scientists for their use cases. These tools will help customers on their way to realize the data platform redesign they have embarked on.

“Deutsche Bank is using Spark for a variety of different use cases. Migrating to GCP and adopting Serverless Spark for Dataproc allows us to optimize our resource utilization and reduce manual effort so our engineering teams can focus on delivering data products for our business instead of managing infrastructure. At the same time we can retain the existing code base and knowhow of our engineers, thus boosting adoption and making the migration a seamless experience.”—Balaji Maragalla, Director Big Data Platform, Deutsche Bank

“We see serverless Spark playing a central role in our data strategy. Serverless Spark will provide an efficient, seamless solution for teams that aren’t familiar with big data technology or don’t need to bother with idiosyncrasies of Spark to solve their own processing needs. We’re excited about the serverless aspect of the offering, as well as the seamless integration with BigQuery, Vertex AI, Dataplex and other data services.” —Saral Jain, Director of Engineering, Infrastructure and Data, Snap Inc.

Dataproc Serverless for Spark

Per IDC, developers spend 40% time writing code, and 60% of the time tuning infrastructure and managing clusters. Furthermore, not all Spark developers are infrastructure experts, resulting in higher costs and productivity impact. With serverless Spark, developers can spend all their time on the code and logic. They do not need to manage clusters or tune infrastructure. They submit Spark jobs from their interface of choice, and processing is auto-scaled to match the needs of the job. Furthermore, while Spark users today pay for the time the infrastructure is running, with serverless Spark they only pay for the job duration.

Spark through BigQuery

BigQuery, the leading data warehouse, now provides a unified interface for data analysts to write SQL or PySpark. The code is executed using serverless Spark seamlessly, without the need for infrastructure provisioning. BigQuery has been the pioneer for serverless data warehousing, and now supports serverless Spark for Spark-based analytics.

Spark through Vertex AI

Data scientists no longer need to go through custom integrations to use Spark with their notebooks. Through Vertex AI Workbench, they can connect to Spark with a single click, and do interactive development. With Vertex AI, Spark can easily be used together with other ML frameworks like TensorFlow, Pytorch, Sci-kit learn, and BigQuery ML. All the Google Cloud security, compliance, and IAM are automatically applied across Vertex AI and Spark. Once you are ready to deploy the ML models, the notebook can be executed as a Spark job in Dataproc, and scheduled as part of Vertex AI Pipelines.

Spark through Dataplex

Dataplex is an intelligent data fabric that enables organizations to centrally manage, monitor, and govern their data across data lakes, data warehouses, and data marts with consistent controls, providing access to trusted data and powering analytics at scale. Now, you can use Spark on distributed data natively through Dataplex. Dataplex provides a collaborative analytics interface, with 1-click access to SparkSQL, Notebooks, or PySpark, and the ability to save, share, search notebooks and scripts alongside data.

Flexibility of consumption

We understand one size does not fit all. Spark is available for consumption in 3 different ways based on your specific needs. For customers standardizing on Kubernetes for infrastructure management, run Spark on Google Kubernetes Engine (GKE) to improve resource utilization and simplify infrastructure management. For customers looking for Hadoop style infrastructure management, run Spark on Google Compute Engine (GCE). For customers, who’re looking for no-ops Spark deployment, use serverless Spark! 

ESG Senior Analyst Mike Leone commented, “Google Cloud is making Spark easier to use and more accessible to a wide range of users through a single, integrated platform. The ability to run Spark in a serverless manner, and through BigQuery and Vertex AI will create significant productivity improvement for customers. Further, Google’s focus on security and governance makes this Spark portfolio useful to all enterprises as they continue migrating to the Cloud.”

Getting started

Dataproc Serverless for Spark will be Generally Available within a few weeks. BigQuery and Dataplex integration is in Private Preview. Vertex AI workbench is available in Public Preview, you can get started here. For all capabilities, you can request for Preview access through this form.

You can work with Google Cloud partners to get started as well.

“We are excited to partner with Google Cloud as we look to provide our joint customers with the latest innovations on Spark. We see Spark being used for a variety of analytics and ML use cases. Google is taking Spark a step further by making it serverless, and available through BigQuery, Vertex AI and Dataplex for a wide spectrum of users.” —Sharad Kumar, Cloud First data and AI Lead at Accenture

For more information, visit our website or the watch announcement video and our conversation with Snap at Next 2021.

As AI-powered tools become increasingly prevalent, prompt engineering is becoming a skill that developers need to master. Large language models (LLMs) and other generative foundation models require contextual, specific, and tailored natural language instructions to generate the desired output. This means that developers need to write prompts that are clear, concise, and informative. 

In this blog, we will explore six best practices that will make you a more efficient prompt engineer. By following our advice, you can begin creating more personalized, accurate, and contextually aware applications. So let’s get started!

Tip #1: Know the model’s strengths and weaknesses

As AI models evolve and become more complex, it is essential for developers to comprehend their capabilities and limitations. Understanding these strengths and weaknesses can help you, as a developer, avoid making mistakes and create safer, more reliable applications.

For example, an AI model that is trained to recognize images of blueberries may not be able to recognize images of strawberries. Why? Because the model was only trained on a dataset of blueberry images. If a developer uses this model to build an application that is supposed to recognize both blueberries and strawberries, the application would likely make mistakes, leading to an ineffective outcome, and poor user experience.

It’s important to note that AI models have the ability to be biased. This is due to AI models being trained on data that is collected from the real world, and so it can reflect the inequitable power dynamics inherently rooted in our social hierarchy. If the data that is used to train an AI model is biased, then the model will also be biased. This can lead to problems if the model is used to make decisions that affect people by reinforcing societal biases. Addressing these biases is important to ensure that data is fair, promoting equality, and ensuring the responsibility of AI technology. Prompt engineers should be aware of training limitations or biases so they can craft prompts more effectively and understand what kind of prompting is even possible for a given model.

Tip #2: Be as specific as possible

AI models have the ability to comprehend a variety of prompts. For instance Google’s PaLM 2 can understand natural language prompts, multilingual text, and even programming codes like Python and JavaScript. Although AI models can be very knowledgeable, they are still imperfect, and have the ability to misinterpret prompts that are not specific enough. In order for AI models to navigate ambiguity, it is important to tailor your prompts specifically to your desired outcome. 

Let’s say you would like your AI model to generate a recipe for 50 vegan blueberry muffins. If you prompt the model with “what is a recipe for blueberry muffins?”, the model does not know that you need to make 50 muffins. It is thus unlikely to list the larger volume of ingredients you’ll need or include tips to help you more efficiently bake such a large number of muffins. The model can only go off the context that is provided. A more effective prompt would be “I am hosting 50 guests. Generate a recipe for 50 blueberry muffins.” The model is more likely to generate a response that is relevant to your request and meets your specific requirements.

Tip #3: Utilize contextual prompts

Utilize contextual information in your prompts to help the model gain an in-depth understanding of your requests. Contextual prompts can include the specific task you want the model to perform, a replica of the output you’re looking for, or a persona to emulate, from a marketer or engineer to a high school teacher. Defining a tone and perspective for an AI model gives it a blueprint of the tone, style, and focused expertise you’re looking for to improve the quality, relevance, and effectiveness of your output. 

In the case of the blueberry muffins, it is important to prompt the model using the context of the situation. The model might need more context than generating a recipe for 50 people. If it needs to be aware that the recipe must be vegan friendly, you might prompt the model by asking it to answer by emulating a skilled vegan chef. 

By providing contextual prompts, you can help ensure that your AI interactions are as seamless and efficient as possible. The model will be able to more quickly understand your request and it will be able to generate more accurate and relevant responses.

Tip #4: Provide AI models with examples 

When creating prompts for AI models, it is helpful to provide examples. This is because prompts act as instructions for the model, and examples can help the model to understand what you are asking for. Providing a prompt with an example looks something like this: “here are several recipes I like – create a new recipe based on the ones I provided.” The model can now understand the your ability and needs in order to make this pastry,

Tip #5: Experiment with prompts and personas

The way you construct your prompt impacts the model’s output. By creatively exploring different requests, you will soon have an understanding of how the model weighs its answers, and what happens when you interfuse your domain knowledge, expertise, and lived experience with the power of a multi-billion parameter large language model. 

Try experimenting with different keywords, sentence structures, and prompt lengths to discover the perfect formula. Allow yourself to step into the shoes of various personas, from work personas such as “product engineer” or “customer service representatives,” to parental figures or celebrities such as your grandmother, a celebrity chef, and explore everything from cooking to coding!

By crafting unique, and innovative, requests replete with your expertise and experience, you can learn which prompts provide you with your ideal output. Further refining your prompts, known as ‘tuning,’ allows the model to have a greater understanding and framework for your next output.

Tip #6: Try chain-of-thought prompting

Chain of thought prompting is a technique for improving the reasoning capabilities of large language models (LLMs). It works by breaking down a complex problem into smaller steps, and then prompting the LLM to provide intermediate reasoning for each step. This helps the LLM to understand the problem more deeply, and to generate more accurate and informative answers. This will help you to understand the answer better and to make sure that the LLM is actually understanding the problem. 

Conclusion

Prompt engineering is a skill that all workers, across industries and organizations, will need as AI-powered tools are becoming more prevalent. Remember to incorporate these five essential tips the next time you communicate with an AI model, so you can generate the accurate outputs that you desire. AI will forever continue to develop, constantly refining itself as we use it, so I encourage you to remember that learning, for mind and machine, is a never ending journey. Happy Prompting!

One of the largest telecommunications companies in the world, Vodafone is at the forefront of building next-generation connectivity and a sustainable digital future.

Creating this digital future requires going beyond what’s possible today and unlocking significant investment in new technology and change. For Vodafone, a key driver is the use of artificial intelligence (AI) and machine learning (ML), enabling predictive capabilities in enhancing the customer experience, improving network performance, accelerating advances in research, and much more.

Following 18 months of hard work, Vodafone has made a huge leap forward in advancing its AI capabilities at scale with the launch of its “AI Booster” AI / ML platform. Led by the Global Big Data & AI organization under Vodafone Commercial, the platform will use the latest Google technology to enable the next generation of AI use cases, such as optimizing customer experiences, customer loyalty, and product recommendations.

Vodafone’s Commercial team has long focused on advancing its AI and ML capabilities to drive business results. Yet as demand grows, it is easier said than done to embed AI and ML into the fabric of the organization and rapidly build and deploy ML use cases at scale in a highly regulated industry. Accomplishing this task means not only having the right platform infrastructure, but also developing new skills, ways of working, and processes.

Having made meaningful strides in extracting value from data by moving it into a single source of truth on Google Cloud, Vodafone had already significantly increased efficiency, reduced data costs, and improved data quality. This enabled a plethora of use cases that generate business value using analytics and data science. The next step was building industrial scale ML capability, capable of handling thousands of ML models a day across 18+ countries, while streamlining data science processes and keeping up with technological growth.

Knowing they had to do something drastically different to scale successfully, along came the idea for AI Booster.

“To maximize business value at pace and scale, our vision was to enable fast creation and horizontal / vertical scaling of use cases in an automated, standardized manner. To do this, 18 months ago we set out to build a next-generation AI / ML platform based on new Google technology, some of which hadn’t even been announced yet.

“We knew it wouldn’t be easy. People said, ‘Shoot for the stars and you might get off the ground…’ Today, we’re really proud that AI Booster is truly taking off, and went live in almost double the markets we had originally planned. Together, we’ve used the best possible ML Ops tools and created Vodafone’s “AI Booster Platform” to make data scientists’ lives easier, maximise value and take co-creation and scaling of use cases globally to another level,” says Cornelia Schaurecker, Global Group Director for Big Data & AI at Vodafone.

AI Booster: a scalable, unified ML platform built entirely on Google Cloud

Google’s Vertex AI lets customers build, deploy, and scale ML models faster, with pre-trained and custom tooling within a unified platform. Built upon Vertex AI, Vodafone’s AI Booster is a fully managed cloud-native platform that integrates seamlessly with Vodafone’s Neuron platform, a data ocean built on Google Cloud.

“As a technology platform, we’re incredibly proud of building a cutting-edge MLOps platform based on best-in-class Google Cloud architecture with in-built automation, scalability and security. The result is we’re delivering more value from data science, while embedding reliability engineering principles throughout,” comments Ashish Vijayvargia, Analytics Product Lead at Vodafone

Indeed, while Vertex AI is at the core of the platform, it’s much more than that. With tools like Cloud Build and Artifact Registry for CI/CD, and Cloud Functions for automatically triggering Vertex Pipelines, automation is at the heart of driving efficiency and reducing operational overhead and deployment times. Today, users simply complete an online form, and then, within minutes, receive a fully functional AI Booster environment with all the right guardrails, controls, and approvals.

Not long ago it could take months to move a model from a proof of concept (PoC) to launching live in production. By focusing on ML operations (MLOps), the entire ML journey is now more cost-effective, faster, and flexible, all without compromising security. PoC-to-production can now be as little as four weeks, an 80% reduction.

Diving a bit deeper, Vodafone’s AI Booster Product Manager, Sebastian Mathalikunnel, summarizes key features of the platform: “Our overarching vision was a single ML platform-as-a-service that scales horizontally (business use cases across markets) and vertically (from PoC to Production). For this, we needed innovative solutions to make it both technically and commercially feasible. Selecting a few highlights, we:

• completely automated ML lifecycle compliance activities (drift / skew detection, explainability, auditability, etc.) via reusable pipelines, containers, and managed services;
• embedded security by design into the heart of the platform;
• capitalized on Google-native ML tooling using BQML, AutoML, Vertex AI and others;
• accelerated adoption through standardized and embedded ML templates.”

For the last point, Datatonic, a Google Cloud data and AI partner, was instrumental in building reusable MLOps Turbo Templates, a reference implementation of Vertex Pipelines, to accelerate building a production-ready MLOps solution on Google Cloud.

“Our team is devoted to solving complex challenges with data and AI, in a scalable way. From the start, we knew the extent of change Vodafone was embarking on with AI Booster. Through this open-source codebase, we’ve created a common standard for deploying ML models at scale on Google Cloud. The benefit to one data scientist alone is significant, so scaling this across hundreds of data scientists can really change the business,” says Jamie Curtis, Datatonic’s Practice Lead for MLOps.

Reimagining the data scientist & machine learning engineer experience

With the new technology platform in place, driving adoption across geographies and markets is the next challenge. The technology and process changes have a considerable impact on people’s roles, learning, and ways of working. For data scientists, non-core work now is supported by machines in the background—literally at the click of a button. They can spend time doing what they do best and discovering new tools to help them do the job.

With AI Booster, data scientists and ML engineers have already started to drive greater value and collaborate on innovative solutions. Supported by instructor-led and on-demand learning paths with Google Cloud, AI Booster is also shaping a culture of experimentation and learning.

Together We Can

Eighteen months in the making, AI Booster would not have happened without the dedication of teams across Vodafone, Datatonic, and Google Cloud. Googlers from across the globe were engaged in supporting Vodafone’s journey and continue to help build the next evolution of the platform.

Cornelia highlights that “all of this was only possible due to the incredible technology and teams at Vodafone and Google Cloud, who were flexible in listening to our requirements and even tweaking their products as a result. Alongside our ‘Spirit of Vodafone,’ which encourages experimenting and adapting fast, we’re able to optimize value for our customers and business. A huge thank you also to Datatonic, who were a critical partner throughout this journey and to Intel for their valuable funding contribution.”

The Google & Vodafone partnership continues to go from strength to strength, and together, we are accelerating the digital future and finding new ways to keep people connected.

“Vodafone’s flourishing relationship with Google Cloud is a vital aspect of our evolution toward becoming a world-leading tech communications company. It accelerates our ability to create faster, more scalable solutions to business challenges like improving customer loyalty and enhancing customer experience, whilst keeping Vodafone at the forefront of AI and data science,” says Cengiz Ucbenli, Global Head of Big Data and AI, Innovation, Governance at Vodafone.

Find out more about the work Google Cloud is doing to help Vodafone here, and to learn more about how Vertex AI capabilities continue to evolve, read about our recent Applied ML Summit.

Editor’s note: Earth Day reminds us that we all can contribute to creating a cleaner, healthier, and more sustainable future. Google Cloud is excited to celebrate innovative startup companies developing new technology and driving sustainable change. On this year’s Earth Day we’re highlighting Enexor BioEnergy and their initiative to produce clean, sustainable energy from plastic and agro-waste with the help of Google Cloud solutions.

Picture a world without electricity where billions of people burn dangerous fuels and trash from landfills to cook meals and heat their homes. Children constantly cough, choking on toxic smoke that lingers in houses and sickens entire families. Because there are no pumps or purification plants, contaminated water is hauled in buckets from dirty rivers and polluted wells. Without power or lights, hospitals and emergency medical clinics can only provide basic services during the day.

Although it is 2022, this harsh life is a reality for billions of people living in developing countries around the world.

Producing clean energy from plastics and organic-waste

This Earth Day, let’s imagine what life would be like in these countries if people had access to safe and inexpensive bioenergy. In this world, indoor air is fresher. Clean water runs from faucets, while homes, businesses, and schools have electricity. Doctors and nurses have the power to treat patients 24 hours a day and save lives with advanced medical equipment. Local economies boom and entrepreneurs thrive.

This is the sustainable future Enexor BioEnergy and its partners are building—one village at a time. With the Enexor Bio-CHP™ system, we produce clean and sustainable energy from discarded plastics, organic, and biomass-waste such as rice and corn husks, seaweed, coconuts, and other biomatter while offsetting significant Carbon emissions. The Bio-CHP can be commissioned and easily installed in just a single day, with multiple systems installed side-by-side when more power is needed. By being offered under its novel Energy-as-a-Service business model, Enexor ensures that the Bio-CHP can generate immediate environmental positive impact and customer adoption.

Each Bio-CHP unit generates enough energy to power over 100 homes—and provides much-needed renewable electricity and thermal power for schools, businesses, manufacturing facilities, water pumps, Wi-Fi systems, and telecommunications towers. The Bio-CHP creates clean energy by safely oxidizing plastics and biomatter in a secure container using a high-temperature system to power a micro-turbine. This produces bioenergy, significantly reducing harmful greenhouse gas emissions.

By solving a community’s waste issues while providing more affordable renewable energy, the Bio-CHP also creates new economic opportunities and improved outcomes where it is installed. This includes using its inexpensive power to expand local services and offerings, and empowering local community clean up efforts by incentivizing waste collection via blockchain-enabled digital currencies and services such as PayGo. Additionally, the Bio-CHP is an ideal solution for businesses who desire to maximize their own sustainability efforts, gain greater energy resiliency, and save money on their current energy and waste costs.

Engaging in Google for Startups Accelerator: Climate Change

When developing the Bio-CHP, we realized we needed a reliable and experienced partner to help us incorporate the best in class machine learning and artificial intelligence into our technology. That’s why we joined the Google for Startups Accelerator and participated in the Google for Startups Accelerator: Climate Change.

The accelerator introduced us to other companies working to create a sustainable future and opened new opportunities for collaboration and partnerships. The accelerator also continues to give our small team access to the best of Google Cloud’s people, programs, and solutions. We especially want to highlight Google Cloud’s dedicated startup experts and the incredible technical support they provide, as well as the Google Cloud research credits we used to explore new solutions.

As an example, Enexor is developing a predictive maintenance tool that automatically adjusts Bio-CHP operations to match fuel composition. Another tool in development proactively identifies and flags potential Bio-CHP system and component failures—before they occur.

Google Cloud’s dedicated startup experts work closely with Enexor to develop predictive models for these tools and Enexor also used the Google Cloud research credits that were provided to build these advanced models on TensorFlow, Vertex AI, Cloud CDN and AutoML. Since each system is remotely monitored from Enexor’s global headquarters in Tennessee, predictive maintenance tools also increase the safety, reliability, and efficiency of Bio-CHP in off-grid locations.

Generating 13 billion kWh of clean power by 2040

In the future, Enexor plans to explore additional Google Cloud solutions such as BigQuery to crunch larger datasets, Google Data Studio to build dashboards, and perhaps even Google Cloud Tensor Processing Units (TPUs) to more efficiently process machine learning (ML) workloads.

These solutions will help Enexor to achieve three primary goals by 2040: to generate 13 billion kWh of clean power, reduce 120 million tons of CO₂, and provide one million people with access to clean and sustainable energy. With each Bio-CHP system, Enexor annually reduces up to 2,000 metric tons of CO₂ equivalent emissions by decreasing methane emissions released from landfills, offsetting fossil fuel-based power generation with carbon credits and minimizing waste disposal transportation emissions.

Enexor is now preparing to deploy the Bio-CHP in Accra, Ghana where it will convert organic and plastic manufacturing waste into sustainable energy. Enexor is also looking forward to working with the local Accra community and NGOs to collect discarded plastics and organic waste diverting it from ending in the rivers and oceans and instead using them as renewable fuel sources. We can’t wait to see what we accomplish next as we celebrate Earth Day 2022 and think about the sustainable future Enexor is helping to empower.

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.