Like virtually every business across the globe, The New York Times had to quickly adapt to the challenges of the coronavirus pandemic last year. Fortunately, our data system with Google Cloud positioned us to perform quickly and efficiently in the new normal. 

How we use data

We have an end-to-end type of data platform; on one side we work very closely with our product teams to collect the right level of data that they’re interested in, such as which articles people are reading, and how long they’re staying onsite. We frequently measure our audience to understand our user segments, and how they come onsite or use our apps. We then provide that data to analysts for end-to-end analytics. 

On the other side, the newsroom is also focused on audience, and we build tools to help them understand how Google Search or different social promotions play a role in a person’s decision to read The New York Times, and also to get a better sense of their behavior on our pages. With this data, the newsroom can make decisions about information that should be displayed on our homepage or in push notifications. 

Ultimately, we’re interested in behavioral analytics—how people engage with our site and our apps. We want to understand different behavioral patterns, and which factors or features will encourage users to register and subscribe with us. 

We also use data to create or curate preferences around personalization, to ensure we’re delivering to our users fresh content, or content that they may not have normally read. Likewise, our data also gets used in our targeting system, so that we can send out the right messaging about our various subscription packages to the right users.

Choosing to migrate to Google Cloud

When I came to The New York Times over five years ago, our data architecture was not working for us. Our infrastructure was gathering data that proved harder for analysts to crunch on a daily basis. We were also hitting hang ups with how that data was streaming into our system and environment. Back then we’d run a query and then go grab some coffee, hoping that the query would finish or give us the right data by the time we came back to our desks. Sometimes it would, sometimes it wouldn’t.

We realized that Hadoop was definitely not going to be the on-premises solution for us, and that’s when we started talking with the Google Cloud team. We began our digital transformation with a migration to BigQuery, their fully managed, serverless database warehouse. We were under a pretty aggressive migration timeline, focusing first on moving over analytics. We made sure our analysts got a top-of-the-line system that treated them the way that they themselves would want to treat the data. 

One significant prominent requirement in our data architecture choice was to enable analysts to be able to work as quickly as they needed to provide high-quality deliverables for their business partners. For our analysts, the transition to BigQuery was night and day. I still remember when my manager ran his very first query on BigQuery and was ready to go grab his coffee, but the query finished by the time he got up from his chair. Our analysts talk about that to this day.

While we were doing the BigQuery transition, we did have concerns about our other systems not scaling correctly. Two years ago, we weren’t sure we’d be able to scale up to the audience we expected on that election day. We were able to band-aid a solution back then, but we knew we only had two more years to figure out a real, dependable solution. 

During that time, we moved our streaming pipeline over to Google Cloud, primarily using App Engine, which has been a flexible environment that enabled quick scaling changes and requirements as needed. Dataflow and Pub/Sub also played significant roles in managing the data. In Q4 of 2020 we had our most significant traffic ever recorded, at 273 million global readers, and four straight days of the highest traffic we’ve had compared to other election weeks. We were proud to see that there was no data loss.

A couple of years ago, on our legacy system, I was up until three in the morning one night trying to keep data running for their needs. This year, for election night, I relaxed and ate a pint of ice cream because I was able to more easily manage our data environment, allowing us to set and meet higher expectations for data ingestion, analysis and insight among our partners in the newsroom.

How COVID-19 changed our 2020 roadmap 

The coronavirus pandemic definitely wasn’t on my team’s roadmap for 2020, and it’s important to mention here that The New York Times is not fundamentally a data company. Our job is to get the news out to our users every single day in paper, on apps, and onsite. Our newsroom didn’t expect the need to build out a giant coronavirus database that would enrich the news they share every day. 

Our newsroom moves quickly, and our engineers have built one of the most comprehensive datasets on COVID-19 in the U.S. With Google, The New York Times decided to make our data publicly available on BigQuery Google’s COVID-19 public dataset. Check out this webinar for more details on our evolution architecture:https://www.youtube.com/embed/mtNlrFpschU?enablejsapi=1&

Flexible approach

We have many different teams that work within Google Cloud, and they’ve been able to pick from the range of available services and tailor project requirements keeping those tools available in mind.

One challenge we think about with the data platform at The New York Times is determining the priorities of what we build. Our ability to engage with product teams at Google though the Data Analytics Customer Council allows us to see into the BigQuery roadmap, or the data analytics roadmap, and plays a significant role in determining where we focus our own development. For example, we’ve built tools like our Data Reporting API, which reads data directly from BigQuery, in order to take advantage of tools like BigQuery BI Engine. This approach encourages our analysts to be better managers of their domains around dimensions and metrics, but not have to focus on building caching mechanisms of their data. Getting that kind of clarity helps us plan how to build The New York Times in the new normal and beyond.

If you are interested to learn more about the data teams at the New York Times, take a look at our open tech roles here and you’ll find many interesting articles at NYT data blog.

We introduced the Google Cloud Ready – Sustainability designation earlier this year to showcase those partners committed to help global businesses and governments accelerate their sustainability programs. These partners build solutions that enhance the capabilities and ease the adoption of powerful Google Cloud technologies, such as Google Earth Engine and BigQuery, allowing customers to leverage data-rich solutions that help reduce their carbon footprints.

Today, we’re pleased to announce growth of the Google Cloud Ready – Sustainability program, with 12 new partners joining the initiative and bringing their climate, ESG, and sustainability platforms to Google Cloud. These partners include:

Aclima is pioneering an entirely new way to diagnose the health of our air and track climate-changing pollution. Powered by its network of roving and stationary sensors, Aclima measures air pollution and greenhouse gasses at unprecedented scales and with block-by-block resolution.

Sustainability at Airbus means uniting and safeguarding the world in a safe, ethical, and socially and environmentally responsible way. Airbus has a comprehensive sustainability strategy built on four core commitments, which guide the company’s approach to the way it does business and how it designs its products and services: Lead the journey toward clean aerospace, respect human rights and foster inclusion, build the business on the foundation of safety and quality, and exemplify business integrity.

Atlas AI is a predictive analytics platform that analyzes, monitors, and forecasts regions of growth, vulnerability, and opportunity around the world to offer insight into where organizations can grow most successfully, and where investment can boost historically underserved communities. Atlas AI’s platform has been used to expand water and sanitation infrastructure, promote new electrification, target community health services, and broaden internet access in countries across Sub-Saharan Africa and South Asia.

BlueSky Resources makes sense of sensors from both public and private sources by harmonizing inputs from ground, aerial and space based inputs. Expertise in atmospheric science and cloud technology allows BlueSky to provide understanding and insights related to the correlation of emissions insights to assets and activities. This powerful combination of data, climate science and delivery of insights is enabling focused sustainability impact across clients in various industries including energy, waste management, industry and natural resource management.

Electricity Maps provides companies with actionable data quantifying the carbon intensity and origin of electricity. This data is available on an hourly basis across 50+ countries and more than 160 regions. Electricity Maps’ mission is to organize the world’s electricity data to drive the transition toward a truly decarbonized electricity system.

FlexiDAO is a global climate tech company based in the Netherlands and Spain. The company works closely with other critical stakeholders to co-create the international standard around energy-related emissions compliance. Thanks to FlexiDAO’s end-to-end 24/7 Carbon-free Energy platform, companies can quantify and confidently showcase their contribution to society’s decarbonization.

LevelTen Energy helps organizations achieve carbon-free energy usage targets (on an annual and 24/7 basis) by delivering access to the world’s largest clean energy marketplace, and the software, data, analytics, and expertise required for efficient transactions. The LevelTen Platform connects energy buyers and over 40 sustainability advisors with more than 1,800 carbon-free energy projects in 24 countries across North America and Europe.

Ren is a SaaS platform built on Google Cloud that enables companies with global supply chains to source the cleanest energy possible. Despite using country-sized amounts of energy, most companies have no idea how to transition to renewables due to complex financial, technical, and logistical challenges. Ren unlocks cost savings, provides the cleanest energy possible, and ensures companies meet their carbon commitments on time.

Sidewalk Labs, an urban innovation unit in Google, builds products to radically improve quality of life in cities for all. Delve is a product that helps real estate teams design more sustainable buildings and neighborhood blocks, faster. Mesa automates building controls to deliver savings and comfort to commercial building owners and tenants. With these products and others, Sidewalk Labs helps commercial real estate developers, building owners and city planners make more sustainable choices for the built environment that are better for communities and the planet.

Tomorrow.io is The World’s Weather and Climate Security Platform, helping countries, businesses, and individuals manage their weather and climate security challenges. The platform is fully customizable to any industry impacted by the weather. Customers around the world use Tomorrow.io to dramatically improve operational efficiency. Tomorrow.io was built from the ground up to help teams prepare for the business impact of weather by automating decision-making and enabling climate adaptation at scale.

UP42 is a geospatial developer platform and marketplace bringing together industry-leading data and ready-to-use processing algorithms. The platform enables organizations to build, run, and scale geospatial products. With the ability to choose from a wide range of high-resolution commercial and open satellite data, aerial, weather, and others, solution providers can apply best-in-class machine learning and/or processing modules to gain valuable geospatial insights and streamline their processes.

Woza is a sustainable innovation platform that leverages deep geospatial knowledge and existing best-in-class technologies to develop a new generation of streamlined analytics workflows focused on sustainability. Companies in agri-food, energy, and public sector are partnering with Woza to accelerate their journey to Industry 4.0.

Adding expertise to accelerate sustainability use cases

New partners in the initiative join our existing Google Cloud Ready – Sustainability partners like Carto, Climate Engine, Geotab, NGIS, and Planet Labs PBC bringing a wealth of industry knowledge, offering solutions for sustainability challenges ranging from first-mile sustainable sourcing and spatial finance to fleet electrification and rich geospatial visualizations.

CARTO is the world’s leading Location Intelligence platform, enabling organizations to use spatial data and analysis for more efficient delivery routes, better behavioral marketing, strategic store placements, and much more. The company’s solutions extend the geospatial capabilities available in BigQuery, while leveraging the near limitless scalability that Google Cloud provides. When it comes to sustainability, CARTO’s platform is trusted by a wide range of organizations, including Greenpeace, Vizzuality, Litterati, Indigo, WWF, the Marine Conservation Institute, The World Bank, and the Institute for Sustainable Cities.

Climate Engine leverages data from Google Earth Engine and other ecosystem partners to help organizations improve their climate change-related risk planning in areas such as water use, agriculture, storm risk, and wildfire spread. By linking the economy and the environment, organizations can understand how environmental risks are affecting their markets and discover opportunities to reduce their emissions and potential supply chain or operational disruptions from climate-related events.

Geotab is advancing security, connecting commercial vehicles to the cloud, and providing data-driven analytics to help customers better manage their fleets. Processing billions of data points daily, Geotab helps businesses improve and optimize fleet productivity, enhance safety, and achieve sustainability goals and stronger compliance.

Geospatial solutions provider NGIS built a SaaS-based first-mile sustainable sourcing solution called TraceMark using Google Cloud’s geospatial platform and technologies from other ecosystem partners. Several global CPG firms have already used TraceMark to modernize their geospatial workflows and help facilitate the use of space-based data for supply chain sustainability transformation.

Planet Labs PBC operates the largest fleet of Earth imaging satellites in history, with approximately 200 satellites in orbit. Planet’s mission is to image the whole earth’s landmass every day to make global change visible, accessible, and actionable. As a Public Benefit Corporation, Planet’s Public Benefit Purpose is to accelerate humanity toward a more sustainable, secure, and prosperous world by illuminating environmental and social change.

How the Google Cloud Ready – Sustainability program works

If you are a Google Cloud partner with sustainability solutions and expertise to share, the Google Cloud Ready – Sustainability program is open for applications. Entry into the program requires that the partner solution delivers quantifiable results for climate mitigation, adaptation, or reporting needs. To apply for the Google Cloud Ready – Sustainability designation, the solution must:

• Be available on Google Cloud
• Address ESG risk, and assist customers in achieving ESG targets and/or support typical ESG goal frameworks, such as the United Nations’ SDGs
• Demonstrate repeatability
• Meet minimum Google Cloud application development best practices, including security, performance, scalability, availability, and carbon footprint reporting for available services
• Have Google Cloud Carbon Footprint Reporting enabled
• Have at least one public customer case study available.

The selection process begins with an evaluation of the solution. If a partner meets the above criteria, Google Cloud provides a suggested roadmap for tier progression within the program and then issues a formal acknowledgement of participation in the Google Cloud Ready – Sustainability program. Together, Google Cloud sustainability partners can deliver platforms that are helping businesses and governments accelerate progress aligned to their environmental goals.

Google Cloud will showcase the validated solutions on the Google Cloud Partner Directory Listing, Google Cloud Ready Sustainability Partner Advantage page, and — if applicable — via the Google Cloud Marketplace. We hope to help customers better understand how these technologies can help them meet their ESG goals, find the right solution for their particular challenge, and implement a solution faster.

Prospective partners can visit the Partner Portal to learn more about the Google Cloud Ready – Sustainability program or complete an application.

The Opportunity for Streamed AR/VR Content 

What if you could get a high quality AR/VR experience without a dedicated physical computer—or even without a physical tether?  

In the past, interacting with VR required a dedicated, high-end workstation and, depending on the headset, wall-mounted sensors and a dedicated physical space. Complex tasks in VR can push the limits of sensor reach, cable-length, and spatial boundaries, entangling the artist and restricting their movement. This solution was not scalable beyond a handful of advanced use cases.

Recently, tetherless VR headsets from manufacturers such as HTC and Oculus have emerged that free the user from these odious restrictions, enabling a new freedom to experience VR and AR from just about anywhere. The enhanced portability and reduced cost has led to substantially increased adoption among consumers and opened up many new exploratory use cases in enterprise.

However, while these headsets are more accessible and portable, a tradeoff in compute power was required in order to achieve these goals. The limited on-device compute power of this new generation of Head Mounted Displays (HMDs) is acceptable for many consumer applications such as casual gaming. However, advanced enterprise workloads with heavy graphics, compute, or memory requirements can be difficult or even impossible to run on these lightweight devices.

By combining NVIDIA CloudXR with their powerful GPUs running NVIDIA RTX Virtual Workstations (vWS) in Google Cloud data centers, you can experience high-fidelity VR and AR applications from just about anywhere with a good internet connection. The heavy computations are performed in the cloud on a GPU-attached VM and the content streams to any CloudXR compatible headset. 

The combination of Google Cloud’s private fiber optic network — the same network we built for global delivery of YouTube content — and CloudXR’s QoS technologies provides the user with the highest possible quality of service. In fact, the streaming experience is comparable to that of a headset tethered to a powerful physical workstation, but without the friction of hardware and cables. 

This combination of power and portability sets the stage to unlock the potential of high-quality gaming and enterprise AR/VR experiences anywhere on Earth.

Case Study: Creating a Masterpiece

Digital character creation is a core skill for many 3D artists today. One of the preferred methods of 3D character creation is digital sculpting, which lets artists create both hard-surface and organic shapes with high levels of accuracy and control.

Sculpting is one of the many tasks in the character designer’s skillset. Artists must also master the texturing, rigging, and posing of characters as part of the digital character creation pipeline. 

Mastering all these tasks can be challenging, and often, the technology gets in the way; working in multiple applications requires switching contexts and workflows, breaking the artist’s creative flow. The traditional user experience can also be unintuitive, forcing the creator to translate what they want to do with their hands and head into mouse movement and keyboard presses.

Masterpiece Studio Pro revolutionizes this character creation workflow by giving artists the first fully immersive 3D creation pipeline. Artists work in VR, giving them a far more intuitive and seamless way to work which combines the best of the digital and physical worlds.

In Masterpiece Studio Pro, the artist can work on the character or object at any scale, using familiar tools and hand gestures to sculpt a model, much as they would a real clay figure. Performing other tasks such as skeleton creation is simple, allowing the artist to work directly with the limbs of a character to place and adjust the joints.

Bringing It All Together

NVIDIA CloudXR streaming from the cloud provides tremendous opportunities for new creative use cases within gaming and across the enterprise. This solution joins Virtual Studio for Gaming as the latest in our series to help developers build better games.

The Masterpiece Studio use case is a powerful demonstration of new modalities for content creation and collaboration. To try CloudXR with NVIDIA RTX vWS on Google Cloud for yourself, see this tutorial. Masterpiece Studio Pro also offers extensive learning materials and a free trial.