May is Asian American Pacific Islander Heritage Month — a time for us to come together to celebrate and remember the important people and history of Asian and Pacific Island heritage. This feature highlights founder Georgene Huang and how her startup Fairygodboss uses easy to use Google Cloud tools to grow and innovate. 

Coming from a male-dominated industry like Wall Street, I never thought much about my gender. I was so focused on work that I didn’t feel like I could focus on my identity as a woman of color. That all changed when I found myself thrust into the job market while two months pregnant.

Traditionally, job-seeking sites are built by – and geared towards – men and their professional needs. There was no recruiting site that broke down what candidates who identify as women might expect from a company, let alone women of color like myself. I founded Fairygodboss to create a space where women can crowdsource information about opportunities and employers in order to make informed career decisions. 

Fairygodboss 

Fair•y•god•boss (noun): A person who elevates women at work.

Fairygodboss is the largest career community for women. In researching potential employers, I had many questions surrounding company culture, maternity leave, and women in leadership positions. While sites like Glassdoor provide high-level details for job seekers, I wanted more concrete examples and data than general salary information and anecdotes about work-life balance. 

Fairygodboss began as a job review site geared towards women, but has expanded significantly since our initial launch. In addition to providing free resources like career connections, job listings, virtual recruiting events, community advice, and real research on how companies treat women, we also proactively help companies improve gender diversity. Fairygodboss partners with hundreds of top employers to create a 360-degree recruiting and employer branding program designed to drive female applicants to job postings by showcasing a company’s track record of women in leadership within a company, and their diversity investments. 

Google Cloud + Fairygodboss: Data defined 

At our core, Fairygodboss is a data-driven company, and we rely on Google Cloud to help us process, store, and analyze user-driven events occurring on the site. As a highly scalable, serverless, and cost-effective data warehousing solution, BigQuery collect millions of events each day, helping us identify where traffic is coming from (and where it goes next), and it produces actionable insights to provide better recommendations, information, and opportunities for women visiting our platform. 

After seamlessly connecting our BigQuery data to Data Studio, we have been able to visualize and make sense of all the data we receive on our platform. Unlike some other data visualization tools, Data Studio allows us to analyze data in real time. With tools like BigQuery, we are now able to identify trends and gain a deep understanding for how our users interact with our site. 

Google for Startups Accelerator: Women Founders 

In order to provide more accurate job recommendations for our users, the Fairygodboss team wanted to build a machine learning prototype to classify our users and jobs into categories to facilitate more effective matching. So we applied for the Google for Startups Accelerator: Women Founders, a three-month digital accelerator program for high-potential Seed to Series A tech startups based in the U.S. and Canada. Along with tailored mentorship and product support from Googlers and industry experts, the Google for Startups Accelerator provided $100K in Google Cloud credits to scale our businesses. The mentorship we received from the Google Cloud technical experts as part of the Accelerator – special shout out to Peter Novig! – empowered us to integrate Cloud AutoML into our systems and ultimately curate more accurate job suggestions for our users.  

Not only did the Google for Startups Accelerator program help to achieve our business goals for the year, it was also extremely beneficial to be connected with a cohort of other women founders of color. While we are all building different kinds of businesses across different industries, the guidance around fundraising, scaling teams, and coping with the struggles of being a founder rang true for all of us. 

Advancing the community

It can be daunting to launch your own business – and even more so as an AAPI woman. Throughout my career, I am certain I have experienced unconscious bias around my race. Specifically, there have likely been “model minority” stereotypes about my demeanor and math abilities, assumptions that I may be mild-mannered or agreeable. While frustrating, these unfair tropes actually inspired me to see myself beyond others’ perceptions. Why be either analytical or creative, meek or brash, inspiring or agreeable? It’s the ors of life that prevent us from seeing ourselves as truly multi-dimensional individuals. I aspired to be more instead of or, and want my company to be as well. 

As an inclusion-focused business, it is extremely important that Fairygodboss mindfully engage with all underrepresented groups. Diversity can be sliced in many different ways, and these intersections lead to great opportunities for change. We have a team of volunteer ‘culture wizards’ at work that provide educational resources and videos for those wanting to learn more about different cultures and identities from members of under-represented groups. We also practice what we preach at Fairygodboss by prioritizing diverse slates and diversity in our own hiring process. We aim for the talent we hire to be representative of the diverse communities we support. It is imperative to take an intentional approach to hiring diverse talent, as it falls on all employers to promote equity and inclusion—there are always ways in which we can all improve. Fairygodboss looks forward to evolving with Google Cloud in 2021 and beyond as we work together to foster inclusive teams around the world. 

If you want to learn more about how Google Cloud can help your startup, visit our Startup Program application page here and sign up for our monthly startup newsletter to get a peek at our community activities, digital events, special offers, and more.

With access to more data than ever before, companies have never been better positioned to adopt precision marketing methods and target the right customers at the right time. Emarsys, a digital marketing platform, enables its clients to collect, analyze, and act on a wide variety of data. From websites to mobile apps to emails, Emarsys’ customers can handle data from all its digital channels on a single, easy-to-use platform. Emarsys also makes sure that customers receive the highest quality data possible, making for smarter decisions and better business practices.

“We were close to the limits of our internal data warehouse, scalability-wise. We didn’t want to get to the point where we’d have to delete data or cancel projects. In Google Cloud we saw a platform that could scale with our ambitions and be optimized for AI and real-time solutions.”

—Levente Otti, Head of Data, Emarsys

Since launching as an email solutions provider in 2000, Emarsys has grown into the world’s largest independent digital marketing platform, with more than 2,500 clients worldwide and reaching more than 1.4 billion people. By 2016, the company felt that its existing data warehouse platform was close to its limits, affecting not just day-to-day operations but also important strategic goals.

“We were close to the limits of our internal data warehouse, scalability-wise. We didn’t want to get to the point where we’d have to delete data or cancel projects,” says Levente Otti, Head of Data at Emarsys. “In Google Cloud, we saw a platform that could scale with our ambitions and be optimized for AI and real-time solutions.”

Minimal maintenance, unlimited scale with Google Cloud

Digital marketing is a highly competitive environment. Emarsys works alongside big players with a huge market share on the one hand and smaller, specialist companies on the other. It has thrived by successfully combining the all-inclusive offerings of the former with the agility of the latter, constantly looking for ways to innovate and improve. In recent years, the company had started to feel that the ability to handle large quantities of data was no longer enough. The next challenge was speed. “We truly believe that in the future, everything will be done in real time, including data processing, analytics, and AI predictive models,” Levente says.

At the start of 2016, Emarsys’ existing data warehouse was a software-as-a-service solution running on-premises, which required hardware and software maintenance in order to keep up with the company’s growing appetite for data-heavy use cases such as prediction and analytics. The existing platform had proven its worth processing large amounts of data in batches, but its real-time capabilities were limited. Moreover, Emarsys had begun to experiment with AI technology, but found that its data warehouse couldn’t scale to accommodate some of the more resource-intensive processes, such as training the predictive models. The company decided that it needed a new, cloud-based data platform.

After evaluating some of the leading cloud providers, Emarsys chose Google Cloud for its mature AI capabilities and its ease of use. “With the other solutions, we still had to rent virtual machines and hardware and be responsible for maintenance. At the time, Google Cloud was the only provider that could take that management overhead away from us, while keeping customers accounted for on every query level,” Levente says.

To implement its new data platform, Emarsys teamed up with Google Cloud Partner Aliz. Over a series of meetings, workshops, and architecture reviews, Aliz helped Emarsys navigate the Google Cloud ecosystem to find the right products for the solution it was looking for. “Aliz really helped us set off in the right direction,” explains Levente.

With Google BigQuery, we can run queries which process terabytes of data, in seconds. We can also develop our own user-defined functions, incorporating Bayesian statistics into our predictive algorithms. That means we can take into account historical data, resulting in much more accurate predictions in a scalable way within seconds.”

—Levente Otti, Head of Data, Emarsys

Emarsys’ new data platform would actually be two: one platform for batch processing data and one for real-time analysis and interactions. Firstly, a proprietary publishing component gathered all the data points from Emarsys’ various channels including the website, mobile, emails, and custom events. With Cloud Pub/Sub and Cloud Dataflow, Emarsys transported and processed the data into BigQuery, which allows for further work and reviews that take into account errors or delayed events. After this, the data was exported to the main batch processing platform, which ran on BigQuery. For the real-time analytics, Emarsys used Cloud Bigtable to access data and Cloud Dataflow to pipeline it into the real-time platform, which could communicate with AI components or interaction components via an API to deliver real-time interactions with customers.

On top of the overall data infrastructure, Emarsys built a new AI platform with Google Cloud components. Training the predictive models had been an issue in the past due to the large number of resources required, so Emarsys chose to use Google Kubernetes Engine clusters, which can scale up and down on demand, without the need for hardware configuration or management. The trained models were held securely in Cloud Storage. From here, they were integrated with BigQuery for power and flexibility, allowing Emarsys to improve not just the speed of its AI predictions but also the quality.

“With Google BigQuery, we can run queries which process terabytes of data, in seconds,” shares Levente. “We can also develop our own user-defined functions incorporating Bayesian statistics into our predictive algorithms. That means we can take into account historical data, resulting in much more accurate predictions in a scalable way within seconds.”

Real-time insight, long-term satisfaction

Google Cloud enabled Emarsys to build a scalable data and AI platform that delivers powerful, actionable insights in real time. According to Levente, the company wanted to spend less time managing overload and more time considering how it should handle data. An immediate result of the new platform has been that data is now available in a scalable way, without hardware additions and management.

“With Google Cloud, we’ve been able to build a truly real-time data platform. The norm used to be daily batch processing of data. Now, if an event happens, marketing actions can be executed within seconds, and customers can react immediately. That makes us very competitive in our market.”

—Levente Otti, Head of Data, Emarsys

The clear and innovative pricing schemes of Google Cloud have also brought a new level of accountability to Emarsys’ costs in a way that wasn’t possible with its on-premises infrastructure. “Now that we only pay for what we use, we can assign costs to specific customers or queries, which has a huge impact on our pricing and product development strategies,” says Levente.

Thanks to the power of BigQuery and the scale at which it can handle data, Emarsys can now apply its analytics and AI tools to their full potential. “It’s very important to enable our clients to create the best possible experience for customers,” says Levente. At the same time, the company has cut its AI platform costs by 70% with Kubernetes while increasing scalability compared to the previous solution. The whole data platform was built to be scalable, and its first big test came during the retail peak of Black Friday, when it comfortably handled 250,000 events per second. “Perhaps the biggest impact on the business came with the real-time nature of the new platform,” says Levente.

“With Google Cloud, we’ve been able to build a truly real-time data platform,” he explains. “The norm used to be daily batch processing of data. Now, if an event happens, marketing actions can be executed within seconds, and customers can react immediately. That makes us very competitive in our market.”

Since implementing the new platform, Emarsys has continued to innovate with it and is about to release a new Real-Time Decision Framework, which will provide customers with even more real-time products and tools. The company continues to work with Aliz and Google Cloud, exploring other products such as Google BigQuery ML and TensorFlow to improve its AI processes. “We had a problem that we wanted to tackle now, and for us, Google Cloud was the best way of doing that,” says Levente. “But it was also about looking ahead. We felt that Google Cloud offered us the best way of future-proofing our platform.”

The value of streaming analytics comes from the insights a business draws from instantaneous data processing, and the timely responses it can implement to adapt its product or service for a better customer experience. “Instantaneous data insights,” however, is a concept that varies with each use case. Some businesses optimize their data analysis for speed, while others optimize for execution cost. 

In this post, we’ll offer some tips on estimating the cost of a job in Dataflow, Google Cloud’s fully managed streaming and batch analytics service. Dataflow provides the ability to optimize a streaming analytics job through its serverless approach to resource provisioning and management. It automatically partitions your data and distributes your worker code to Compute Engine instances for parallel processing, optimizes potentially costly operations such as data aggregations, and provides on-the-fly adjustments with features like autoscaling and dynamic work rebalancing. 

The flexibility that Dataflow’s adaptive resource allocation offers is powerful; it takes away the overhead of estimating workloads to avoid paying for unutilized resources or causing failures due to the lack of processing capacity. Adaptive resource allocation can give the impression that cost estimation is unpredictable too. But it doesn’t have to be. To help you add predictability, our Dataflow team ran some simulations that provide useful mechanisms you can use when estimating the cost of any of your Dataflow jobs. 

The main insight we found from the simulations is that the cost of a Dataflow job increases linearly when sufficient resource optimization is achieved. Under this premise, running small load experiments to find your job’s optimal performance provides you with a throughput factor that you can then use to extrapolate your job’s total cost. At a high level, we recommend following these steps to estimate the cost of your Dataflow jobs: 

1. Design small load tests that help you reach 80% to 90% of resource utilization
2. Use the throughput of this pipeline as your throughput factor
3. Extrapolate your throughput factor to your production data size and calculate the number of workers you’ll need to process it all
4. Use the Google Cloud Pricing Calculator to estimate your job cost

This mechanism works well for simple jobs, such as a streaming job that moves data from Pub/Sub to BigQuery or a batch job that moves text from Cloud Storage to BigQuery. In this post, we will walk you through the process we followed to prove that throughput factors can be linearly applied to estimate total job costs for Dataflow.  

Finding the throughput factor for a streaming Dataflow job

To calculate the throughput factor of a streaming Dataflow job, we selected one of the most common use cases: ingesting data from Google’s Pub/Sub, transforming it using Dataflow’s streaming engine, then pushing the new data to BigQuery tables. We created a simulated Dataflow job that mirrored a recent client’s use case, which was a job that read 10 subscriptions from Pub/Sub as a JSON payload. Then, the 10 pipelines were flattened and pushed to 10 different BigQuery tables using dynamic destinations and BigQueryIO, as shown in the image below.

The team ran 11 small load tests for this job. The first few tests were focused on finding the job’s optimal throughput and resource allocation to calculate the job’s throughput factor. 

For the tests, we generated messages in Pub/Sub that were 500 KB on average, and we adjusted the number of messages per topic to obtain the total loads to feed each test. We tested a range of loads from 3MB/s to 250MB/s. The table below shows five of the most representative jobs with their adjusted parameters:

In order to ensure maximum resource utilization, we monitored the backlog of each test using the backlog graph in the Dataflow interface. We recommend targeting an 80% to 90% utilization so that your pipeline has enough capacity to handle small load increases. We considered 86% to 91% of CPU utilization to be our optimal utilization. In this case, it meant a 2.5MB/s per virtual CPU (vCPU) load. This is job #4 on the table above. In all tests, we used n1-standard-2 machines, which are the recommended type for streaming jobs and have two vCPUs. The rest of the tests were focused on proving that resources scale linearly using the optimal throughput, and we confirmed it.

Using the throughput factor to estimate the approximate total cost of a streaming job

Let’s assume that our full-scale job runs with a throughput of 1GB/s and runs five hours per month. Our throughput factor estimates that 2.5MB/s is the ideal throughput per worker using the n1-standard-2 machines. To support a 1GB/s throughput, we’ll need approximately 400 workers, so 200 n1-standard-2 machines.

We entered this data in the Google Cloud Pricing Calculator and found that the total cost of our full-scale job is estimated at $166.30/month. In addition to worker costs, there is also the cost of streaming data processed when you use the streaming engine. This data is priced by volume measured in gigabytes, and is typically between 30% to 50% of the worker costs. For our use case, we took a conservative approach and estimated 50%, totaling $83.15 per month. The total cost of our use case is $249.45 per month.

Finding the throughput factor for a simple batch Dataflow job

The most common use case in batch analysis using Dataflow is transferring text from Cloud Storage to BigQuery. Our small load experiments read a CSV file from Cloud Storage and transformed it into a TableRow, which was then pushed into BigQuery in batch mode. The source was split into 1 GB files. We ran tests with file sizes from 10GB to 1TB to demonstrate that optimal resource allocation scales linearly. Here are the results of these tests:

These tests demonstrated that batch analysis applies autoscaling efficiently. Once your job finds an optimized resource utilization, it scales to allocate the resources needed to complete the job with a consistent price per unit of processed data in a similar processing time.

Let’s assume that our real scale job here processes 10TB of data, given that our estimated cost using resources in us-central1 is about $0.0017/GB of processed data. The total cost of our real scale job would be about $18.06. This estimation follows this equation: cost(y) = cost(x) * Y/X, where cost(x) is the cost of your optimized small load test, X is the amount of data processed in your small load test, and Y is the amount of data processed in your real scale job. 

The key in this and the previous examples is to design small-load experiments to find your optimized pipeline setup. This setup will give you the parameters for a throughput factor that you can scale to estimate the resources needed to run your real scale job. You can then input these resource estimations in the Pricing Calculator to calculate your total job cost.