Getting started with Kubernetes is often harder than it needs to be. While working with a cluster “from scratch” can be a great learning exercise or a good solution for some highly specialized workloads, often the details of cluster management can be made easier by utilizing a managed service offering. Google Kubernetes Engine (GKE) allows for an easier end-to-end developer experience with convenient tooling and built-in integrations along with the convenience of offering Kubernetes clusters as a managed service.

GKE is the most mature container orchestration service available today, delivering a fully-managed service and hands-off experience with the GKE Autopilot mode of operation. GKE provides industry-first capabilities such as release channels, multi-cluster support, unique four-way auto scaling, node auto repair, and can support up to 15K nodes in a single cluster. 

Our modern, end-to-end platform is built on cloud-native principles you are already familiar with and prioritizes speed, security, and flexibility, in ways that are highly differentiated from other cloud platforms. 

We have put together a new reference guide for you as you begin your journey developing on GKE. It covers every step of your journey from writing, running, operating, to managing code. Even if it isn’t your first time using GKE, this e-book will be a valuable resource highlighting important considerations and best practices. By implementing the technical recommendations, following the steps, and utilizing the tools described, you can reach the following goals:

• Write, deploy, and debug code faster with Cloud Code and Cloud Shell
• Continuously integrate and deliver updates with Cloud Build
• Run easily, securly, cost effectively at scale with GKE
• Monitor and troubleshoot with Google Cloud’s operations suite 

Kick-start your journey by downloading the e-book and join us live June 22 at 9am PDT for our half-day Cloud OnBoard event: Getting Started with Google Kubernetes Engine. 

Businesses around the globe are realizing the benefits of replacing legacy data silos with cloud-based enterprise data warehouses, including easier collaboration across business units and access to insights within their data that were previously unseen. However, bringing data from numerous disparate data sources into a single data warehouse requires you to develop pipelines that ingest data from these various sources into your enterprise data warehouse. Historically, this has meant that data engineering teams across the organization procure and implement various tools to do so. But this adds significant complexity to managing and maintaining all these pipelines and makes it much harder to effectively scale these efforts across the organization. Developing enterprise-grade, cloud-native pipelines to bring data into your data warehouse can alleviate many of these challenges. But, if done incorrectly, these pipelines can present new challenges that your teams will have to spend their time and energy addressing. 

Developing cloud-based data ingestion pipelines that replicate data from various sources into your cloud data warehouse can be a massive undertaking that requires significant investment of staffing resources. Such a large project can seem overwhelming and it can be difficult to identify where to begin planning such a project. We have defined the following principles for data pipeline planning to begin the process. These principles are intended to help you answer key business questions about your effort and begin to build data pipelines that address your business and technical needs. Each section below details a principle of data pipelines and certain factors your teams should consider as they begin developing their pipelines.

Principle 1: Clarify your objectives

The first principle to consider for pipeline development is clarify your objectives. This can be broadly defined as taking a holistic approach to pipeline development that encompasses requirements from several perspectives: technical teams, regulatory or policy requirements, desired outcomes, business goals, key timelines, available teams and their skill sets, and downstream data users. Clarifying your objectives clearly identifies and defines requirements from each key stakeholder at the beginning of the process and continually checks development against these requirements to ensure the pipelines built will meet these requirements.This is done by first clearly defining the desired end state for each project in a way that addresses a demonstrated business need of downstream data users. Remember that data pipelines are almost always the means to accomplish your end state, rather than the end state itself. An example of an effectively defined end-state is “enabling teams to gain a better understanding of our customers by providing access to our CRM data within our cloud data warehouse” rather than “move data from our CRM to our cloud data warehouse”. This may seem like a merely semantic difference, but framing the problem in terms of business needs helps your teams make technical decisions that will best meet these needs. 

After clearly defining the business problem you are trying to solve, you should facilitate requirement gathering from each stakeholder and use these requirements to guide the technical development and implementation of your ingestion pipelines. We recommend gathering stakeholders from each team, including downstream data users, prior to development to gather requirements for the technical implementation of the data pipeline. These will include critical timelines, uptime requirements, data update frequency, data transformation, DevOps needs, and security, policy, or regulatory requirements by which a data pipeline must meet.

Principle 2: Build your team

The second principle to consider for pipeline development is build your team. This means ensuring you have the right people with the right skills available in the right places to develop, deploy, and maintain your data pipelines. After you have gathered your pipeline requirements, you can begin to develop a summary architecture that will be used to build and deploy your data pipelines. This will help you identify the human talent you will need to successfully build, deploy, and manage these data pipelines and identify any potential shortfalls that would require additional support from either third-party partners or new team members.

Not only do you need to ensure you have the right people and skill sets available in aggregate, but these individuals need to be effectively structured to empower them to maximize their abilities. This means developing team structures that are optimized for each team’s responsibilities and their ability to support adjacent teams as needed.

This also means developing processes that prevent blockers to technical development whenever possible, such as ensuring that teams have all of the appropriate permissions they need to move data from the original source to your cloud data warehouse without violating the concept of least privilege. Developers need access to the original data source (depending on your requirements and architecture) in addition to the destination data warehouse. Examples of this are ensuring that developers have access to develop and/or connect to a Salesforce Connected App or read access to specific Search Ads 360 data fields.

Principle 3: Minimize time to value

The third principle to consider for pipeline development is minimize time to value. This means considering the long-term maintenance burden of a data pipeline prior to developing and deploying it in addition to being able to deploy a minimum viable pipeline as quickly as possible. Generally speaking, we recommend the following approach to building data pipelines to minimize their maintenance burden: Write as little code as possible. Functionally, this can be implemented by:

1. Leveraging interface-based data ingestion products whenever possible. These products minimize the amount of code that requires ongoing maintenance and empower users who aren’t software developers to build data pipelines. They can also reduce development time for data pipelines, allowing them to be deployed and updated more quickly. 

• Products like Google Data Transfer Service and Fivetran allow for managed data ingestion pipelines by any user to centralize data from SaaS applications, databases, file systems, and other tooling. With little to no code required, these managed services enable you to connect your data warehouse to your sources quickly and easily.
• For workloads managed by ETL developers and data engineers, tools like Google Cloud’s Data Fusion provide an easy-to-use visual interface for designing, managing and monitoring advanced pipelines with complex transformations.

2. Whenever interface-based products or data connectors are insufficient, use pre-existing code templates. Examples of this include templates available for Dataflow that allow users to define variables and run pipelines for common data ingestion use cases, and the Public Datasets pipeline architecture that our Datasets team uses for onboarding.

3. If neither of these options are sufficient, utilize managed services to deploy code for your pipelines. Managed services, such as Dataflow or Dataproc, eliminate the operational overhead of managing pipeline configuration by automatically scaling pipeline instances within predefined parameters.

Principle 4: Increase data trust and transparency

The fourth principle to consider for pipeline development is increase data trust and transparency. For the purposes of this document, we define this as the process of overseeing and managing data pipelines across all tools. Numerous data ingestion pipelines that each leverage different tools or are not developed under a coordinated management plan can result in “tech sprawl”, which significantly increases the management overhead of data ingestion pipelines as the quantity of data pipelines increases. This becomes especially cumbersome if you are subject to service-level agreements, or legal, regulatory, or policy requirements for overseeing data pipelines. Preventing tech sprawl is, by far, the best strategy for dealing with it by developing streamlined pipeline management processes that automate reporting. Although this can theoretically be achieved by building all of your data pipelines using a single cloud-based product, we do not recommend doing so because it prevents you from taking advantage of features and cost optimizations that come with choosing the best product for your use case. 

A monitoring service such as Google Cloud Monitoring Service or Splunk that automates metrics, events, and metadata collection from various products, including those hosted in on-premise and hybrid computing environments, can help you centralize reporting and monitoring of your data pipelines. A metadata management tool such as Google Cloud’s Data Catalog or Informatica’s Enterprise Data Catalog can help you better communicate the nuances of your data so users better understand which data resources are best fit for a given use case. This significantly reduces your pipeline’s governance burden by eliminating manual reporting processes that often result in inaccuracies or lagging updates.

Principle 5: Manage costs

The fifth principle to consider for pipeline development is manage costs. This encompasses both the cost of cloud resources and the staffing costs necessary to design, develop, deploy, and maintain your cloud resources. We believe that your goal should not necessarily be to minimize cost, but rather maximizing the value of your investment. This means maximizing the impact of every dollar spent by minimizing waste in cloud resource utilization and human time. There are several factors to consider when it comes to managing costs:

• Use the right tool for the job – Different data ingestion pipelines will have different requirements for latency, uptime, transformations, etc. Similarly, different data pipeline tools have different strengths and weaknesses. Choosing the right tool for each data pipeline can help your pipelines operate significantly more efficiently. This can reduce your overall cost, free up staffing time to focus on the most impactful projects, and make your pipelines much more efficient.
• Standardize resource labeling –  Implement and utilize a consistent labeling schema across all tools and platforms to have the most comprehensive view of your organization’s spending. One example is requiring all resources to be labeled by the cost center or team at time of creation. Consistent labeling allows you to monitor your spend across different teams and calculate the overall value of your cloud spending.
• Implement cost controls – If available, leverage cost controls to prevent errors that result in unexpectedly large bills. 
• Capture cloud spend – Capture your spend on all cloud resource utilization for internal analysis using a cloud data warehouse and a data visualization tool. Without it, you won’t understand the context of changes in cloud spend and how they correlate with changes in business.
• Make cost management everyone’s job – Managing costs should be part of the responsibilities of everyone who can create or utilize cloud resources. To do this well, we recommend making cloud spend reporting more transparent internally and/or implementing chargebacks to internal cost centers based on utilization.

Long-term, the increased granularity in cost reporting available within Google Cloud can help you better measure your key performance indicators. You can shift from cost-based reporting (i.e. – “We spent $X on BigQuery storage last month”) to value-based reporting (i.e. – “It costs $X to serve customers who bring in $Y revenue”). 

To learn more about managing costs, check out Google Cloud’s “Understanding the principles of cost optimization” white paper.

Principle 6: Leverage continually improving services

The sixth principle is leverage continually improving services. Cloud services are consistently improving their performance and stability, even if some of these improvements are not obvious to users. These improvements can help your pipelines run faster, cheaper, and more consistently over time. You can take advantage of the benefits of these improvements by:

• Automating both your pipelines and pipeline management: Not only should data pipelines be automated, but almost all aspects of managing your pipelines can also be automated. This includes pipeline/data lineage tracking, monitoring, cost management, scheduling, access management and more. This helps reduce long-term operational costs of each data pipeline that can significantly alter your value proposition and prevent any manual configurations from negating the benefits of later product improvements.
• Minimizing pipeline complexity whenever possible: While ingestion pipelines are relatively easy to develop using UI-based or managed services, they also require continued maintenance as long as they are in use. The most easily maintained data ingestion pipelines are typically the ones that minimize complexity and leverage automatic optimization capabilities. Any transformation in a data ingestion pipeline is a manual optimization of the pipeline that may struggle to adapt or scale as the underlying services improve. You can minimize the need for such transformations by building ELT (extract, load, transform) pipelines rather than ETL (extract, transform, load) pipelines. This pushes transformations down to the data warehouse that is use a specifically optimized query engine to transform your data rather than manually configured pipelines.

Next steps

If you’re looking for more information about developing your cloud-based data platform, check out our Build a modern, unified analytics data platform whitepaper. You can also visit our data integration site to learn more and find ways to get started with your data integration journey.

Once you’re ready to begin building your data ingestion pipelines, learn more about how Cloud Data Fusion and Fivetran can help you make sure your pipelines address these principles.

Google Cloud Results

• Replaced conflicting, time-consuming paper logs with a near real-time, transparent app
• Scaled rapidly and easily to accommodate new teams thanks to Google App Engine
• Enables photographic and video communication to replace confusing text
• New factory solution rolled out in just 15 days

In 1877, Pietro Barilla set up a small bakery to make pasta and baked goods for the people of Parma, Italy. Today, Barilla applies its 140 years of baking knowledge on a global scale, with six major manufacturing sites in Italy and an international network employing over 8,000 people. As the world’s leading producer of pasta, Barilla knows that when it comes to making quality food, great communication is key. That’s why the company plans to become a completely digital, looking to technology to improve the way it works.

Teams at the Barilla factory in Cremona work along a production line more than one-kilometer long, staffed by three shifts of workers a day. When one shift handed over to the next or requested machine maintenance teams, they used paper notebooks and unofficial instant messaging to communicate. That meant there was no authoritative, real-time record of events, communication was messy, oversight was poor, and teams had to hold daily morning meetings to synchronise notes.

Barilla worked with the Google Cloud Partner Injenia to create a solution, beginning with a consultative process on the factory floor.

“We had the idea to to start from the bottom and work up,” says Cristiano Boscato at Injenia. “Barilla’s top staff were brilliant about letting us do it. Eight of us from Injenia spent months on the factory lines with Barilla workers, collecting ideas on Google Docs, making presentations with Slides and collecting feedback with Forms. The CollaborAction app we created is the result of an amazing partnership.”

Co-designing a team social network

“Everything at the Cremona plant was managed offline, with paper,” explains Alessandra Ardrizzoia, Digital Engagement Senior Manager at Barilla. “Workers on the line would track events in notebooks, the shift leader would have another notebook, and the leader of the maintenance team would have yet another notebook. Everybody wrote their own text description of events, so there would be mismatches in the information going around.”

To resolve this, teams would meet at 8:30am every day to reconstruct a consistent narrative. In addition, machine maintenance workers were already using instant messaging to communicate with the line. Barilla and Injenia looked for a solution that could deliver a searchable, single version of events, with the ease of use of a mobile messaging application.

After consulting factory workers for ideas, Injenia created CollaborAction, a custom-built app that brought G Suite collaboration tools together on an Google App Engine platform, using Google Cloud SQL to index files. Google+, Google Drive and Hangouts were not only highly available and easy-to-use, they also “helped with fast adoption, with interfaces that workers could already relate to.” Meanwhile Google App Engine enabled the Injenia team to deliver updates and new versions at speed, as part of a feedback process with workers who offered suggestions through a link to Forms embedded in the app.

Google+ provides an intuitive social media dashboard that workers felt comfortable with. Now teams use company tablets placed at intervals along the line to log in, report issues to other teams, photograph problems, schedule maintenance, give status updates through Hangouts chat, and have visibility on the whole process as it takes place.

“Everyone in the Cremona plant was really happy with the new social collaboration process. Because they were involved in designing the solution, they felt involved and really engaged with the process,” says Alessandra. “And now that everyone is aligned with CollaborAction, all the work in the plant is more effective. They are more agile and can use their time in more added-value activities.”

Optimization and a national roll-out

Created in Cremona, now CollaborAction connects over 1,000 users in six of Barilla’s factories in Italy. “The pilot at Cremona took one month, and adoption has been easier and faster in every plant we’ve taken it to,” says Cristiano. “We have another five or six plants more, and it takes no more than 15 days to introduce. That’s incredible.”

Because CollaborAction is a mobile app built on Google App Engine, scaling to meet new demand has been simple. Now maintenance teams use the app on smartphones, line workers use it on tablets, and shift leaders use it on laptops, so the entire team is aligned in close to real-time on a single version of events. And now teams communicate with video and photographs as well as text, there’s less room for confusion, as Alessandra explains. “It’s no problem understanding what’s happening in a video or picture, compared to a message that just says ‘something is going wrong.’ On a production line, where one part leads into the next, that speed makes a difference, and means we don’t have to throw as much food away when something breaks down.”

“Now we’re collecting feedback from all of the plants using CollaborAction and using it to create a standardised solution that we can apply across all of our plants,” says Alessandra. “We’re side-by-side with the workers in that sense, trying to address their needs with new features. It’s a way to make the workers feel like part of the solution, and that the app represents their needs and their voice.”

Solving a universal problem

By the end of 2018, Barilla and Injenia aim to have deployed CollaborAction to 2,700 employees at 18 factories worldwide. Barilla has already collected more than 50,000 posts with the app, including around 20,000 photographs and videos, and is now considering ways to apply Cloud Machine Learning Engine to create a maintenance chatbot or direct IoT connection with machinery.

“CollaborAction hasn’t just made our maintenance processes faster and more efficient, its also exponentially increased the knowledge and understanding employees have about their work,” says Alessandra. “It’s improving team spirit, too, such as when employees use CollaborAction to arrange to play soccer. It’s become the main communication tool for the entire plant.”

Workplace collaboration continues to evolve as hybrid work expands its footprint. While both “same-time” and “staggered-time” (or asynchronous) collaboration modes each have their place in the hybrid work environment, organizations and teams sometimes overuse one mode over another. That’s probably because in-person meetings or collaborating in real time have been the default for many organizations, leading to a “same-time” bias, even as new tools make working across time zones and locations seamless. Ultimately, the key for making collaboration scalable and to support employee wellbeing is choosing the right method and tool for the task at hand. With that in mind, here are some ways to get the most out of same-time and staggered-time collaboration in your organization.

Making same-time collaboration more valuable

Historically, many organizations have valued same-time interactions for the fast, direct exchange of information and feedback. They also bring teams together and build human connections. While all these outcomes are powerful and important, same-time collaboration also has some drawbacks — especially in a hybrid environment.

First, same-time interactions can be difficult to arrange. Coordinating schedules requires advanced planning, especially when employees are distributed across time zones or working outside of traditional business hours. Second, back-to-back scheduling can contribute to meeting fatigue. In our recently commissioned global hybrid work survey with Economist Impact, 72% of respondents acknowledged that virtual meetings improve inclusion and participation. At the same time, 68% said there are too many virtual meetings in general, suggesting that even though employees value same-time collaboration, they want it in moderation.

To address these challenges, organizations should consider the collaboration scenarios in which same-time interactions can make the greatest impact. It’s particularly useful when consensus or immediate action is needed, as in briefings, decision-making, and crisis management.

For effective same-time collaboration in a hybrid environment, your employees need tools that help them coordinate real-time interactions and minimize time spent switching apps and context. With Google Workspace, teams can create and access a shared agenda or Google Doc inside a Google Calendar invite, helping align expectations and goals before, during, and after scheduled same-time work sessions. And, the more integrated your collaboration tools are, the easier it is for employees to get right to work when inspiration strikes or a quick decision is needed. For example, team members can initiate spontaneous work sessions by starting a Google Meet directly from Docs, Sheets, or Slides.

Additionally, consider how you can bring real-world collaboration activities into virtual environments to enhance same-time collaboration for distributed teams. For instance, digital whiteboard tools, like Jamboard and Miro in Google Meet, allow people to brainstorm, ideate, and problem-solve in real time from anywhere. Interactive solutions like these empower employees to work together in new ways.

Expanding what’s possible with staggered-time collaboration

Widespread adoption of hybrid work has prompted organizations to turn more frequently to staggered-time collaboration — where team members are free to share and respond to information in their own time. A key benefit of staggered collaboration is improving the experience and wellbeing of employees — specifically, by promoting participation across the team and support for flexible work habits. Allowing team members to engage when they’re most productive, or when it doesn’t interfere with their scheduled focus time, supports their wellbeing. It can also help employees feel heard, valued, and empowered. Staggered-time collaboration can also give team members who may feel uncomfortable or vulnerable speaking up in real time, or who just need time to think about their feedback, a more inviting way to share their perspectives.

Staggered-time collaboration can enhance a range of collaboration scenarios, including assigning tasks and action items, knowledge- and resource-sharing, brainstorming, and giving feedback. To make these interactions successful, employees need secure, intuitive tools that give them the information they need to do their jobs from anywhere, at any time. For example, in Spaces, team members can initiate and take part in group chat threads devoted to project- and topic-based discussions, as well as access files and tasks, to stay in the loop whenever they’re able to participate. For people working on a task or project at varied times, creating a dynamic, single source of truth makes all the difference.

When collaborating on shared documents, a feature like smart canvas enables teammates to assign tasks, build checklists, and share files using @-mentions for easy information access. Meanwhile, automated, built-in summaries in Docs help the team get up to speed without losing focus. Features like these help everyone stay on the same page (literally), even if they’re contributing to a project at different times.

Flexibility and collaboration are not mutually exclusive

Helping people know which tasks call for real-time collaboration and which tasks are better done on their own time is critical for sustaining productivity and wellbeing in a hybrid work world. Employees who are empowered to work in ways that complement their needs, preferences, and schedules typically reward their organization for that flexibility. As Holger Reisinger, Paul Sephton, and Dane Fetterer wrote for the Harvard Business Review, “When leaders give employees the freedom to choose where and when they work, it signals they trust them to do the job they were hired to do. The data shows that that trust is then paid back…at a very high rate, building a tight-knit culture of inclusivity and belonging.”

When you create team norms around same-time and staggered-time collaboration, you can help your employees stay productive and connected — to each other and the broader organization. And by giving them tools that support seamless, secure collaboration experiences from anywhere, the impact on productivity, employee wellbeing, and morale can be transformational.

Whether with the cloud or within their own data centers, enterprises have undergone a period of remarkable consolidation and centralization of their compute resources. But with the rise of ever more powerful mobile devices, and increasingly capable cellular networks, application architects are starting to think beyond the confines of the data center, and looking out to the edge. 

What exactly do we mean by edge? Think of the edge as distributed compute happening on a wide variety of non-traditional devices — mobile phones of course, but also equipment sensors in factories, industrial equipment, or even temperature and reaction monitoring in a remote lab. Edge devices are also connected devices, and can communicate back to the mothership over wireless or cellular networks. 

Equipped with increasingly powerful processors, these edge devices are being called upon to perform tasks that have thus far been outside the scope of traditional IT. For enterprises, this could mean pre-processing incoming telemetry in a vehicle, collecting video in kiosks at a mall, gathering quality control data with cameras in a warehouse, or delivering interactive media to retail stores. Enterprises are also relying on edge to ingest data from outposts or devices that have even more intermittent connectivity, e.g., oil rigs or farm equipment, filtering that data to improve quality, reducing it to right-size information load, and processing it in the cloud. New data and models are then pushed back to the edge; in addition, we can also push configuration, software, and media updates and decentralize processing workload.

Edge isn’t all about enabling new use cases – it’s also about right-sizing environments and improving resource utilization. For example, adopting an edge model can also relieve load on existing data centers. 

But while edge computing is full of promise for enterprises, there are many pieces that are still works in progress. Further, developing edge workloads is very different from developing traditional applications, which enjoy the benefits of persistent data connections and run on well-resourced hardware platforms. As such, cloud architects are still in the early days of figuring out how to use and implement edge for their organizations. 

Fortunately, there are tools you can use to help ease the transition to edge computing — and that likely fit into your organization’s existing computing systems. Kubernetes, of course, but also higher level management tools like Anthos, which provides a consistent control plane across cloud, private data center and edge locations. Other parts of the Anthos family – Anthos Config Management and Anthos Service Mesh — go one step further and provide consistent, centralized management to your edge and cloud deployments. And there’s more to come.

For the remainder of this blog post, we’ll dive deeper into the past and current state of edge computing, and the benefits that architects and developers can expect to see from edge computing. In a next post, we’ll take a deeper look at some of the challenges that designing for edge introduces, and some of the advantages the average enterprise has in adopting the edge model. Finally, we’ll look at the Google Cloud tools that are available today to help you build out your edge environment, and look at some early customer examples that highlight what’s possible today — and that will spark your imagination for what to do tomorrow. 

The evolution of edge computing 

The edge is not a new concept. In fact, it’s been around for the last two decades, spanning many use cases that are prevalent today. One of the first applications for edge was to use content delivery networks (CDN) to cache and serve daily static website pages near clients, for example, web servers in California data centers serving financial data to European customers. 

As connectivity has improved and software evolved, the edge has evolved too, and the focus has shifted towards using edge to distribute services. First, simple services expanded from static HTML to javascript libraries or image repositories. Common functions like image transformation, credit and address validation support services followed. Soon, organizations were deploying more complex cloudlet and clustered microservices installations, as well as distributed and replicated datasets. The term “endpoint” became ubiquitous, and APIs profilerated. 

In parallel, there’s been an explosion of creativity in hardware, microcontrollers and dedicated edge devices. Fit-for-purpose products were deployed globally. Services like Google Cloud IoT Core extended our ability to manage and securely connect these dispersed devices, allowing platform managers to register tools and leverage managed services like Pub/Sub and Dataflow for data ingestion. And with Kubernetes, large remote clusters — mini private clouds in and of themselves — operate as self-healing, autoscaling services across the broader internet, opening the door to new models for applications and architectural patterns. In short, both distributed asynchronous systems and economies have blossomed.

What does this mean for enterprises? For the purposes of this series, edge means you can now go beyond the corporate network, beyond cloud VPCs, and beyond hybrid. The modern edge is not sitting at a major remote data center, nor is it a CDN, cloud provider, or in a corporate data center rack — it’s just as likely to look like 100 of these attached to a thousand sensors.

Edge, in short, is about having hardware and devices installed at remote locations that can process and communicate back the information they collect and generate. The edge management challenge, meanwhile, is being able to push configuration and software/model/media updates to these remote locations when they are connected.

Enable new use cases

Today, we have reached a new threshold for edge computing — one where micro-data-processing centers are deployed as the edge of a fractal arm, as it were. Together, they form a broad, geographically distributed, always-on framework for streaming, collecting, processing and serving asynchronous data. This big, loosely coupled application system lives, breathes and grows. Always changing, always learning from the data it collects — and always pushing out updated models when the tendrils are connected. 

Right now, the rise of 5G is pushing the limits of edge even further. Devices enabled with 5G can transmit using a mobile network — no ISP required — enabling connectivity anywhere within reach of a cell tower. Granted, these networks have lower bandwidth, but they are often more than adequate for certain types of data, for example fire sensors in forests bordering remote towns that emit temperature or carbon monoxide data periodically. Recently, Google Cloud partnered with AT&T to enhance business use of 5G edge technology but there is so much more that can be done. 

Reduce data center investments

In addition to enabling the digitization of a broad range of new use cases, adopting edge can also benefit your existing data center.

Let’s face it: data centers are expensive to maintain. Moving some data center load to edge locations can reduce your data center infrastructure investment, as well as compute time spent there. Edge services tend to have much lower service level objectives (SLOs) than data center services, driving lower levels of hardware investment. Edge installations also tend to tolerate disconnectedness, and thus function perfectly well with lower SLOs — and lower costs. 

Let’s look at an example of where edge can really reduce costs: big data. Back in the day, we used to build monolithic serial processors — state machines — that had to keep track of where they were in processing in case of failure. But time and again, we’ve seen that smaller, more distributed processing can break down big, expensive problems into smaller, more cost-effective chunks. 

Starting with the explosion of MapReduce almost 20 years ago, big-data workloads were parallelized across clusters on a network, and state management was simplified with intermediate output to share, wait for, or restart processing from checkpoints. Those monolithic systems were replaced by cheaper, smarter, networked clusters and data repositories where parallel work could be executed and rendered into workable datasets. 

Flash forward to today, and we are seeing those same concepts applied and distributed to edge data-collection points. In this evolution of big data processing, we are scaling up and out to the point where observation data is so massive that it must first be prefiltered, and then preprocessed down to a manageable size and still be actionable. Only then should it be written back to the main data repositories for more resource-intensive processing and model building.

In short, data collection, cleanup, and potentially initial aggregation happens at the edge location, which reduces the amount of junk data sitting in costly data stores. This increases performance of the core data warehouse, and reduces the size and cost of network transfers and storage! 

The edge is a huge opportunity for today’s enterprises. But designing environments that can make effective use of the edge isn’t without its challenges. Stick around for part two of this series, where we look at some of the architectural challenges typically encountered while designing for the edge and how we begin to address them.