KRM Series Part 5: Learn to Manage and Configure Hosted Resources with Kubernetes
3216
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
This is the fifth and final post in a multi-part series about the Kubernetes Resource Model. Check out parts 1, 2, 3, and 4 to learn more.
In part 2 of this series, we learned how the Kubernetes Resource Model works, and how the Kubernetes control plane takes action to ensure that your desired resource state matches the running state.
Up until now, that “running resource state” has existed inside the world of Kubernetes- Pods, for example, run on Nodes inside a cluster. The exception to this is any core Kubernetes resource that depends on your cloud provider. For instance, GKE Services of type Load Balancer depend on Google Cloud network load balancers, and GKE has a Google Cloud-specific controller that will spin up those resources on your behalf.
But if you’re operating a Kubernetes platform, it’s likely that you have resources that live entirely outside of Kubernetes. You might have CI/CD triggers, IAM policies, firewall rules, databases. The first post of this series introduced the platform diagram below, and asserted that “Kubernetes can be the powerful declarative control plane that manages large swaths” of that platform. Let’s close that loop by exploring how to use the Kubernetes Resource Model to configure and provision resources hosted in Google Cloud.
Why use KRM for hosted resources?
Before diving into the “what” and “how” of using KRM for cloud-hosted resources, let’s first ask “why.” There is already an active ecosystem of infrastructure-as-code tools, including Terraform, that can manage cloud-hosted resources. Why use KRM to manage resources outside of the cluster boundary?
Three big reasons. The first is consistency. The last post explored ways to ensure consistency across multiple Kubernetes clusters- but what about consistency between Kubernetes resources and cloud resources? If you have org-wide policies you’d like to enforce on Kubernetes resources, chances are that you also have policies around hosted resources. So one reason to manage cloud resources with KRM is to standardize your infrastructure toolchain, unifying your Kubernetes and cloud resource configuration into one language (YAML), one Git config repo, one policy enforcement mechanism.
The second reason is continuous reconciliation. One major advantage of Kubernetes is its control-loop architecture. So if you use KRM to deploy a hosted firewall rule, Kubernetes will work constantly to make sure that resource is always deployed to your cloud provider- even if it gets manually deleted.
A third reason to consider using KRM for hosted resources is the ability to integrate tools like kustomize into your hosted resource specs, allowing you to customize resource specifications without templating languages.
These benefits have resulted in a new ecosystem of KRM tools designed to manage cloud-hosted resources, including the Crossplane project, as well as first-party tools from AWS, Azure, and Google Cloud.
Let’s explore how to use Google Cloud Config Connector to manage GCP-hosted resources with KRM.
Introducing Config Connector
Config Connector is a tool designed specifically for managing Google Cloud resources with the Kubernetes Resource Model. It works by installing a set of GCP-specific resource controllers onto your GKE cluster, along with a set of Kubernetes Custom Resources for Google Cloud products, from Cloud DNS to Pub/Sub.
How does it work? Let’s say that a security administrator at Cymbal Bank wants to start working more closely with the platform team to define and test Policy Controller constraints. But they don’t have access to a Linux machine, which is the operating system used by the platform team. The platform team can address this by manually setting up a Google Compute Engine (GCE) Linux instance for the security admin. But with Config Connector, the platform team can instead create a declarative KRM resource for a GCE instance, commit it to the config repo, and Config Connector will spin up the instance on their behalf.
What does this declarative resource look like? A Config Connector resource is just a regular Kubernetes-style YAML file- in this case, a custom resource called Compute Instance. In the resource spec, the platform team can define specific fields, like what GCE machine type to use.
apiVersion: compute.cnrm.cloud.google.com/v1beta1kind: ComputeInstancemetadata:annotations:cnrm.cloud.google.com/allow-stopping-for-update: "true"name: secadmin-debianlabels:created-from: "image"network-type: "subnetwork"spec:machineType: n1-standard-1zone: us-west1-abootDisk:initializeParams:size: 24type: pd-ssdsourceImageRef:external: debian-cloud/debian-9...
Once the platform team commits this resource to the Config Sync repo, Config Sync will deploy the resource to the cymbal-admin GKE cluster, and Config Connector, running on that same cluster, will spin up the GCE resource represented in the file.
This KRM workflow for cloud resources opens the door for powerful automation, like custom UIs to automate resource requests within the Cymbal Bank org.
Integrating Config Connector with Policy Controller
By using Config Connector to manage Google Cloud-hosted resources as KRM, you can adopt Policy Controller to enforce guardrails across your cloud and Kubernetes resources.
Let’s say that the data analytics team at Cymbal Bank is beginning to adopt BigQuery. While the security team is approving production usage of that product, the platform team wants to make sure no real customer data is imported. Together, Config Connector and Policy Controller can set up guardrails for BigQuery usage within Cymbal Bank.
Config Connector supports BigQuery resources, including Jobs, Datasets, and Tables. The platform team can work with the analytics team to define a test dataset, containing mocked data, as KRM, pushing those resources to the Config Sync repo as they did with the GCE instance resource.
apiVersion: bigquery.cnrm.cloud.google.com/v1beta1kind: BigQueryJobmetadata:name: cymbal-mock-load-jobannotations:configsync.gke.io/cluster-name-selector: cymbal-adminspec:location: "US"jobTimeoutMs: "600000"load:sourceUris:- "gs://cymbal-bank-datasets/cymbal-mock-transactions.csv"
From there, the platform team can create a custom Constraint Template for Policy Controller, limiting the allowed Cymbal datasets to only the pre-vetted mock dataset:
rego: |package bigquerydatasetallownameviolation[{"msg": msg}] {input.review.object.kind == "BigQueryDataset"input.review.object.metadata.name != input.parameters.allowedNamemsg := sprintf("The BigQuery dataset name %v is not allowed", [input.review.object.metadata.name])}apiVersion: constraints.gatekeeper.sh/v1beta1
These guardrails, combined with IAM, can allow your organization to adopt new cloud products safely- not only defining who can set up certain resources, but within those resources, what field values are allowed.
Manage existing GCP resources with Config Connector
Another useful feature of Config Connector is that it supports importing existing Google Cloud resources into KRM format, allowing you to bring live-running resources into the management domain of Config Connector.
You can use the config-connector command line tool to do this, exporting specific resource URIs into static files:
config-connector export "//sqladmin.googleapis.com/sql/v1beta4/projects/cymbal-bank/instances/cymbal-dev" \--output cloudsql/
Output:
apiVersion: sql.cnrm.cloud.google.com/v1beta1kind: SQLInstancemetadata:annotations:cnrm.cloud.google.com/project-id: cymbal-bankname: cymbal-devspec:databaseVersion: POSTGRES_12region: us-east1resourceID: cymbal-dev...
From here, we can push these KRM resources to the config repo, and allow Config Sync and Config Controller to start lifecycling the resources on our behalf. The screenshot below shows that the cymbal-dev Cloud SQL database now has the “managed-by-cnrm” label, indicating that it’s now being managed from Config Connector (CNRM = “cloud-native resource management”).
This resource export tool is especially useful for teams looking to try out KRM for hosted resources, without having to invest in writing a new set of YAML files for their existing resources. And if you’re ready to adopt Config Connector for lots of existing resources, the tool has a bulk export option as well.
Overall, while managing hosted resources with KRM is still a newer paradigm, it can provide lots of benefits for resource consistency and policy enforcement. Want to try out Config Connector yourself? Check out the part 5 demo.
This post concludes the Build a Platform with KRM series. Hopefully these posts and demos provided some inspiration on how to build a platform around Kubernetes, with the right abstractions and base-layer tools in mind.
Thanks for reading, and stay tuned for new KRM products and features from Google.
S4 Agtech Transforms Agriculture with Google Cloud
7508
Of your peers have already read this article.
8:30 Minutes
The most insightful time you'll spend today!
Like countless other industries, farming is going digital and undergoing big changes—driven by access to more actionable information. The agriculture business can now gather and analyze georeferenced data from satellites, combined with data from IoT sensors in fields, crop rotation and yield histories, weather patterns, seed genotypes and soil composition to help increase the quantity and quality of crops.
This is essential for businesses in the agriculture industry, but it’s also critical to address growing food shortages around the world.
At S4, we create technology to de-risk crop production. We provide customers seeking agricultural risk management solutions with the tools to make better, data-driven decisions for their crop planning, based on machine learning and proprietary algorithms.
We interpret plant evolution on a global scale with predictive modeling and analytics, and offer super-efficient risk-transferring solutions. Our multi-cloud platform includes a petabyte-scale database, an open source stack, and—after 50 proof-of-concept evaluations—BigQuery for our data warehouse and the Cloud SQL database service to handle OLTP queries to our PostgreSQL database.
These PoCs included, among others, Microsoft Azure Data Lake Analytics, IBM Netezza, Postgres/PostGIS running on IBM bare-metal servers with SATA SSDs and on Google’s Compute Engine with NVMe disks, and on-premises memSQL, CitusData and Yandex ClickHouse.
Weeding out risk in an uncertain market
According to recent research, climate extreme events like drought, heat waves, and heavy precipitation are responsible for 18-43% of global variation in crop yields for maize, spring wheat, rice, and soybeans. This is a clear trend for other crops as well. Such variation poses risks of food shortages as well as large financial risks to farmers, insurers, and regions dependent on successful crop yields. Also, it creates vast humanitarian difficulties.
Our mission at S4 is to help de-risk crop production by matching the right data with analytics tools so farmers and other participants in the agricultural value chain can plan better, resulting in more reliable food supplies.
In a nutshell, we create indices out of biological assets. These indices measure yield losses on crops that are caused by the effects of weather and other factors, which are then used as underlying assets for products, such as swap/derivative contracts and parametric insurance policies, to transfer risk to the financial markets.
We enable insurers and lenders to buy and sell agricultural risks through the futures market. Also, our other products help farmers and seed and fertilizer companies provide customized genotype recommendations and fertilization requirements. This helps to optimize planting by geography, resources, and crop species, monitor phenological, pests and humidity evolution throughout the crop season, and estimate yields.
Local communities benefit from S4’s technology, as the ability to manage weather risks allows farmers to stabilize their cash flows, invest more to produce more with fewer risks, and develop in a more sustainable manner.
Growing data sources, reducing costs, accelerating performance
With the volume of diverse data sources and analytical complexity both growing at a very fast pace, we decided that using a major cloud services provider with a broad roadmap and global partnerships would be beneficial to S4’s future evolution.
At the same time, we wanted to bring our services to users faster and cut costs by consolidating our on-premises technology stack. When we started evaluating providers, our leading criteria included a powerful geospatial database and data analytics tools along with excellent support, all at a competitive price. GCP prevailed in nearly all criteria categories among the 50 companies we measured.
Our previous platform architecture included a hybrid relational database that used Compute Engine for virtual machines and Cloud Storage for database backup. The RDBMS was slow. Maintaining our own data warehouse was complex and expensive.
We wanted to use machine learning and neural networks, but couldn’t do so easily and affordably. The complexity of that system meant that products or services requiring small changes or additions to the data model translated to expensive expansions of infrastructure or project time.
Also, agronomical or product teams couldn’t test these changes by themselves, always requiring the intervention on no small part of the IT team, which led to further delays.
We added GCP services like BigQuery as S4’s cloud data warehouse and use BigQuery GIS for geospatial analysis, Cloud Dataflow for simplified stream and batch data processing, and Cloud SQL for queries to the S4 database platform, which have all made a huge impact on our services and bottom line.
Database and analytics costs have decreased by 40% and customers are receiving our analytical results 25% faster. In addition, we’ve eliminated the time-consuming downloading of images, reducing storage and processing costs by 80%, because we no longer need expensive tools licenses, and have greatly reduced classification processing times.
Our customers working in the agriculture industry are also benefiting from this infrastructure change. They are now able to speed up their data analytics using our GCP-based platform.
“S4 products and technologies unlock the full potential of satellite imagery for crop prescriptions, monitoring and yield estimates,” says Nicolás Loria, Manager of Marketing Services, Southern Cone, Corteva Agriscience.
“We’ve worked with S4 for the last three (and starting year number four) crop seasons as its team capabilities, data integration capacities, and analytics insights have allowed Corteva to perform an entire new solution. Thanks to S4’s customized 360° approach, fast response and delivery times, we have safely outsourced our remote crop analytic technical needs.”
Also, this new architecture has allowed us to scale our models and databases with almost no limits, at a fraction of the cost vs. the previous models.
We’ve saved a lot of time on executing processes and reduced work needed by our internal teams to do certain tasks, like preparing images, converting them, validating results, and more. Using Google Earth Engine has decreased the execution time of daily tasks anywhere from 50% to 90% of the previous time, going from an average time of 30 minutes to between four and 15 minutes, depending on the task.
In addition to saving money and time, we are able to focus on innovation with the GCP performance and features we’re using. We’re able to seamlessly add satellite data to analytics using both public datasets and our own private data, and deliver GIS data management, analytics, crop classification and monitoring in real time.
We can do semi-automatic crop classification and classification using spectral signatures with Google Earth Engine. Later this year, we’ll be using neural networks for pattern recognition and machine learning in new applications to improve crop yields and fine-tune risk models. And using GCP and Google Earth Engine infrastructure means we can run models for customers in South America and around the world, since Google Earth Engine has global satellite imagery available.
We’ve heard from our customer Indigo Argentina that they’re able to bring customers data insights faster.
“We are working with S4 in the development of two different applications for satellite crop monitoring and yield assessment,” says Carlos Becco, CEO, Indigo Argentina. “S4’s technology allowed us to manage and analyze multiple sources and layers of information in real time, letting us uncover valuable insights in Indigo’s own microbiome technologies, and at a very competitive cost.”
Analytical products and app development thrive with GCP
With GCP, we are updating and improving algorithms that we built manually with machine learning processes to develop drought indices for upcoming crop seasons. Algorithms can recognize specific phases of crop phenology (e.g., bud burst, flowering, fruiting, leaf fall) and correlate them with photosynthetic activity, light, water, temperature, radiation, and plant genetics factors. Other analytical products like crop monitoring, pre-planting recommendations, financial scoring, and yield estimation can now do a lot more for users by offering multiple layers and datasets, faster image processing, and real-time access via APIs.
We also replaced our bare-metal S4 app deployment with the App Engine serverless application platform. It provides tighter integration between the S4 platform and our BigQuery data warehouse for integration with marketplaces and third-party solutions.
We get all of these Google Cloud features with all the benefits of managed cloud services, from multiversioning and security to automatic backups and high availability.
At S4, we trust technology to decode plant growth and help protect farmers and their communities from climate change. With growing food shortages due to increasing populations and intensifying weather, data and analytics can have a huge impact in lowering financial risks and improving agricultural yields. It’s one sector where cloud, database, analytics, and other technologies are combining to improve business outcomes and affect the lives of billions of people. Learn more about S4’s work and learn more about data analytics on Google Cloud.
Learn to Access Process Metrics for Full-visibility into Software and Infrastructure behind Your Apps
3404
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
When you are experiencing an issue with your application or service, having deep visibility into both the infrastructure and the software powering your apps and services is critical. Most monitoring services provide insights at the Virtual Machine (VM) level, but few go further. To get a full picture of the state of your application or service, you need to know what processes are running on your infrastructure. That visibility into the processes running on your VMs is provided out of the box by the new Ops Agent and made available by default in Cloud Monitoring. Today we will cover how to access process metrics and why you should start monitoring them.
Better visibility with process metrics
The data gathered by process metrics include CPU, memory, I/O, number of threads, and more, for any running processes and services on your VMs. When the Ops Agent or the Cloud Monitoring agent is installed, these metrics are captured at 60-second intervals and sent to Cloud Monitoring so you can visualize, analyze, track, and alert on them. A single VM may run tens or hundreds of processes, while you may have tens of thousands running across your fleet of VMs.
As a developer, you may only care about seeing inside a single VM to troubleshoot and identify memory leaks or the source of performance issues.
As an operator or IT Admin, you may be interested in aggregate resource consumption, building baseline views of compute, storage, and networking usage across your VM fleet. Then, when those baseline consumption levels break normal behaviors, you will know when to investigate your systems.
Built for scale and ease of use
Cloud Monitoring is built on the same advanced backend that powers metrics across Google. This proven scalability means your metrics ingestion will be supported despite the extremely high cardinality. Additionally, our agents do not require any config file changes to turn on process metric monitoring.
Lastly, our goal is to provide you the observability and telemetry data where, and when, you need it. So, like the rest of the operations suite, we deliver process metrics in the context of your infrastructure, directly in the VM admin console.
The navigation is simple. Once you have the Ops Agent or the Cloud Monitoring agent installed in your VMs:
- Go to the Compute Engine console page and click on VM Instances
- Select the VM that you want to investigate
- In the navigation menu on the top, click Observability
- Click on Metrics
- Lastly, click on Processes
In the window on the right you will see a chart and a table with all of the processes in your VM. You can also filter by time frame and sort by name or value. You do not need to do anything, other than have the agent installed, for the process to be detected and displayed.
Fleet-wide metrics monitoring
Cloud Monitoring gives you a look across your fleet of VMs so you can identify the aggregated usage of resources by processes. This level of broad, yet granular, insight can drive your decisions around which software to run or how many VMs you need to optimally power your apps and services. Admins can perform a cost-savings analysis if they determine that certain processes are slowing down the work of a large number of VMs. The larger numbers of less powerful VMs can be replaced by fewer, more capable VMs.
To get this fleet-wide view:
- Navigate to Cloud Monitoring
- Click Dashboards in the left menu
- In the All Dashboards list, click on VM Instances
- Towards the top of the window, click on Processes
This provides many charts detailing the processes running across your fleet of VMs.
Get started today
To start identifying and monitoring your process metrics, you must first install the Ops Agent, or have installed the legacy Cloud Monitoring agent. Once that is complete, the process metrics data will automatically be ingested into Cloud Monitoring and the VM admin console.
If you have any questions, or to join the conversation with other developers, operators, DevOps, and SREs, visit the Cloud Operations page in the Google Cloud Community.
Finding Your Favorite Google Cloud Product is Now Easy!
3326
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Welcome to a new way of exploring Google Cloud products. Finding your favorite products and discovering new ones requires a user interface that’s easy-to-use, clear, informative, and delightful. Google Cloud users have primarily used our side menu to navigate, but with almost one hundred products and growing, it’s safe to say our product list has outgrown the side menu. Over time, we have listened long and hard to feedback from Google Cloud users, who have highlighted challenges navigating the console to explore our products. We’ve heard over and over that it’s difficult and time consuming to scroll through the long list of products and remember what each one offers.
That’s why we created a new All products page to help you easily navigate to your favorite Google Cloud products. This page showcases all of the Google Cloud products as well as our key partner products in one, easy-to-navigate place. With one click, you can discover the right product that is right for your solution.
Explore all Google Cloud products
The page is organized into different categories, including Management (ie. IAM, Billing), Compute, Storage, Operations, Security, CI/CD, Artificial Intelligence, Support, and more. Quickly jump to the category of interest through the panel on the left, or you can scroll the entire page. Then you can click each product name to navigate directly to the product homepage. Each product listing also includes a short and long description so you can quickly understand what a product does and whether it fits your needs. This lets you compare categories at-a-glance, saving you the hassle of digging up product overviews elsewhere.
Under each product, you’ll also find a link to documentation and Quickstarts so you can understand it in more depth and try it out right away, removing the extra step of navigating to documentation in another tab.
Customize your navigation
To make navigation even easier, you can pin products directly from the All products page, and they will show up at the top of your side menu. You can also customize your navigation by reordering your pins in the side menu. That way, you can quickly access your most-used products directly from the side menu instead of scrolling through the panel or the All products page.
Save time and get more done faster
With the new All products page you can save time scrolling and cut straight to the good stuff – finding your products, discovering new ones, learning, and getting hands on. Try it out for yourself by heading to the Google Cloud Console. Click the side panel and click “View All products,” or on the home dashboard you’ll see a call out to try out the All products page.
If you have any feedback about this new experience, I want to hear! Reach out to me on Twitter at @stephr_wong or on Linkedin at stephrwong.
3476
Of your peers have already watched this video.
3:30 Minutes
The most insightful time you'll spend today!
Rightmove’s Right Move to Google Cloud
In 2020, Rightmove, UK’s renowned property website app saw over a billion minutes from users and clocked about 100 busy days in 2021. To continue innovating their products and improve customer experiences while achieving sustainability goals, Rightmove chose Google Cloud to migrate their infrastructure!
The property search application platform already boasts of dedicated tech teams running heavily code-driven multi-data center infrastructure and high velocity CI/CD platform. To reduce time to product and time to market, Rightmove selects Google Cloud. Watch further to learn how Google Cloud Products helped them add new features, update their existing services and adhere to sustainability objectives.
Google Cloud Garners Highest Score in Forrester New Wave for Computer Vision Platforms
DOWNLOAD WHITEPAPER5345
Of your peers have already downloaded this article
8:30 Minutes
The most insightful time you'll spend today!
In Forrester’s evaluation of the emerging market for computer vision platforms, it identified the 11 most significant providers in the category — Amazon Web Services, Chooch AI, Clarifai, Deepomatic, EdgeVerve, Google, Hive, IBM, Microsoft, Neurala, and SAS — and evaluated them.
Its report details its findings about how well each vendor scored against 10 criteria and where they stand in relation to each other.
Google Cloud was classified as “differentiated” (the highest class) across all 10 criteria.
Find out more. Download The Forrester New Wave™: Computer Vision Platforms, Q4 2019.
More Relevant Stories for Your Company
Accelerating business growth with a seamless migration to Google Cloud
In today’s hybrid office environments, it can be difficult to know where your most valuable, sensitive content is, who’s accessing it, and how people are using it. That’s why Egnyte focuses on making it simple for IT teams to manage and control a full spectrum of content risks, from accidental
Turning the Tide: How PrestaShop Regained Trust in Data
Since 2007, PrestaShop has helped companies unlock the power of e-commerce through its open-source platform. Over 300,000 merchants worldwide use the PrestaShop platform to grow their business and serve online shoppers. "Our open-source strategy to ecommerce enablement sets us apart," says Rémi Paulin, Ph.D., Data Architect at PrestaShop. "Customization is
IT Team Figures Out Easiest Way to Build Data Pipelines and Create ML Models
Building a strong brand in today's hyper-competitive business environment takes vision. It also requires a flexible, easily managed approach to digital asset management (DAM), so marketing professionals and other stakeholders can easily share, store, track, and manipulate assets to build the brand. Many of today's leading companies, including JetBlue, Slack,
Building Unique Customer Experiences with Speed & Scale: Sprinklr & Google Cloud
Enterprises are increasingly seeking out technologies that help them create unique experiences for customers with speed and at scale. At the same time, customers want flexibility when deciding where to manage their enterprise data, particularly when it comes to business-critical applications. That’s why I’m thrilled that Sprinklr, the unified customer
