Kubernetes on Oxide: How Customer Needs Shaped Our Integrations

In late 2024, customers and prospects were eager to run Kubernetes on Oxide, but we had no supported integrations to help them do it. Kubernetes and Oxide are a natural fit. Kubernetes defines the infrastructure behavior it expects through standard extension points, while Oxide exposes the primitives needed to implement that behavior through APIs. The foundation for integration was there. What was missing was the software and an understanding of which integrations customers actually needed. That was the situation when I joined Oxide as its first Solutions Software Engineer,[1] focused on building software to solve customer problems. My first assignment was to make it easier to deploy and operate Kubernetes on Oxide. In my first week, I was handed two resources to help me get started: A customer-submitted pull request for a Rancher node driver An early draft of RFD 493 Initial Kubernetes Integrations What began with those two resources grew into a team effort shaped by a feedback loop. Rather than design integrations in the abstract, we followed the problems customers encountered as they moved from provisioning clusters to operating workloads. This post follows those problems across the Kubernetes lifecycle rather than in strict chronological order. Different provisioning workflows led us to Rancher, Omni, and Cluster API. Running clusters required infrastructure reconciliation, exposing applications revealed networking gaps, and stateful workloads exposed storage constraints. At each stage, customer workflows exposed the next gap, shaping both the integrations we built and the platform work still ahead. How do I provision a Kubernetes cluster on Oxide? The first gap we tackled was provisioning. Our immediate goal was to unblock the customer who had submitted the Rancher node driver pull request. Working through their use case would also give us firsthand experience creating Kubernetes clusters on Oxide and help us uncover the next problems to solve. No single provisioning approach fit all customers' workflows, so we ended up publishing three integrations. Rancher Node Driver Before we could maintain the customer-submitted integration, we needed to understand the workflow it supported. I had never used Rancher or worked with a node driver, so reviewing the contribution meant learning both. A Rancher node driver is an executable plugin that teaches Rancher how to create and manage virtual machines on a particular infrastructure platform. The Oxide Rancher node driver translates those operations into Oxide API requests. Once installed in Rancher, it lets customers provision Oxide instances as nodes in Rancher-managed Kubernetes clusters. Testing confirmed that the customer’s implementation worked. I merged the pull request, added CI/CD and documentation improvements, and published the initial release. Oxide officially had its first Kubernetes integration—and a customer was already using it successfully in production! If you’re a Rancher shop looking to run Kubernetes on Oxide, see our Rancher guide to get started. Omni Infrastructure Provider Customers expressed interest in using Sidero Labs' Omni to provision Kubernetes clusters running Talos Linux. Omni connects to infrastructure platforms through infrastructure providers, programs that create Talos Linux instances and register them with Omni. With KubeCon North America 2025 a few months away, we saw an opportunity to partner with Sidero Labs to build and showcase an Oxide infrastructure provider for Omni. We had seven weeks to complete it before our Oxide+Sidero event.[2] Building against a second provisioning platform would also test Oxide’s APIs across distinct customer workflows. The integration work uncovered several issues across Omni and Talos Linux. I brought those issues to Sidero Labs in siderolabs/omni#1633, where their team was eager to work with us—a lovely reminder of RFD 68 Partnership as Shared Values. The most memorable issue was siderolabs/talos#11948. Oxide uses a FAT12 filesystem for cloud-init user-data, not ISO 9660, but Talos’s filesystem probe only attempted to read an ISO 9660 superblock from the NoCloud configuration disk. When that read failed, the probe stopped instead of trying other formats such as VFAT or MS-DOS. As a result, Talos never read the Oxide user-data containing the configuration needed to join Omni. The fix would not be released in time for KubeCon, leaving us with a rather funny workaround. The workaround right now is to pad the user-data with comments to increase its size enough that it uses an ISO 9660 superblock. KubeCon arrived and we hosted an Oxide+Sidero event to showcase the Oxide infrastructure provider for Omni. Customers could now use this infrastructure provider to provision Oxide instances running Talos Linux as nodes in Omni-managed Kubernetes clusters. If you’re an Omni or Talos Linux shop looking to run Kubernetes on Oxide, see our Omni guide to get started. Cluster API Provider We knew we wanted to build an infrastructure provider for Kubernetes Cluster API (CAPI) when we first wrote RFD 493 Initial Kubernetes Integrations. Cluster API offered something our first two integrations did not—an upstream, provider-extensible API for managing clusters without requiring a third-party platform like Rancher or Omni. CAPI lets operators declaratively create, scale, upgrade, and delete Kubernetes clusters through Kubernetes custom resources. Infrastructure providers handle the platform-specific work, such as creating and deleting virtual machines. Building one is a significant investment. At the time, customer demand and engineering capacity did not yet justify that investment, so the project was deferred. Eventually, both changed. Customers began asking for a CAPI provider, and the Solutions Software Engineering team grew. My teammates Josh and Brandon took ownership of the work and released Cluster API Provider Oxide (CAPOx), giving customers a Kubernetes-native way to provision clusters on Oxide. The Cluster API workflow also exercises several of our other integrations, allowing us to dogfood[3] the end-to-end cluster workflow. The Kubernetes Image Builder uses our Packer plugin to create CAPI-ready Oxide VM images, which CAPOx uses when provisioning instances. Clusters provisioned with CAPOx also use the separately installed Oxide cloud controller manager (CCM) to integrate Kubernetes with Oxide at runtime. If you want to provision Kubernetes clusters on Oxide with Cluster API, see our Cluster API guide to get started. How does Kubernetes track Oxide instances? Provisioning integrations create and manage Oxide instances, but they do not reconcile those instances with Kubernetes Node objects. Without that reconciliation, a cluster could not reliably determine whether an unreachable Kubernetes node was temporarily unavailable or whether its backing Oxide instance had been deleted. We needed a component that ran in each cluster, spoke to the Oxide API, and continuously reconciled Oxide infrastructure with Kubernetes state. Kubernetes provides a standard extension point for this purpose: the cloud controller manager (CCM). A CCM lets infrastructure-specific controllers integrate Kubernetes resources with an infrastructure provider’s API without adding provider-specific code to Kubernetes itself. We built the Oxide cloud controller manager to connect Kubernetes with Oxide. Its node controller keeps Kubernetes Node objects synchronized with their backing Oxide instances, recording details such as instance IDs and network addresses, and reporting whether each instance is running, shut down, or no longer exists. Kubernetes uses this information to initialize nodes and safely remove them when their backing instances are deleted. The CCM does not create instances or provision clusters. That remains the job of provisioning integrations such as the Rancher node driver, the Omni infrastructure provider, and CAPOx. Instead, it provides a runtime integration shared across those provisioning workflows. Importantly, building the CCM gave us a durable extension point inside each cluster. As Oxide evolves, we can add new infrastructure-aware controllers to the CCM rather than update every provisioning integration. With that runtime extension point in place, we could address another layer of the Kubernetes experience: exposing applications. The CCM architecture also defines a service controller for Kubernetes LoadBalancer services, giving us a place to address the next customer problem. How do I use LoadBalancer services? One of the capabilities customers expect from cloud-integrated Kubernetes is support for Service objects of type LoadBalancer . When a user creates one, Kubernetes asks the cloud provider’s service controller to provision the necessary infrastructure and publish its address in the Service status. There was just one problem: Oxide did not yet offer a native load balancer. Oxide did, however, have floating IPs. Floating IPs are addresses from a rack’s external IP pools that can be attached to and detached from instances, making those instances reachable from outside their VPCs. Using floating IPs offered a way to unblock LoadBalancer services. A floating IP would deliver traffic to a single Kubernetes node, and the Kubernetes Service dataplane could distribute that traffic to the appropriate pods. Making that work required accounting for how Oxide floating IPs appear to an instance. They are transparent to the guest in two important ways. First, Oxide translates the destination address of inbound traffic to the instance’s internal IP before sending the traffic to the instance. Second, the instance has no network interface configured with the floating IP. The resulting traffic flow looks like this: LoadBalancer service using floating IPs.┌────────────────────────────────────────────────────────────┐ │ Client │ │ Request to floating IP: 45.154.216.233:80 │ └────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ Oxide networking │ │ Translates destination to internal IP: 172.30.0.5:80 │ └────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ Kubernetes node │ │ Packet arrives at internal IP: 172.30.0.5:80 │ └────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ Kubernetes Service dataplane │ │ Selects a Service endpoint │ └────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ Pod │ │ Receives traffic on its target port │ └────────────────────────────────────────────────────────────┘ That address translation created a subtle integration problem. The Kubernetes Service dataplane needed to treat the node’s internal IP as a Service frontend because that was the destination address packets actually carried when they reached the guest. The service controller therefore publishes two entries in status.loadBalancer.ingress :[4] The attached floating IP in Proxy modeThe node’s internal IP in VIP mode The status entries look like this: status: loadBalancer: ingress: - ip: 45.154.216.233 ipMode: Proxy - ip: 172.30.0.5 ipMode: VIP As a result, the kubectl output looks a little unusual: $ kubectl get service nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx LoadBalancer 10.106.122.233 45.154.216.233,172.30.0.5 80:30605/TCP 37h Users see both the floating IP and the node’s internal IP in the EXTERNAL-IP column, even though only the floating IP is externally reachable. This is an imperfect abstraction, but it allows us to support a common Kubernetes workflow while waiting for a native Oxide load balancer. This implementation currently supports externalTrafficPolicy: Cluster ,[5] which allows the selected node to forward traffic to a Service endpoint anywhere in the cluster. If that node disappears, the CCM moves the floating IP to another eligible node and updates the internal address in the Service status. When Oxide introduces a native load-balancing service, we can update the service controller to use it without changing the Kubernetes interface. Customers will continue creating the same LoadBalancer services and only the infrastructure behind them will change. To install the Oxide CCM on your cluster, see our CCM guide to get started. How do I use Oxide storage in Kubernetes? With clusters provisioned, reconciled with Oxide, and reachable from outside their VPCs, storage for stateful workloads became the next layer to address. Kubernetes users request persistent storage through PersistentVolumeClaim objects and expect a Container Storage Interface (CSI) driver to create, attach, and mount the underlying volumes. Oxide had disks, but Kubernetes had no native way to manage their lifecycle. Without an Oxide CSI driver, customers could deploy a third-party Kubernetes storage system such as Longhorn. Longhorn provides its own CSI driver and replicates data across disks attached to Kubernetes workers. However, using Longhorn meant backing its replicas with Oxide distributed disks, which already store three replicas on distinct sleds. Layering one replicated storage system on another can create substantial write fan-out. When a three-replica Longhorn volume is backed by three-way-replicated Oxide distributed disks, one application write can fan out to as many as nine disk writes. The exact physical write amplification depends on the workload and configuration, but customers wanted to avoid that duplicated replication. The introduction of Oxide local disks provided a way to remove the second layer of replication. Local disks have no built-in replication and remain tied to their sled, making them well suited to systems such as Longhorn that replicate data across Kubernetes nodes. Our Rancher showcase uses this approach today. It avoids stacking two replicated storage systems, though Longhorn still manages the storage lifecycle rather than a native Oxide integration. For a native integration, my teammate Luiz wrote RFD 595 Oxide CSI Plugin. The workflow seemed straightforward on paper. When a user creates a PersistentVolumeClaim , the CSI controller creates an Oxide distributed disk. After Kubernetes schedules the pod, the controller attaches that disk to the selected Oxide instance, and the CSI node plugin formats and mounts it for the pod. If the pod is rescheduled onto another node, the controller detaches the disk and reattaches it to the new node. Prototyping that workflow immediately exposed a blocker. Oxide requires an instance to be stopped before attaching or detaching a disk. Kubernetes, however, expects a CSI driver to attach storage to a running worker after scheduling a pod. Stopping the worker would disrupt every other workload on the node and could trigger cascading scheduling and attachment operations. Before we can release our CSI plugin, we need to add support for disk hot-plug throughout the Oxide stack, from the hypervisor all the way up to the API. What began as a Kubernetes integration has turned into a project spanning multiple layers of the Oxide software stack. Disk hot-plug and the Oxide CSI plugin remain under active development as of this writing. In the meantime, customers can use software such as Longhorn with Oxide local disks for dynamically provisioned persistent storage without stacking two layers of replication. When the native CSI plugin ships, customers will be able to use familiar Kubernetes storage APIs backed directly by Oxide distributed disks with replication and durability built in. What’s next? The result is not a single Kubernetes integration but a growing ecosystem. Rancher, Omni, and Cluster API provide different paths for provisioning, while the Oxide CCM provides a shared runtime integration for node reconciliation and LoadBalancer services. Customers already use some of these integrations in production, and we dogfood several in our own production workloads. Together, they provide a solid foundation to build on. Our next step is to expand our dogfooding with the newly released Cluster API provider. Using it to provision and operate more of our clusters will test how these integrations work together day to day. We still have plenty to build and polish. Our near-term work includes completing disk hot-plug and shipping the CSI plugin, adding autoscaling support, and extending the CCM service controller to support external subnets. Longer term, as we ship resource tagging, OIDC support, and native load balancing, we’ll extend our Kubernetes integrations to take advantage of them. Building these integrations showed how the architectures of Kubernetes and Oxide complement one another. Kubernetes gives infrastructure providers standard extension points, while Oxide exposes infrastructure primitives through APIs. Oxide’s hardware and software co-design lets us address integration blockers at the layer where they belong and carry the necessary changes through the full stack. This work also lets us exercise our SDKs and APIs from our customers' perspectives and turn customer friction into product improvements. That feedback loop is how we will continue growing this ecosystem. Customer needs shaped each integration in this post, and they will shape the next one, too. See it in action To see the Cluster API and cloud controller manager integrations in action, watch the video below, in which I deploy a Kubernetes cluster on Oxide.