Virtual Kubernetes Clusters

Virtual clusters are a new, advanced approach to implementing multi-tenancy for Kubernetes. Users can divide a single physical Kubernetes cluster into multiple logical (virtual) clusters, enabling more functionality than namespace-based partitioning.

A virtual cluster is an abstraction layer on top of a physical or host Kubernetes cluster. The virtual cluster will have its own control plane (like an API server and etcd storage) in which users deploy objects directly. The new control plane will provide users with a complete environment capable of running any workloads compatible with physical Kubernetes clusters, with few exceptions.

There are many benefits to implementing virtual clusters. The virtual cluster environment's isolation significantly increases security by sandboxing the objects deployed to it and enforcing network and resource utilization restrictions. The simplicity and speed of creating virtual clusters are much better than when launching physical clusters while consuming minimal resources to run the virtual cluster control plane. Running many virtual clusters within the same physical cluster instead of managing additional physical clusters leads to a cost optimization benefit while reducing operational overhead for administrators.

For multi-tenant environments, partitioning a physical cluster into multiple virtual clusters provides essential benefits and enables advanced use cases that aren’t possible with traditional Kubernetes clusters. This article will help you understand how virtual clusters work, their benefits, and how to get started with them.

Summary of key concepts related to virtual Kubernetes clusters

What are virtual clusters?

Virtual clusters are real Kubernetes-certified distributions deployed within an existing namespace of a physical Kubernetes cluster. The physical Kubernetes cluster will run any number of virtual clusters as long as its worker nodes have available compute capacity.

To see how virtual clusters change the way we use Kubernetes clusters, we can look at how virtual machines (VMs) changed how we use physical server hardware. VMs enabled system administrators to divide physical hardware into smaller units that are easier to create and manage. The VM environment is also considered secure because changes to the VM’s operating system don’t impact the underlying physical server or other VMs on the same host. This lets system administrators maintain a strong security posture while providing a good user experience for common use cases requiring flexible operating system customization. Virtual clusters implement a similar principle of isolation from the host cluster while allowing users to deploy any objects necessary for their use cases without disrupting other users on the same physical cluster.

Virtual clusters are also faster to create and delete than physical clusters. On most cloud providers, Kubernetes clusters can take 10 to 20 minutes to create. In contrast, virtual clusters can be created within just 60 seconds, making them ideal for time-sensitive use cases like creating ephemeral test environments.

The virtual cluster comprises an operator resource acting as a Kubernetes control plane. The operator will configure a new API server endpoint for users to connect to, allowing users to interact with the virtual cluster API server in the same manner that one would interact with the physical Kubernetes API server. The virtual cluster operator will store newly created Kubernetes object data (like deployments and daemonsets) just like a physical Kubernetes cluster. The difference is that creating objects in the virtual cluster will be contained and isolated from the rest of the cluster. For example, users can create role-based access control (RBAC) objects within the virtual cluster, which will not impact the physical cluster’s RBAC objects. This gives users fine-grained control over their virtual cluster environments while maintaining security in the physical host cluster.

The newly created RBAC objects will not impact the physical cluster since these objects are only visible from inside the virtual cluster. This is a significant development compared to the default Kubernetes behavior, which is that all cluster-wide objects—like RBAC, namespaces, and custom resource definitions (CRDs) impact all cluster users. Virtual clusters enable an approach for users to isolate cluster-wide resources into the virtual cluster only.

Objects like deployments, daemonsets, and stateful sets are also isolated. A virtual cluster can isolate these resources via tools like resource quotas and network policies to protect other workloads on the physical cluster.

Virtual clusters are a significant advancement for scaling dev and test environments in comparison to traditional Kubernetes clusters. They enable significant improvements in environment management and enhance an organization’s overall development velocity.

What are the use cases for virtual clusters?

Several use cases make virtual clusters a key technology for advanced Kubernetes users and platform teams. Virtual clusters enable users to extend the default capabilities of Kubernetes to access more advanced use cases and improve the developer experience.

Enhanced security

The security posture of virtual cluster environments is significantly more robust than namespace multi-tenancy. Virtual clusters improve security with features like a dedicated control plane, restricted network access by default, and isolated objects by default. Implementing security with virtual clusters will increase isolation beyond traditional Kubernetes features like namespaces and RBAC by providing additional layers of protection enforced by default. This makes virtual clusters a valuable option for high-security workloads.

{{banner-1="/utility-pages/banners"}}

Faster dev and test cycles

Virtual clusters can be deployed in under 60 seconds—significantly faster than creating a physical Kubernetes cluster. This velocity allows users to more quickly launch cluster resources to run their workloads for time-sensitive use cases. For example, using virtual clusters as an ephemeral test environment to validate Kubernetes workloads will require easily deployable clusters to avoid bottlenecking developer productivity. Relying on physical clusters will take too long—at least 10-20 minutes—so virtual clusters fit time-sensitive requirements more effectively.

Another example of a time-sensitive use case is disaster recovery: Implementing failover from a broken cluster to a functioning one is more feasible with virtual clusters due to the faster launch time. Failover time is critical in disaster recovery scenarios, and virtual clusters accelerate the pace at which cluster infrastructure can be replaced. This is similar to how a broken VM environment may be mitigated by replacing it with a fresh VM.

Accessing new Kubernetes versions

New Kubernetes versions are released frequently by upstream maintainers, and users need access to clusters running the upgraded version to validate that their workloads are compatible with new changes (like API deprecations). Virtual clusters are an easy way to set up a cluster with a more recent target Kubernetes version because virtual clusters can be launched running Kubernetes versions newer than the host physical cluster version. This enables users to use a virtual cluster to validate workload compatibility with newer Kubernetes versions before upgrading the physical cluster and without the operational overhead of launching new physical clusters. Considering how quickly virtual clusters can be deployed, this is an enormously useful way to deploy and validate new Kubernetes versions rapidly.

Simplified multi-tenancy (aka access with guardrails)

Managing multi-tenant environments may be easier when using virtual clusters. The isolated nature of objects deployed to virtual clusters means a reduced blast radius when users misconfigure their resources and also reduces the attack surface if a workload is compromised. Onboarding new users into the multi-tenant environment is fast due to the rapid launch time of virtual clusters, and cleaning up an unused cluster is done in seconds without the complexity of deleting physical cluster infrastructure.

How do virtual clusters improve security?

Improved security is a critical feature of virtual clusters, setting them apart from regular Kubernetes distributions. Raising the bar for security is priority zero for most organizations, and virtual clusters help with this goal by implementing several features that enhance the security posture of multi-tenant environments.

Dedicated control planes

The virtual cluster has its own control plane, including its own API endpoint and API data storage. This means users are isolated from accessing the physical Kubernetes API by only being granted credentials to the virtual cluster API. Users only have credentials to access their virtual clusters, ensuring that they can only deploy and modify workloads in their virtual clusters. The physical API server is isolated from the user, protecting it from unnecessary and unwanted access.

Any data submitted to the virtual cluster’s API will be recorded in an isolated data store, separate from the physical control plane’s etcd database. Data storage between virtual clusters is not shared, maintaining the isolation of sensitive information. These measures allow a substantial degree of isolation between the virtual and physical clusters, reducing the attack surface and preventing accidental misconfigurations.

Network isolation by default

By default, virtual clusters do not allow communication to workloads outside the cluster. This can be enforced via network policies and by disabling DNS resolution for services outside the cluster.

Disabling access to unnecessary traffic is a cornerstone of maintaining network security because a compromised pod could lead to a compromised network. Enforcing network security by default with virtual clusters significantly improves on default Kubernetes behavior, which allows all traffic between services across the cluster without restrictions.

Object isolation by default

Virtual clusters improve security for users by enforcing strong isolation between workloads on the same physical cluster. Objects like CRDs are restricted and visible only to the virtual cluster environment.

By default, Kubernetes allows cluster-wide objects like CRDs to impact the entire cluster, posing a security risk when compromised or misconfigured. By using virtual clusters to isolate cluster-wide objects, we can ensure that users cannot affect other workloads on the same physical cluster. This improves security while still allowing the user to benefit from implementing these useful object types. Namespaces are not an effective tool for isolating cluster-wide objects; virtual clusters fill this gap.

Resource quotas by default

Making sure that resource quota objects are applied by default will help ensure that proper constraints are applied to tenant workloads. The chance of human error is reduced when this functionality is available by default. This differs from regular Kubernetes clusters, where additional steps must be taken to create and configure resource quota controls manually.

Managing resource consumption is essential to maintaining security because attacks like DDOS rely on resource exhaustion. Restricting a user’s resource access by default with virtual clusters helps reduce the blast radius of resource exhaustion issues and prevents “noisy neighbor” issues from impacting other tenants on the same physical cluster.

How do virtual clusters improve scalability?

Virtual clusters can improve scalability in several ways. The overhead of running the lightweight virtual cluster operator is minimal, so there is a negligible resource cost in enabling the additional scalability features.

Less overhead than dedicated physical clusters

Virtual clusters are faster and more resource-efficient to launch than physical clusters. Scaling cluster infrastructure to handle dynamic workloads is easier in a virtual cluster environment than modifying a physical cluster. For example, resource allocation, quotas, and compute capacity can be easily modified by adjusting the virtual cluster configuration. In a multi-tenant environment, it’s helpful to change the resources allocated to a tenant without making physical changes to the underlying cluster infrastructure.

If new clusters must be created to support new workloads, virtual clusters’ under-60-second launch time is significantly faster than launching dedicated cluster infrastructure. Virtual clusters offer a far more scalable way to deploy and access a full Kubernetes cluster than creating physical clusters and, as mentioned earlier, are ideal for time-sensitive use cases.

Reduced pressure on the physical API server

Almost all user API calls will be received by the virtual cluster control plane instead of the physical control plane. This results in less pressure on the physical control plane by delegating to the individual virtual clusters to process the majority of requests.

This is a great benefit for users of high-volume clusters. Typically, cloud providers significantly delay the scaling of control plane nodes in managed Kubernetes environments, so reducing the load on the control plane by balancing requests to the virtual clusters will help avoid control plane scaling bottlenecks.

Resource quotas improve scalability

Enforcing resource quotas by default helps control resource utilization, particularly in multi-tenant environments. Kubernetes does not implement resource quotas by default, allowing any workload to consume unlimited resources. Virtual clusters improve on this behavior by ensuring that resource quotas are configured for every virtual cluster, allowing users only to access a limited set of resources (like CPU, memory, and disk space). This prevents resource exhaustion in the physical cluster and ensures that workloads in all virtual clusters can scale effectively within their quotas without worrying about noisy neighbors overconsuming resources. Ensuring that resources are reasonably available for scaling is essential in multi-tenant environments.

Isolating cluster-wide objects

Cluster-wide objects like CRDs and RBAC resources can be hard to manage in traditional multi-tenant clusters because the objects from all tenants will overlap and be present in the physical cluster’s control plane. Virtual clusters provide a neater and more scalable setup by ensuring that cluster-wide objects are only deployed and visible to individual virtual clusters, avoiding conflicting overlaps if multiple users deploy the same object type. This reduces the operational overhead for administrators who need to maintain, upgrade, and clean up the cluster-wide objects in the physical cluster. Limiting the object’s presence to the virtual cluster improves scalability by allowing any number of users to utilize cluster-wide objects without adversely impacting other tenants or the physical host cluster.

How do virtual clusters help with cost optimization?

Virtual clusters can provide significant cost optimization benefits compared to deploying dedicated physical cluster infrastructure. The fully fledged API server managed by the virtual cluster can eliminate the need for excessive physical clusters, significantly reducing operational overhead, costs, and management complexity.

The lightweight footprint of compute resources consumed by virtual clusters will cost a fraction of what is required to run dedicated physical control planes on a managed cloud provider, which will typically charge for control planes regardless of utilization level. Implementing new control planes via virtual clusters reduces the need to spend on cloud provider clusters.

Partitioning a physical cluster into multiple virtual clusters allows administrators to share tooling infrastructure effectively, such as the observability stack. A physical cluster with any number of virtual clusters can be monitored via a single observability setup, reducing resource duplication and engineering time spent on managing multiple disparate configurations. Observability tools like Prometheus, Grafana, FluentBit, and others can simply run on the host physical cluster and collect telemetry data for all virtual clusters, simplifying the administrator’s responsibilities. Deploying separate physical clusters will require duplicating the observability stack, consuming additional compute resources, and increasing the operational overhead. Sharing other cluster tools—like security, auditing, and service meshes—is also more manageable in a multi-tenant environment that leverages virtual clusters, since every virtual cluster can benefit from the shared tools running on the host cluster.

Another aspect of how virtual clusters can reduce costs is by more effectively sharing compute resources. Virtual clusters running on the same physical host cluster can easily share compute resources, reducing unused worker node capacity. Deploying separate physical clusters will mean compute capacity cannot be shared, and spare capacity in one cluster will not be accessible by workloads in other clusters. Given that rightsizing compute capacity is a challenging problem, sharing resources between workloads in a virtual cluster environment is key to minimizing wasted compute capacity and cost optimization.

Overall, virtual clusters can provide a significant cost benefit for users compared to deploying dedicated physical clusters. Leveraging virtual clusters allows users to more effectively share observability tools, share unused compute resources to minimize waste, and reduce costs on managed cloud provider clusters.

How can virtual clusters be created?

Kubernetes does not provide any native virtual cluster functionality, so users need to leverage other projects to enable this advanced technology in their environments. The Uffizzi Cluster Operator is a project that uniquely provides virtual cluster management capabilities.

The Uffizzi Cluster Operator can be installed on any Kubernetes cluster by following the instructions here. Deploying the project will create a new custom resource type called “UffizziCluster,” which is a virtual cluster object. The new object’s schema is defined by a custom resource definition (CRD) provided by Uffizzi. The schema will define what properties are supplied when new objects of this type are created. Similarly, when a pod object is created, users can supply “spec.containers” values to define their pods.

The CRD for UffizziCluster enables properties that are relevant for creating virtual clusters. For example, the object will accept a “spec.apiServer.image” value to define what container image version of the Kubernetes API Server should be deployed in the virtual cluster, allowing users to customize the Kubernetes version of the virtual cluster.

The CRD for the UffizziCluster object looks like this (some fields are truncated for simplicity; the full CRD can be viewed here):

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: uffizziclusters.uffizzi.com
spec:
  group: uffizzi.com
  names:
	kind: UffizziCluster
	listKind: UffizziClusterList
	plural: uffizziclusters
	shortNames:
	- uc
	singular: uffizzicluster
  scope: Namespaced
  versions:
    schema:
      openAPIV3Schema:
        properties:
      	   spec:
            properties:
          	 apiServer:
            	   properties:
                  image:
                  type: string

There are many more fields in the UffizziCluster CRD, allowing users to supply many kinds of values to customize their virtual cluster objects.

Once the CRD has been created, users can then launch any number of virtual clusters by creating instances of the “UffizziCluster” object type, the same way they create any other Kubernetes objects.

Here is an example of the UffizziCluster object that can be deployed to a cluster that has defined the CRD object schema described above. The object below will create a virtual cluster called “my-first-virtual-cluster” running Kubernetes 1.27 in under 60 seconds:

kind: UffizziCluster
apiVersion: uffizzi.com/v1alpha1
metadata:
  name: my-first-virtual-cluster
spec:
  distro: k8s
  apiServer:
	image: "registry.k8s.io/kube-apiserver:v1.27.3"

In a multi-tenant environment, administrators may deploy many of these UffizziCluster objects to generate virtual clusters for each tenant. Each tenant will then have access to its own secure, strongly isolated cluster with the full flexibility of the Kubernetes API. The administrator will benefit from being able to launch clusters quickly, improve security posture, reduce operational overhead, and reduce costs compared to launching dedicated physical clusters.

{{banner-2="/utility-pages/banners"}}

Conclusion

Virtual clusters are a key technology for modern Kubernetes platforms. They are the modern approach to implementing multi-tenancy in a secure and scalable fashion.

With the importance of security in multi-tenant environments, the strong isolation provided by virtual clusters is a major benefit compared to namespaces. Clusters are sandboxed by default, restricting the blast radius of compromised or misconfigured environments. Adding network connectivity restrictions and deployed objects being partitioned by default adds further layers of security and isolation beyond what Kubernetes namespace objects are capable of.

The scalability of virtual clusters is far beyond what physical clusters offer. The ability to launch new clusters within 60 seconds is beyond the capabilities of any cloud provider’s managed Kubernetes platforms, enabling use cases where clusters need to be quickly launched to meet user demand. The operational overhead of managing virtual clusters is reduced compared to physical clusters, allowing administrators to manage virtual clusters as just another workload running in a cluster instead of managing a separate piece of physical infrastructure.

Using virtual instead of physical clusters also ties into cost benefits: Cloud provider control planes have fixed costs significantly higher than the compute capacity cost required to run lightweight virtual clusters. Resources are also easier to share between virtual clusters on the same physical host, minimizing the amount of unused, wasted resources compared with running multiple physical clusters.