OpenShift Virtualization Training

Footer

Mon, 01 Jan 0001 00:00:00 +0000

Footer

Mon, 01 Jan 0001 00:00:00 +0000

Create a VM

Mon, 01 Jan 0001 00:00:00 +0000

Creating a virtual machine

In OpenShift’s web console in the menu on the left, navigate to Virtualization → Overview. Make sure your project is active:

It’s still pretty empty here and a lot of zeroes look at us. However, we are going to change this.

Task 1.1.1: Create your first virtual machine

Open the VirtualMachines menu entry. Click on Create VirtualMachine and choose From template.

On the left of the view that just opened, click on User templates and choose the Cirros VM template.

Lab environment

Mon, 01 Jan 0001 00:00:00 +0000

The necessary URLs and credentials to access the lab environment will be provided by your trainer. Use Chrome/Chromium for the best experience.

OpenShift’s web console

We are calling OpenShift’s web interface the web console. You access it by opening a browser, open the following URL and log in.

https://console.apps.lab.clusters.t-k.ch/

During the whole training, make sure you’re working on the Virtualization view:

Web terminal

OpenShift’s web terminal is a convenient way to work with, e.g., oc as it doesn’t require you to install anything locally on your computer. It is fully integrated into OpenShift’s web console explained in the chapter above.

KubeVirt configuration

Mon, 01 Jan 0001 00:00:00 +0000

KubeVirt installation

On the Kubernetes cluster used for this training, the following Operators have been installed:

Operator name	Namespace
KubeVirt	kubevirt
Containerized Data Impoter (CDI)	cdi

As the configuration and the required feature flags are subject to change, we do not highlight them in every section.

Emulation

For local development and test environments, the emulation can be turned on. This may have a huge impact to the performance.

apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
 name: kubevirt
 namespace: kubevirt
spec:
 configuration:
 developerConfiguration:
 useEmulation: true
[...]

The Lab cluster is not using emulation.

Start and stop a VM

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section, we wrote a VirtualMachine specification and applied the manifest to the Kubernetes cluster.

Lifecycle

When the underlying technology libvirt refers to a VM it often also uses the concept of so-called guest domains.

According to libvirt.org , a guest domain can be in several states:

Undefined: This is a baseline state. Libvirt does not know anything about domains in this state because the domain hasn’t been defined or created yet.
Defined or Stopped: The domain has been defined but it’s not running. This state is also called stopped. Only persistent domains can be in this state. When a transient domain is stopped or shut down, it ceases to exist.
Running: The domain has been created and started either as transient or persistent domain. Either domain in this state is being actively executed on the node’s hypervisor.
Paused: The domain execution on the hypervisor has been suspended. Its state has been temporarily stored until it is resumed. The domain does not have any knowledge whether it was paused or not.
Saved: Similar to the paused state, but the domain state is stored to persistent storage. Again, the domain in this state can be restored and it does not notice that any time has passed.

Accessing the console

Mon, 01 Jan 0001 00:00:00 +0000

Console access

Make sure you are on your VM’s details page and that it is started.

Now open your VM’s console by clicking on Open web console. A new window opens. Wait for the console to connect.

You can now use the default credentials to log in:

User: cirros
Password: gocubsgo

Try executing a couple of commands, such as:

whoami
ping tim-koko.ch

Task 1.3.2: Entering the console using virtctl

By executing the following command in your openshift terminal, you can also use the virtctl cli to conncet to the virtual machines console:

Tolerations

Mon, 01 Jan 0001 00:00:00 +0000

Necessary tolerations

Our lab cluster uses tainted nodes, therefore some of the to-be-applied manifests will need a toleration. The labs will tell you when to use it, but you can always find it here as an easy reference:

[...]
tolerations:
 - effect: NoSchedule
 key: baremetal
 operator: Equal
 value: "true"
[...]

Exposing VM Ports

Mon, 01 Jan 0001 00:00:00 +0000

In the previous chapter, we accessed our VM console using the virtctl tool. In this section we will expose the SSH port of our VM and access it directly.

Note

This can be done for any port you want to use. For example, if your virtual machine provides a webserver, you can expose the webserver port.

Checking available Services resources

As you see with the following command, creating the VM does not create any Kubernetes Service for it.

Changing files

Mon, 01 Jan 0001 00:00:00 +0000

In the previous chapter, we gained access to the VM’s console. In this chapter, we will use this access to make a change to our running VM and observe what happens if we restart the VM.

Task 1.5.1: Create a file

Connect to your VM’s web console.

touch myfile

Verify that the file really is present. Check it with:

stat myfile

Task 1.5.2: Restart VM

Exit the console and restart your VM.

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

The Containerized Data Importer Project¹ is an independently installed Operator. It is primarily used to provide a declarative way to build Virtual Machine Disks on PVCs for OpenShift Virtualization VMs. The CDI Operator provides the custom resource DataVolume which is an abstraction above the PVC.

CDI Operator components

Note

The Operator has already been installed in namespace cdi. If you want to deploy the Operator, check the Deploy section of the CDI documentation.

You may see the CDI Operator components with:

Create a DataVolume

Mon, 01 Jan 0001 00:00:00 +0000

Provisioning a Virtual Machine Disk

In the following lab, we will provision a disk from a given url. We will use a DataVolume to create the Disk. We then create a VM with the disk attached.

We want to provision a VM running an Alpine Cloud Image¹.

Task 2.2.1: Write your own DataVolume manifest

Create a file dv_lab02-alpinedisk.yaml in folder labs/lab02and start with the content from the previous section:

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
 name: lab02-blankdv
spec:
 source:
 blank: {}
 contentType: "kubevirt"
 storage:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 64Mi

Now adapt your DataVolume to fulfill the following specifications:

Using the Alpine disk

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we have provisioned a custom Alpine cloud disk image. We will now create a VM which uses this disk image.

Task 2.3.1: Attach the disk

Write a new VirtualMachine manifest named lab02-alpine.yaml in the directory labs/lab02.

Note

This Alpine Cloud Image has cloud-init included. We will need another disk (cloudinitdisk) which configures our environment. We will look into cloud-init in a later section. For now, just use the cloudinitdisk specification from the snippet below.

Block needed to reference the PVC as a disk and the cloudinitdisk:

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

Even if OpenShift Virtualization provides advanced options and a lot of configuration options for different VMs, we usually have a common set of VM specifications which we will use for most of our VMs. Therefore, it may make sense to define such specifications as instance types and / or preferences.

To achieve this, OpenShift Virtualization provides multiple CustomResourceDefinitions like VirtualMachineInstancetype, VirtualMachineClusterInstancetype or VirtualMachinePreference.

VM VirtualMachineInstancetype

For an instance type we have the option of using the cluster-wide VirtualMachineClusterInstancetype or the namespaced VirtualMachineInstancetype.

Common instance types and preferences

Mon, 01 Jan 0001 00:00:00 +0000

The OpenShift Virtualization project provides a set of common instance types and preferences . In this lab, we are going to have a look at these and how they can help us create virtual machines.

Common instance types and preferences

By default OpenShift Virtualization comes with a set of common instance types and preferences. You can reference those instance types and preferences when creating virtual machines.

The cluster provided as well as the user provided instance types and preferences can either be displayed in the web console or by cli.

Create an instance type

Mon, 01 Jan 0001 00:00:00 +0000

In this section, we will create and use our own instance type.

Task 3.3.1: Create your own instance type

In the previous section we have seen that the Cirros preference requests 256Mi of memory. However, the smallest instance type available requests 512Mi of memory. In order to adapt to this, let’s create our own instance type and assign it to our VirtualMachines.

Define the following instance type:

Request 256Mi of memory
Reduce the overcommit percentage from 50% to 25%
Name the resource lab03-o1-pico

Task hint

Create the following resource using the import YAML functionality of the OpenShift Web Console:

Re-assign an instance type

Mon, 01 Jan 0001 00:00:00 +0000

In this section we will change the instance type of our running VMs to our newly created instance type.

Changing an instance type

Whenever a VM referencing an instance type or preference is created, the definition at time of creation is stored in a ControllerRevision. This revision is then referenced in a new field .spec.instancetype.revisionName in our VM manifest.

This field ensures that our VirtualMachine knows the original specification even when the type or preference would change. This ensures that there are no accidental changes to the VM resources or preferences.

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

When we create a virtual machine, we often want to configure the virtual machine to fit in our environment. To achieve this, OpenShift Virtualization supports the assignment of startup scripts which are executed automatically when the VM initializes. They are typically used to provide SSH keys, required configuration to run further configuration with ansible, deploy network configuration and so on.

These startup script methods are supported:

Cloud-init¹ and Ignition² which are targeting Linux and Unix systems
Sysprep³ to initialize Windows-based VMs

Cloud-init is the most-widely adopted method, and you will find great support on cloud providers such as AWS, GCP and Azure.

Cloud-init

Mon, 01 Jan 0001 00:00:00 +0000

In this section we will use cloud-init to initialize a Fedora Cloud¹ VM. Cloud-init is the de-facto standard for providing startup scripts to VMs.

Cloud-init is widely adopted. Some of the known users of cloud-init are:

Ubuntu
Arch Linux
CentOS
Red Hat
FreeBSD
Fedora
Gentoo Linux
openSUSE

Supported data sources

OpenShift Virtualization supports the cloudInitNoCloud and cloudInitConfigDrive data source methods.

Note

As it is the simplest data source you should stick to cloudInitNoCloud as the go-to data source. Only if cloudInitNoCloud is not supported by the cloud-init implementation you should switch to cloudInitConfigDrive. For example the implementation of coreos-cloudinit was known to require the cloudInitConfigDrive data source. However, as CoreOS has built Ignition this implementation is superseded but there may be more implementations.

`cloudInitNoCloud` data source

cloudInitNoCloud is a flexible data source to configure an instance locally. It can work without network access but can also fetch configuration from a remote server. The relevant configuration of a cloudInitNoCloud data source in a VM looks like this:

Ignition

Mon, 01 Jan 0001 00:00:00 +0000

In the previous section we created a VM using a cloud-init configuration Secret. This time we will do something similar but with Ignition and Linux Fedora CoreOS.

Known users of Ignition are:

Fedora CoreOS
Red Hat Enterprise Linux CoreOS
Flatcar
openSUSE MicroOS
SUSE Linux Enterprise Micro

Supported data sources

To provide Ignition data we have to use the cloudInitNoCloud data source.

`cloudInitNoCloud` data source

The relevant configuration of a cloudInitNoCloud data source in a VM looks like this:

Sysprep

Mon, 01 Jan 0001 00:00:00 +0000

This lab is not yet implemented. Please check the official documentation for additional information.

VM disk images

Mon, 01 Jan 0001 00:00:00 +0000

If we want to run VMs at scale it makes sense to manage a set of base images to use for these VMs. It is not very convenient to spin up and install some requirements for each single VM oneself. There are several ways we can distribute VM images in our cluster.

Distribute images as ephemeral container disks using a container registry
- Be aware of the non-persistent root disk
- Depending on the disk size, this approach may not be the best choice
Create a namespace (e.g., vm-images) with pre-provisioned PVCs containing base disk images
- Each VM would then use CDI to clone the PVC from the vm-images namespace to the local namespace

At the end of this section, we will have two PVCs containing base disks in our namespace:

Virtual machine pools

Mon, 01 Jan 0001 00:00:00 +0000

A VirtualMachinePool tries to ensure that a specified number of virtual machines are always in a ready state.

However, the virtual machine pool does not maintain any state or provide guarantees about the maximum number of VMs running at any given time. For instance, the pool may initiate new replicas if it detects that some VMs have entered an unknown state, even if those VMs might still be running.

Using a virtual machine pool

Using the custom resource VirtualMachinePool, we can specify a template for our VM. A VirtualMachinePool consists of a VM specification just like a regular VirtualMachine. This specification resides in spec.virtualMachineTemplate.spec. Besides the VM specification, the pool requires some additional metadata like labels to keep track of the VMs in the pool. This metadata resides in spec.virtualMachineTemplate.metadata.

Virtual machine replica sets

Mon, 01 Jan 0001 00:00:00 +0000

Just like a VirtualMachinePool, a

VirtualMachineInstanceReplicaSet resource tries to ensure that a specified number of virtual machines are always in a ready state. The VirtualMachineInstanceReplicaSet is very similar to the Kubernetes ReplicaSet¹.

However, the VirtualMachineInstanceReplicaSet does not maintain any state or provide guarantees about the maximum number of VMs running at any given time. For instance, the VirtualMachineInstanceReplicaSet may initiate new replicas if it detects that some VMs have entered an unknown state, even if those VMs might still be running.

Mounting storage

Mon, 01 Jan 0001 00:00:00 +0000

There are multiple ways of mounting a disk to a virtual machine. In this section we will mount various disks to our VM.

Kubernetes storage

There are multiple methods to provide storage for your virtual machines. For storage to be attached you have to specify a volume in the spec.templates.spec.volumes block of your virtual machine. This volume must then be referenced as a device in spec.templates.spec.domain.devices.disks or spec.templates.spec.domain.devices.filesystems.

A sample configuration of a PersistentVolumeClaim mounted as a disk:

Using StorageProfiles

Mon, 01 Jan 0001 00:00:00 +0000

When working with storage and the containerized data importer, one usually wants to have meaningful defaults. Let us have a look at how we can configure storage profiles to be used with OpenShift Virtualization.

Warning

Due to the cluster-wide configuration of storage classes, the resources and commands in this lab are not meant to be created and executed!

What are storage profiles

For each available StorageClass, OpenShift Virtualization creates a StorageProfile resource. StorageProfiles serve as a source of information about the recommended parameters for a PVC. They are used when provisioning a PVC using a DataVolume. Having recommended parameters defined centrally in a StorageProfile reduces the complexity of your DataVolume definition.

Hotplugging disks

Mon, 01 Jan 0001 00:00:00 +0000

In this section we will hotplug a disk to a running virtual machine.

Task 6.3.1: Starting a virtual machine

First, create a file vm_lab06-cirros.yaml in the folder labs/lab06 and start with the following virtual machine definition:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
 name: lab06-cirros
spec:
 runStrategy: Halted
 template:
 metadata:
 labels:
 kubevirt.io/domain: lab06-volume
 spec:
 domain:
 devices:
 disks:
 - name: rootdisk
 disk:
 bus: virtio
 - name: cloudinitdisk
 disk:
 bus: virtio
 interfaces:
 - name: default
 masquerade: {}
 resources:
 requests:
 memory: 64M
 networks:
 - name: default
 pod: {}
 volumes:
 - name: rootdisk
 containerDisk:
 image: quay.io/kubevirt/cirros-container-disk-demo
 - name: cloudinitdisk
 cloudInitNoCloud:
 userData: |
 #cloud-config

Create and start the virtual machine with:

VM snapshot and restore

Mon, 01 Jan 0001 00:00:00 +0000

OpenShift Virtualization provides a snapshot and restore functionality. This feature is only available if your storage driver supports VolumeSnapshots and a VolumeSnapshotClass is configured.

You can list the available VolumeSnapshotClass with:

oc get volumesnapshotclass --namespace lab-<username>

NAME DRIVER DELETIONPOLICY AGE
longhorn-snapshot-vsc driver.longhorn.io Delete 21d

You can snapshot virtual machines in running or stopped state. Using the QEMU guest agent the snapshot can temporarily freeze your VM to get a consistent backup.

Resizing a Disk

Mon, 01 Jan 0001 00:00:00 +0000

In this section we will resize the root disk of a virtual machine.

Requirements

Resizing disks depends on the Kubernetes storage provider. The CSI driver must support resizing volumes and must be configured with AllowVolumeExpansion.

Further, it may depend on the operating system you use. Whenever the volume is resized, your VM might see the change in disk size immediately. But there might still be the need to resize the partition and filesystem. Fedora Cloud for instance has the package cloud-utils-growpart installed. This rewrites the partition table so that the partition takes up all the space available. This makes it a very handy choice for resizing disk images.

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

The live migration feature (LiveMigration) is enabled by default in recent OpenShift Virtualization versions. Previously it needed to be enabled in the OpenShift Virtualization configuration running on the cluster (oc get kubevirt kubevirt -n kubevirt -o yaml):

apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
 name: kubevirt
 namespace: kubevirt
spec:
 configuration:
 developerConfiguration:
 featureGates:
 - Sidecar
 - CommonInstancetypesDeploymentGate
 - ExperimentalIgnitionSupport
 - HotplugVolumes
 - ExpandDisks
 - Snapshot
 - VMExport
 - BlockVolume

Additionally, the live migration feature can be configured under spec.configuration.migrations with a set of configuration parameters. The full list can be found in the API reference .

Perform a live migration

Mon, 01 Jan 0001 00:00:00 +0000

This lab demonstrates how to perform a live migration of a running virtual machine.

Task 7.2.1: Creating a virtual machine

For this lab we are going to create a Fedora Cloud VM and provide a cloud-init userdata configuration to initialize our VM.

We will install

an nginx Webserver running on port 8080
and configure it to return a predefined response (Hello OpenShift Virtualization) when we send requests to it.

Create the following Secret resource using the import YAML functionality of the OpenShift Web Console:

Guest agent

Mon, 01 Jan 0001 00:00:00 +0000

In many of the available cloud images, the qemu-guest-agent package is already installed. If it’s not preinstalled you can use one of the previously learned concepts to install the package.

Task 8.1.1: Start a virtual machine and explore guest agent information

In this lab we are going to reuse the virtual machine we created in the 4.2. Cloud-init lab.

Start the cloud-init virtual machine using the following command:

virtctl start lab04-cloudinit --namespace lab-<username>

The presence of the guest agent in the virtual machine is indicated by a condition in the VirtualMachineInstance’s status. This condition shows that the guest agent is connected and ready for use.

Readiness and liveness probes

Mon, 01 Jan 0001 00:00:00 +0000

Liveness and readiness probes can be configured for VirtualMachineInstances similarly to how they are defined for containers. You can find more information about these probes in Kubernetes’ documentation .

Liveness probes will stop the VirtualMachineInstance if they fail, allowing higher-level controllers, such as VirtualMachine or VirtualMachineInstanceReplicaSet, to create new instances that should be responsive.

Readiness probes signal to Services and Endpoints whether the VirtualMachineInstance is ready to handle traffic. If these probes fail, the VirtualMachineInstance will be removed from the list of Endpoints backing the Service until the probe recovers.

Prometheus monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It collects real-time metrics from services and systems, stores them in a time-series database, and provides powerful querying capabilities. Prometheus operates with a pull-based model, scraping metrics from endpoints at regular intervals. It supports multi-dimensional data through labels, enabling flexible queries and insights. Paired with tools like Grafana for visualization, Prometheus is widely used for monitoring cloud-native applications, infrastructure, and system performance, with built-in alerting to notify users of potential issues.

Prometheus Node Exporter

Mon, 01 Jan 0001 00:00:00 +0000

The Prometheus Node Exporter is a key component used for collecting operating system metrics from Linux and Windows systems. It exposes a wide range of system-level metrics that Prometheus can scrape, making it useful for monitoring the health and performance of physical and virtual machines.

Some of the key metrics collected by Node Exporter include:

CPU usage: Tracks how much CPU time is being used by user and system processes
Memory usage: Monitors free and used memory, swap space, and buffer/cache utilization
Disk I/O: Provides insights into disk read/write operations and storage usage
Network statistics: Captures metrics on data sent and received over network interfaces
File system usage: Monitors available and used space on file systems

Node Exporter runs as a lightweight daemon on each node and is easy to install and configure. It works out of the box, exposing most common system metrics through the /metrics endpoint, but can also be extended with additional collectors to gather more specialized data. These metrics can be visualized through tools like Grafana, helping administrators monitor and troubleshoot infrastructure performance.

Requirements

Mon, 01 Jan 0001 00:00:00 +0000

For this lab we do not provide a step-by-step guide for the implementation. The goal is that you implement the required manifests yourself. However, we provide a sample solution in the next chapter. Your solution may vary from ours.

Basic Information

The goal of this super lab is to deploy a database and a web application connecting to this database. The database should be run within a OpenShift Virtualization virtual machine. We do provide the web application as container image.

Sample Solution

Mon, 01 Jan 0001 00:00:00 +0000

As we did not provide a step-by-step guide for the super lab, you can find a sample solution for the lab as kubernetes resources. This is a sample solution and your outcome may vary. The following code blocks contain multiple manifests separated with ---. You can write them to one file (e.g. multiple-manifests.yaml) and apply them all together.

oc create -f multiple-manifests.yaml

If you want to inspect the used cloud-init scripts you can decode the base64 string from the secrets in the code blocks below. Copy the base64 string and use the following command:

OpenShift Virtualization Training

Footer

Footer

Create a VM

Creating a virtual machine

Task 1.1.1: Create your first virtual machine

Lab environment

Login

OpenShift’s web console

Web terminal

KubeVirt configuration

KubeVirt installation

Emulation

Start and stop a VM

Lifecycle

Accessing the console

Console access

Task 1.3.2: Entering the console using virtctl

Tolerations

Necessary tolerations

Exposing VM Ports

Note

Checking available Services resources

Changing files

Task 1.5.1: Create a file

Task 1.5.2: Restart VM

Introduction

CDI Operator components

Note

Create a DataVolume

Provisioning a Virtual Machine Disk

Task 2.2.1: Write your own DataVolume manifest

Using the Alpine disk

Task 2.3.1: Attach the disk

Note

Introduction

VM VirtualMachineInstancetype

Common instance types and preferences

Common instance types and preferences

Create an instance type

Task 3.3.1: Create your own instance type

Re-assign an instance type

Changing an instance type

Introduction

Cloud-init

Supported data sources

Note

cloudInitNoCloud data source

Ignition

Supported data sources

cloudInitNoCloud data source

Sysprep

VM disk images

Virtual machine pools

Using a virtual machine pool

Virtual machine replica sets

Mounting storage

Kubernetes storage

Using StorageProfiles

Warning

What are storage profiles

Hotplugging disks

Task 6.3.1: Starting a virtual machine

VM snapshot and restore

Resizing a Disk

Requirements

Introduction

Perform a live migration

Task 7.2.1: Creating a virtual machine

Guest agent

Task 8.1.1: Start a virtual machine and explore guest agent information

Readiness and liveness probes

Prometheus monitoring

Prometheus Node Exporter

Requirements

Basic Information

Sample Solution

`cloudInitNoCloud` data source

`cloudInitNoCloud` data source