Virt

[jtaleric]

Type of change

• Refactor
• New feature
• Bug fix
• Optimization
• Documentation Update

Description

Adding support for Virtual Machines Instances (VMI).

This integration will create a svc + route to communicate with the Virtual Machine.

The workflow is :

1. Prepare VMI instance with cloud-init, inject the machine running k8s-netperf sshkey into the guest
2. Deploy the Client VMI
3. Deploy the Server VMI (cloud-init will start the workload driver servers, ie, netserver)
4. Create svc and route for the Client VMI
5. Attempt to ssh to the Client VMI using ssh-key
6. Check to see if cloud-init has completed (it installs packages at startup)
7. Attempt to launch the workload -- In some cases the server might not be done installing packages, so we have a retry.
8. Run workload (from here on, the workflow is the same as podNet)

Future - We could look to generate our own VM Image with our packages pre-loaded, so the install timing mess can be avoided.

Related Tickets & Documents

• Related Issue #
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.

Testing

• Please describe the System Under Test.
• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

[jeniferh]

We may need to test different VM network "binding" types in the future, i.e. in spec.devices

            interfaces:
            - masquerade: {}
              model: virtio
              name: default

masquerade could eventually be "bridge" and/or "passt"

[rsevilla87]

Why don't you import kubevirt as dependency rather than including it in the repo?

[jtaleric]

Why don't you import kubevirt as dependency rather than including it in the repo?

It is not recommended to import it as a dependency. I attempted this at first but ran into significant issues. When I reached out to the CNV / Kubevirt community - they shared with me how they implemented this in their own csi-driver https://github.com/kubevirt/csi-driver/blob/main/pkg/kubevirt/client.go#L26

Others have ran into this problem too - https://groups.google.com/g/kubevirt-dev/c/vuuvW2E6Kow

Recommended way is described here - https://github.com/kubevirt/api/?tab=readme-ov-file#how-to-use-it

[vishnuchalla]

I have to try this out personally. But looks like there is a way to ssh into a VM without any ssh keys.

If our cluster has VLAN support and have a VLAN created like this

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: vlan-network
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge",
    "ipam": {
      "type": "host-local",
      "subnet": "192.168.1.0/24",
      "rangeStart": "192.168.1.100",
      "rangeEnd": "192.168.1.200",
      "routes": [{"dst": "0.0.0.0/0"}],
      "gateway": "192.168.1.1"
    }
  }'

We can create VM with the below configuration and ssh into it.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: my-vm
spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
          - name: default
            bridge: {}
          - name: vlan-network
            bridge: {}
      networks:
      - name: default
        pod: {}
      - name: vlan-network
        multus:
          networkName: vlan-network

[jtaleric]

I have to try this out personally. But looks like there is a way to ssh into a VM without any ssh keys.

If our cluster has VLAN support and have a VLAN created like this

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: vlan-network
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge",
    "ipam": {
      "type": "host-local",
      "subnet": "192.168.1.0/24",
      "rangeStart": "192.168.1.100",
      "rangeEnd": "192.168.1.200",
      "routes": [{"dst": "0.0.0.0/0"}],
      "gateway": "192.168.1.1"
    }
  }'

We can create VM with the below configuration and ssh into it.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: my-vm
spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
          - name: default
            bridge: {}
          - name: vlan-network
            bridge: {}
      networks:
      - name: default
        pod: {}
      - name: vlan-network
        multus:
          networkName: vlan-network

This is significantly more complex and not an option. The assumption of vlan creation isn't always true, where we can always inject a SSH key.

[jeniferh]

Yeah on the bridge network type, that would allow direct L2 connectivity to the VM from the bastion, however VMs typically use linux bridge not macvlan, which requires an nncp on a secondary interface (i.e. can't be shared w/ the OVN port), and that changes which network type we measure against.

[jtaleric]

We may need to test different VM network "binding" types in the future, i.e. in spec.devices

            interfaces:
            - masquerade: {}
              model: virtio
              name: default

masquerade could eventually be "bridge" and/or "passt"

Trying to future proof things here -- will this be the name? -- Model seems to map to bridge 😕

type Interface struct {
    // Logical name of the interface as well as a reference to the associated networks.
    // Must match the Name of a Network.
    Name string `json:"name"`
    // Interface model.
    // One of: e1000, e1000e, ne2k_pci, pcnet, rtl8139, virtio.
    // Defaults to virtio.
    // TODO:(ihar) switch to enums once opengen-api supports them. See: https://github.com/kubernetes/kube-openapi/issues/51
    Model string `json:"model,omitempty"`
    // BindingMethod specifies the method which will be used to connect the interface to the guest.
    // Defaults to Bridge.
    InterfaceBindingMethod `json:",inline"`
    // Binding specifies the binding plugin that will be used to connect the interface to the guest.
    // It provides an alternative to InterfaceBindingMethod.
    // version: 1alphav1
    Binding *PluginBinding `json:"binding,omitempty"`
    // List of ports to be forwarded to the virtual machine.
    Ports []Port `json:"ports,omitempty"`
    // Interface MAC address. For example: de:ad:00:00:be:af or DE-AD-00-00-BE-AF.
    MacAddress string `json:"macAddress,omitempty"`
    // BootOrder is an integer value > 0, used to determine ordering of boot devices.
    // Lower values take precedence.
    // Each interface or disk that has a boot order must have a unique value.
    // Interfaces without a boot order are not tried.
    // +optional
    BootOrder *uint `json:"bootOrder,omitempty"`
    // If specified, the virtual network interface will be placed on the guests pci address with the specified PCI address. For example: 0000:81:01.10
    // +optional
    PciAddress string `json:"pciAddress,omitempty"`
    // If specified the network interface will pass additional DHCP options to the VMI
    // +optional
    DHCPOptions *DHCPOptions `json:"dhcpOptions,omitempty"`
    // If specified, the virtual network interface address and its tag will be provided to the guest via config drive
    // +optional
    Tag string `json:"tag,omitempty"`
    // If specified, the ACPI index is used to provide network interface device naming, that is stable across changes
    // in PCI addresses assigned to the device.
    // This value is required to be unique across all devices and be between 1 and (16*1024-1).
    // +optional
    ACPIIndex int `json:"acpiIndex,omitempty"`
    // State represents the requested operational state of the interface.
    // The (only) value supported is `absent`, expressing a request to remove the interface.
    // +optional
    State InterfaceState `json:"state,omitempty"`
}

func (*v1.Interface).DeepCopy() *v1.Interface
func (*v1.Interface).DeepCopyInto(out *v1.Interface)
func (v1.Interface).SwaggerDoc() map[string]string```

[jeniferh]

@jtaleric I think its InterfaceBindingMethod

[vishnuchalla]

lgtm

[jtaleric]

Need to add virt to the results until we get this captured in go-commons.

[rsevilla87]

lgtm

[vishnuchalla]

lgtm