Kickstart on XCP-ng: Five Silent Traps

It's 2026 and I'm doing Kickstart at a data center.

What is Kickstart?

An automated OS installation system. A single config file defines partitions, networking, users, and packages, and the installer runs to completion with no human interaction. Red Hat introduced it with Red Hat Linux 5.2 in 1998. Solaris had JumpStart around the same time — automated OS provisioning was not a novel concept in the UNIX world. Over a quarter century later, Kickstart remains the standard provisioning mechanism for RHEL and its derivatives (AlmaLinux, Rocky Linux).

Not the cloud. Bare-metal servers running XCP-ng, with HVM virtual machines on top. There are GUI management tools — XenCenter on Windows, Xen Orchestra on the web. Click around, create a VM. I don't use them. Partly because I only have a Mac. Mostly because a stubborn sense of pride wouldn't allow a GUI. If it's CLI, it's scriptable. If it's scriptable, it's reproducible. SSH and xe commands only. Take an AlmaLinux minimal ISO, automate the install with Kickstart, bake a template VM. That's all. It took an entire day.

What is XCP-ng?

An open-source fork of Citrix XenServer. When Citrix restricted the free edition of XenServer in 2018, Vates launched XCP-ng as a fully open alternative. It uses the Xen hypervisor, and guest VMs are managed via xe commands from dom0 (the privileged management VM). Think of it as the OSS counterpart to VMware ESXi or Hyper-V.

I hit five traps. Every single one fails silently. No error messages.

Trap 1: CD VBD fails silently

Created a VM, inserted the ISO, booted. It halted in 40 seconds. Disk was empty.

What is a VBD (Virtual Block Device)?

A virtual disk device in Xen. It's the virtual equivalent of a SATA port or CD drive on a physical machine — a connection point where you mount a VDI (Virtual Disk Image). A CD VBD is a virtual CD drive.

VMs created with xe vm-install don't have a CD drive. xe vm-cd-insert does nothing if no CD VBD exists. No error. I thought I'd inserted the ISO. Nothing had happened.

# This silently does nothing (no CD VBD exists)
xe vm-cd-insert vm=my-vm cd-name=AlmaLinux-minimal.iso

# Explicitly create a CD VBD, then mount
ISO_VDI=$(xe vdi-list sr-uuid=${ISO_SR} name-label="AlmaLinux-minimal.iso" params=uuid --minimal)
xe vbd-create vm-uuid=${VM_UUID} vdi-uuid=${ISO_VDI} device=3 type=CD mode=RO bootable=true

Don't trust vm-cd-insert. Create the CD VBD explicitly with vbd-create. First lesson.

Trap 2: Can't pass boot args to an HVM VM

Kickstart needs inst.ks= in the kernel parameters. I tried setting it with PV-args and xenstore-write. Ignored.

HVM and PV — Xen's two virtualization modes

Xen has historically had two virtualization modes. PV (Paravirtualization) modifies the guest OS kernel to communicate directly with Xen. It was the core technology in Xen's early days (2003 onward). In PV mode, Xen loads the kernel directly, so you can freely pass kernel parameters via PV-args. HVM (Hardware Virtual Machine) uses CPU hardware virtualization extensions (Intel VT-x / AMD-V) to run unmodified guest OSes. HVM boots through BIOS/UEFI into GRUB, then into the kernel — Xen has no way to inject parameters. HVM is the default in modern XCP-ng. PV is legacy.

HVM VMs boot via BIOS/GRUB. Xen's paravirtualization parameters don't apply. The only option is to build a custom ISO with the GRUB config modified directly.

The ISO authoring tool matters here. genisoimage (mkisofs) doesn't support the --grub2-boot-info flag. That flag is required for GRUB2 boot. xorriso is the only option.

From genisoimage to xorriso

mkisofs was the long-standing tool for creating ISO 9660 images. Its fork genisoimage ships with Debian and RHEL-based distros. But it doesn't support the --grub2-boot-info flag required for GRUB2 BIOS boot. xorriso, developed by the GNU xorriso project, fully supports modern boot configurations including GRUB2. For building custom RHEL-family ISOs today, xorriso is the only viable tool.

The xorriso command is long. But this is the only way to pass Kickstart parameters to an HVM VM.

# Extract ISO
xorriso -osirrox on -indev ${ISO_SRC} -extract / ${WORKDIR}/

# Add kernel parameters to GRUB cfg
KERNEL_ARGS="console=tty0 console=ttyS0,115200n8 earlycon=uart8250,io,0x3F8,115200n8 inst.text inst.headless"
sed -i "s|inst.stage2=hd:LABEL=AlmaLinux quiet|inst.stage2=hd:LABEL=AlmaLinux ${KERNEL_ARGS}|g" \
  ${WORKDIR}/boot/grub2/grub.cfg

# Rebuild ISO (--grub2-mbr and --grub2-boot-info are required)
xorriso -as mkisofs \
  -V "AlmaLinux" \
  --grub2-mbr --interval:local_fs:0s-15s:zero_mbrpt,zero_gpt:${ISO_SRC} \
  --protective-msdos-label \
  -partition_cyl_align off -partition_offset 0 \
  -partition_hd_cyl 92 -partition_sec_hd 32 \
  --boot-catalog-hide \
  -b images/eltorito.img -no-emul-boot -boot-load-size 4 \
  -boot-info-table --grub2-boot-info \
  -o ${ISO_DST} ${WORKDIR}/

Trap 3: Media check halts for 12 hours

Booted the custom ISO. Nothing on screen. Waited five minutes. Ten minutes. Nothing.

The cause was set default="1" in grub.cfg. In the stock AlmaLinux ISO, menu entry 0 is "Install" and entry 1 is "Test this media & install." Default is 1 — media check runs automatically.

An ISO rebuilt with xorriso has no checksum data. Media check fails. Then: Media check failed! System will halt in 12 hours. Twelve hours.

# Change default to 0 ("Install" selected automatically)
sed -i 's/set default="1"/set default="0"/' ${WORKDIR}/boot/grub2/grub.cfg

A one-line fix. Forget it and you wait twelve hours. If you hit this trap before setting up the serial console, you don't even see the message. Nothing happens. Time passes. The sneakiest trap of the five.

Trap 4: Serial console shows nothing

Without a GUI, xl console is the only way in. If the console doesn't work, you're flying blind. The reason I waited twelve hours in Trap 3 was that the console wasn't working in the first place.

The serial console requires four settings. Miss any one and you get nothing.

Setting	Where	Purpose
serial --speed=115200 ... + terminal_input/output serial	GRUB cfg header	Route GRUB menu to serial
earlycon=uart8250,io,0x3F8,115200n8	Kernel parameters	Early-stage kernel output
inst.headless	Kernel parameters	Suppress Anaconda's VGA init
platform:hvm_serial=pty	XCP-ng VM config	PTY allocation on dom0 side

What is earlycon?

A mechanism for the Linux kernel to output to the serial port during early boot, before the normal console drivers are initialized. earlycon=uart8250,io,0x3F8,115200n8 means "use the 8250-compatible UART at I/O port 0x3F8 at 115200 bps." This is the address of COM1 on a physical machine. In Xen HVM, dom0 emulates the UART. Without this parameter, the kernel can't properly initialize the emulated UART.

Add the GRUB settings to the top of grub.cfg in the custom ISO.

serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1
terminal_input serial console
terminal_output serial console

VM-side setting via xe.

xe vm-param-set uuid=${VM_UUID} platform:hvm_serial=pty

Without earlycon, the kernel can't initialize Xen HVM's UART emulation. You get the GRUB menu on serial, then nothing after kernel boot. A half-working state that's worse than fully broken.

Trap 5: Kickstart syntax breaks across AlmaLinux generations

Took a Kickstart file that worked on the previous generation. Brought it to the new one. Parse errors everywhere.

What is AlmaLinux?

A 1:1 compatible RHEL distribution born in 2021 after CentOS 8 reached EOL. When Red Hat announced the shift to CentOS Stream in 2020 — effectively ending CentOS as a stable downstream rebuild — CloudLinux launched AlmaLinux as a fork. Along with Rocky Linux, it's one of the two major CentOS successors. RHEL major version upgrades bring breaking changes to Kickstart syntax.

Item	Old generation	New generation
EULA	eula --agreed	Removed. Causes error
Bootloader	bootloader --location=mbr	bootloader --boot-drive=xvda
NTP	timezone --ntpservers=...	timesource --ntp-server=... (split out)
Partitions	/boot + swap + /	+ biosboot 1MB (required for GPT+BIOS)
network continuation	network ... \ (next line)	No line continuation. Single line

What is the biosboot partition?

A small partition required for BIOS boot from a GPT (GUID Partition Table) disk. With MBR, GRUB's stage 1.5 code is written into the gap between the MBR and the first partition (the MBR gap). GPT has no such gap, so a biosboot partition (typically 1 MB) holds GRUB's core image instead. Not needed with UEFI, but XCP-ng HVM VMs use BIOS boot, so it's mandatory.

GPT is now the default. BIOS boot requires the biosboot partition. Without it, GRUB doesn't get installed. The OS won't boot.

part biosboot --fstype=biosboot --size=1
part /boot --fstype=ext4 --size=1024
part swap  --size=4096
part /     --fstype=xfs --grow --size=1

Line continuation in the network command is also gone. Static IP configs get very long on a single line.

network --bootproto=static --device=link --onboot=yes --activate --ip=192.168.1.200 --netmask=255.255.255.0 --gateway=192.168.1.1 --nameserver=8.8.8.8,1.1.1.1 --hostname=tmpl-alma10

Hard to read. No other way.

The final config

No DHCP at the data center. Static IPs in the Kickstart file. I reserved 192.168.1.200 as a template address and use it for every install. Could have separated the Kickstart file from the GRUB config, but I embedded everything into a single custom ISO. One ISO, then xe commands do the rest.

Here's the final Kickstart layout.

#--- Base ---
text                              # Text mode install
firstboot --disable
reboot                            # Auto-reboot on completion

#--- Locale ---
lang en_US.UTF-8
keyboard --xlayouts='jp'
timezone Asia/Tokyo --utc
timesource --ntp-server=ntp.nict.jp          # ← split from timezone
timesource --ntp-server=ntp.jst.mfeed.ad.jp

#--- Network (template static IP, single line) ---
network --bootproto=static --device=link --onboot=yes --activate --ip=192.168.1.200 --netmask=255.255.255.0 --gateway=192.168.1.1 --nameserver=8.8.8.8,1.1.1.1 --hostname=tmpl-alma10

#--- Disk ---
ignoredisk --only-use=xvda
clearpart --all --initlabel --drives=xvda
zerombr
bootloader --boot-drive=xvda                 # ← --location=mbr is gone

part biosboot --fstype=biosboot --size=1     # ← required for GPT+BIOS
part /boot --fstype=ext4 --size=1024
part swap  --size=4096
part /     --fstype=xfs --grow --size=1

#--- User ---
rootpw --iscrypted $6$...
user --name=libraz --shell=/bin/bash ...

#--- Packages ---
%packages --ignoremissing
@minimal-environment
%end

#--- Post-install ---
%post --log=/root/ks-post.log
# SSH keys, sudoers, sshd_config, shell environment...
# No dnf install here
%end

Change the IP after cloning. Rewriting the Kickstart network config for every VM is tedious. Build one template, xe vm-clone to replicate. More practical.

What the five traps have in common

No error messages. The CD doesn't mount. Parameters are ignored. Media check runs silently. The console shows nothing. Syntax changes without warning.

Kickstart is designed to run to completion without human intervention. It doesn't stop and ask. That's the right design. But when the config is wrong, it doesn't stop and ask either.

Secure the serial console first. Everything starts there. If the console works, you can see what's happening. If you can see what's happening, you can fix it.