This guide is intended to accompany and extend the official Cobbler Installation and Quick Start guides.

Cobbler is an open-source, system deployment and provisioning software which can simplify and automate operating system installations over the network, as well as rudimentary configuration and patch management capabilities and some other features too. It is a modular solution for deploying systems via PXE, where you would normally have to setup and maintain your own DHCP, TFTP and HTTP servers to provide the full PXE boot+install functionality, Cobbler abstracts all (or some) of those services and manages them for you. Cobbler was initially developed at Red Had and is best suited to deploy Red Hat Linux flavors, like Fedora, but it is fairly capable of deploying just about anything, albeit sometimes requiring some advanced customizations.

Table of Contents

1. Objective
2. Environment
3. Fedora Server Basics
   1. Updating
   2. customizing
4. Cobbler Basics
   1. Dependencies
   2. Installation
   3. Configuration
5. Fedora 34 PXE Deployment
6. Fedora 37 PXE Deployment
7. Security Hardening
   1. SELinux
      1. Enabling SELinux
      2. Enforcing SELinux
      3. Cobbler SELinux Config
   2. Network Security
      1. Firewalld
      2. Service confinement
8. Tips & Troubleshooting

Objective

Starting with a fresh installation of Fedora 34 Server (or Workstation - Fedora download archive), this guide will detail preparing the system for Cobbler v3.3.6 installation from source, including prerequisite/dependency installations and network setup, and then provide all necessary steps to install and configure Cobbler to automatically provision another LAN host with Fedora 34 or 37 Server through PXE network boot.

Environment

This guide assumes that both the Cobbler Server and PXE Clinet(s) are running as virtual machines (VMs) under the same Oracle VM VitualBox (Windows 10) host

• The Cobbler server VM is configured with the following system specs:

  • BIOS or UEFI firmware

  • 64 GB HDD

  • 4 GB RAM

  • 2 vCPUs

  • 2 virtual network adapters.

    1. VirtualBox VM Bridged Adapter connected to the primary physical NIC on the Windows host so that SSH to the Cobbler server will be more accessible.

    2. VirtualBox VM Internal Network adapter named “cobbler” which will have no external network connectivity (closed LAN; no internet access).

• The PXE client VM (the target for network installations) has the following specs:

  • BIOS or UEFI firmware with the following boot order (See Tips & Troubleshooting section below for more info):

    1. Hard Drive

    2. Network

  • 32 GB HDD

  • 4 GB RAM

  • 2 vCPUs

  • 1 virtual network adapter which is connected to the same “cobbler” VirtualBox adapter described above.

Physical Network Diagram

The Cobbler server and PXE client(s) are not shown here, as they are VMs, not physical workstations.

Logical Network Diagram

The VirtualBox host (my Windows 10 laptop) is only shown for completeness, it is not used in the procedures other than the VM settings described above.

This exact configuration is not a strict requirement for deploying systems through Cobbler, but it proves that a PXE client is able to boot and install an operating system over the local network without public internet connectivity, using only the resources immediately available on the VM and the Cobbler server, aleviating a potential bandwidth bottleneck at the WAN.

Fedora Server Basics

As stated above, this document outlines the procedures necessary to install and configure Cobbler v3.3.6 on a Fedora 34 host server for local network installations/provisioning through PXE. Additionally, necessary adjustments will be made for security through selinux and firewalld on the Cobbler server. As such, it is recommended to keep things simple and limit the number of additional applications installed to the Cobbler server to minimize unexpected firewall and selinux complications.

Updating

If Cobbler is to be installed on a Fedora 34 Server host, and automatic partitioning was used during initial OS installation, be sure to extend the LVM Logical Volme to a more usable capacity (recommended 95%):

lvextend /dev/mapper/fedora_$HOSTNAME-root -l+95%FREE -r

Update the Fedora 34 host with the latest YUM repo updates:

yum update

Install VirtualBox VM Guest Additions, either from the YUM repos, or the Guest Adds. CD included with VirtualBox Manager (YUM method is shown below, but VirtualBox usually recommends using the Guest Adds. CD):

yum install virtualbox-guest-additions

customizing

Install your preferred Linux sys admin tools and configure some convenient bash settings, for example (PS1 variable controls bash shell prompt):

systemctl enable --now ssh
yum install neovim fzf tmux htop net-tools tftp
[ -e ~/.bash_profile ] && echo 'export PS1="\[\e[1;32m\][\u@\h \W]\$ \[\e[0m\]"' | tee -a ~/.bash_profile || echo 'export PS1="\[\e[1;32m\][\u@\h \W]\$ \[\e[0m\]"' | tee -a ~/.bashrc

SELinux and firewalld will be configured after Cobbler has been tested successfully, disable them for now.

sed -i -E "s/SELINUX=(disabled|enforcing)/SELINUX=permissive/" /etc/selinux/config
setenforce permissive

systemctl disable --now firewalld

Cobbler Basics

Now would be a good time to power off the Cobbler server to take a VM snapshot.

Dependencies

Download the Fedora 34 & 37 Server installation media (.iso files):

[ -d ~/Downloads ] || mkdir ~/Downloads
cd ~/Downloads && wget https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/34/Server/x86_64/iso/Fedora-Server-dvd-x86_64-34-1.2.iso
cd ~/Downloads && wget https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/37/Server/x86_64/iso/Fedora-Server-dvd-x86_64-37-1.7.iso

Install dependencies for building and installing Cobbler from Source

yum install make git python3-devel python3-sphinx python3-coverage openssl httpd-devel

Cobbler is also available through the YUM repos, using yum install cobbler

Download cobbler v3.3.6 source code from the official GitHub repo

wget -P  ~/Downloads/. https://github.com/cobbler/cobbler/archive/refs/tags/v3.3.6.zip

mkdir -p /usr/src/cobbler
cd /usr/src/cobbler
unzip -d . ~/Downloads/v3.3.6

Install the runtime dependencies and enable system services (systemd will probably report some errors…)

yum install httpd wget curl rsync dnf-plugins-core dosfstools createrepo_c xorriso python3-mod_wsgi python-cheetah python3-pyyaml python-netaddr python-librepo python-schema syslinux tftp-server dhcp-server pykickstart ipxe-bootimgs ipxe-roms koan mod_ssl python3-pyflakes python3-pycodestyle rpm-build python3-dns python3-file-magic python3-pymongo
systemctl enable --now httpd tftp dhcpd

Install the GRUB bootloader and module packages so GRUB can be used as the network bootloader for PXE clients:

yum install grub2-pc grub2-pc-modules grub2-efi-x64-modules grub2-efi-aa64-modules grub2-efi-arm-modules grub2-efi-ia32-modules grub2-emu-modules grub2-emu-modules grub2-ppc64le-modules grub2-emu

Installation

Install Cobbler

cd /usr/src/cobbler/cobbler-3.3.6
make install
systemctl restart httpd

Reconfigure the cobblerd daemon as described in Cobbler’s official docs:

sed -i "s,usr/bin/cobblerd,usr/local/bin/cobblerd," /etc/cobbler/cobblerd.service
cp /etc/cobbler/cobblerd.service /etc/systemd/system/.
systemctl daemon-reload
systemctl enable cobblerd

Configuration

Reconfigure the default Cobbler settings file to match the local server:

sed -i "s/manage_dhcp: false/manage_dhcp: true/" /etc/cobbler/settings.yaml
sed -i "s/dhcp_v4: false/dhcp_v4: true/" /etc/cobbler/settings.yaml
sed -i "s/server: 127.0.0.1/server: 10.0.0.10/" /etc/cobbler/settings.yaml
sed -i "s/next_server_v4: 127.0.0.1/next_server_v4: 10.0.0.10/" /etc/cobbler/settings.yaml

Optionally, toggle some additional settings:

sed -i "s/pxe_just_once: true/pxe_just_once: false/" /etc/cobbler/settings.yaml
sed -i "s/enable_ipxe: false/enable_ipxe: true/" /etc/cobbler/settings.yaml

Edit the subnet decaration in /etc/cobbler/dhcp.tempate to match the enp0s8 interface:

subnet 10.0.0.0 netmask 255.255.255.0 {
     option routers             10.0.0.1;
     option domain-name-servers 10.0.0.1;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        10.0.0.100 10.0.0.199;

Cobbler should now be able to start up and run sync and check tasks without fatal errors:

systemctl restart cobblerd && sleep 10
cobbler sync && sleep 5
cobbler check
cobbler mkloaders

Finally, pull the latest official Cobbler signatures

cobbler signature update
systemctl restart cobblerd && sleep 5
cobbler sync

The Cobbler server is now ready to begin importing distros and deploying systems

Fedora 34 PXE Deployment

Mount the Fedora 34 Server installation media and run the cobbler import

mkdir /mnt/Fedora
mount -t iso9660 -o loop,ro /home/fedora/Downloads/Fedora-Server-dvd-x86_64-34-1.2.iso /mnt/Fedora
cobbler import --name=Fedora34 --arch=x86_64 --path=/mnt/Fedora

Optionally, inspect the imported Distribution and child Profile:

cobbler distro report
cobbler profile report

Create a new generic Fedora 34 kickstart file from Cobbler’s default sample.ks and configure the new Cobbler Profile to use it:

cat /var/lib/cobbler/templates/sample.ks | grep -v "\--useshadow" | grep -v ^install | sed 's,selinux --disabled,selinux --permissive,' | sed 's,rootpw --iscrypted \$default_password_crypted,rootpw --iscrypted \$default_password_crypted\nuser --groups=wheel --name=fedora --password=\$default_password_crypted --iscrypted --gecos="fedora",' | tee /var/lib/cobbler/templates/Fedora34.ks
cobbler profile edit --name Fedora34-x86_64 --autoinstall Fedora34.ks

This new kickstart file has removed some options that cause fatal errors during installation, and also configures SELinux in permissive mode on the resulting system, as well as creating a new admin user named fedora which allows for immediate SSH access to the installed system.

Now, create the PXE Client VM according to the specs described in the Environment section above and note down the vNIC’s MAC address.

At this point, you may run the following commands on the Cobbler server, then start the PXE Client VM to test a manual installation over PXE.

systemctl restart cobblerd
cobbler sync

Finally, create a new Cobbler System, replacing aa:bb:cc:dd:ee:ff with the MAC Address of the PXE Client VM created above, and sync up Cobbler (More information on Cobbler autoinstall templates and kickstart files in the Tips & Troubleshooting section below):

cobbler system add --name Fedora34 --profile Fedora34-x86_64 --netboot-enabled true --hostname fedora34 --interface enp0s3 --static true --mac-address "aa:bb:cc:dd:ee:ff" --ip-address 10.0.0.11 --gateway 10.0.0.1 --netmask 255.255.255.0 --name-servers "10.0.0.1"
systemctl restart cobblerd && sleep 10
cobbler sync

leave out the --static true option in the above command to configure the PXE client to use DHCP for its IP configuration post-install; if using DHCP, you may omit all the command parameters following --hostname fedora34.

The PXE Client VM can now be powered on, and should automatically boot to PXE and install Fedora 34 to the VM HDD using the “Fedora34.ks” kickstart template created above.

The newly installed system will have the password “cobbler” for both users “root” and “fedora” which is configurable through the default_password_crypted setting in /etc/cobbler/settings.yaml as well as the kickstart template file created above.

Fedora 37 PXE Deployment

Take similar steps as above to import and autoinstall Fedora 37 Server, being sure to use unique names for Cobbler Distros, Profiles, and Systems, as well as unique MAC addresses and IP addresses for Systems (unless configured differently in /etc/cobbler/settings.yaml)

mount -t iso9660 -o loop,ro /home/fedora/Downloads/Fedora-Server-dvd-x86_64-37-1.7.iso /mnt/Fedora
cobbler import --name=Fedora37 --arch=x86_64 --path=/mnt/Fedora
cobbler distro edit --name Fedora37-x86_64 --kernel-options ""
cat /var/lib/cobbler/templates/sample.ks | grep -v "\--useshadow" | grep -v ^install | sed 's,selinux --disabled,selinux --permissive,' | sed 's,rootpw --iscrypted \$default_password_crypted,rootpw --iscrypted \$default_password_crypted\nuser --groups=wheel --name=fedora --password=\$default_password_crypted --iscrypted --gecos="fedora",' | tee /var/lib/cobbler/templates/Fedora37.ks
cobbler profile edit --name Fedora37-x86_64 --autoinstall Fedora37.ks
cobbler system add --name Fedora37 --profile Fedora37-x86_64 --netboot-enabled true --hostname fedora37 --interface enp0s3 --static true --mac-address "aa:bb:cc:dd:ee:ff" --ip-address 10.0.0.12 --gateway 10.0.0.1 --netmask 255.255.255.0 --name-servers "10.0.0.1"
systemctl restart cobblerd && sleep 10
cobbler sync

Security Hardening

SELinux and firewall operations can seem daunting at first, especially with an application as complex as Cobbler, but the basic configurations here are relatively simple.

It is possible to lock yourself out of your Linux server with SELinux, so now would be another good time to take a VM snapshot.

SELinux

SELinux should come enabled by default on a fresh Fedora 34 Server installation, and we set it to permissive in the Base Updates and Configs section above, but sometimes selinux can be disabled on the kernel command line

If selinux=0 is set in the /proc/cmdline file, then follow the instructions found in the /etc/selinux/config file for the grubby command to enable selinux at boot time. The kernel command line parameters will take precedence over the other configurations.

If the output of the getenforce command is already permissive skip to the Enforcing SELinux section below; if it is enforcing skip to the Cobbler SELinux Config section.

Enabling SELinux

If SELinux was disabled at the kernel command line or in the /etc/selinux/config file, be sure to create the .autorelabel file at the root of the filesystem, and set SELINUX mode to permissive and NOT enforcing in the /etc/selinux/config file.

sed -i -E "s/SELINUX=(disabled|enforcing)/SELINUX=permissive/" /etc/selinux/config
touch /.autorelabel

With selinux enabled at the kernel command line and set to permissive in “/etc/selinux/config” reboot the Fedora 34 Cobbler server and observe the filesystem relabel as the system boots.

The local filesystem maintains no security context labels when SELinux is disbled, so rebooting directly to SELinux enforcing mode will prevent the system from booting properly.

Enforcing SELinux

Change the active runtime and boot state/mode of SELinux to enforcing:

setenforce enforcing

sed -i 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config

Cobbler SELinux Config

WIth SELinux operating in enforcing mode, enable the necessary SELinux Booleans which will allow for basic operation of Cobbler:

setsebool -P cobbler_can_network_connect 1
setsebool -P httpd_can_network_connect_cobbler 1
setsebool -P httpd_serve_cobbler_files 1

With SELinux enabled, some files generated/copied/moved by Cobbler will not retain the correct security context labels necessary for proper operation. Create a Cobbler sync-trigger to correct the SELinux labels upon any and all successful completions of a cobbler sync task.

echo '#!/bin/bash
restorecon -R /var/lib/tftpboot
restorecon -R /var/www/cobbler' | tee /var/lib/cobbler/triggers/sync/post/zzz-selinux-restorecon

chmod u+x /var/lib/cobbler/triggers/sync/post/zzz-selinux-restorecon

issue another cobbler sync to check for errors (should show *** TASK COMPLETE ***):

systemctl restart cobblerd && sleep 10
cobbler sync

Cobbler Triggers will run at different times depending on which directory they are placed in, and in alphabetical order within that directory. This sync-trigger specifically should typically be run after ALL sync tasks/triggers are completed

Additionally, if there is an error with a cobbler sync task (trigger, or otherwise) before reaching this trigger, the sync task will terminate, and the trigger will not run, causing Cobbler files to not be relabeled appropriately… however, there are likely bigger issues at hand if the cobbler sync task is erroring out.

More info on Cobbler Triggers

Network Security

reduce network exposure with firewalld and secure service/daemon settings

Firewalld

Firewalld is the default firewall installed on Fedora and most Red Hat distros, verify that it is enabled and running:

systemctl enable --now firewalld

As described above, this Cobbler server has 2 ethernet NICs: enp0s3 & enp0s8

• enp0s3 being the upstream interface which can access the internet

• enp0s8 being the downstream interface which will server the PXE clients

Create a new firewalld zone which enp0s8 will operate in:

firewall-cmd --permanent --new-zone=cobbler
firewall-cmd --permanent --zone=cobbler --set-target=DROP
firewall-cmd --permanent --zone=cobbler --add-service=ssh
firewall-cmd --permanent --zone=cobbler --add-service=tftp
firewall-cmd --permanent --zone=cobbler --add-service=http
firewall-cmd --permanent --zone=cobbler --add-service=https
firewall-cmd --reload

Linux ISC DHCP server opens a raw UDP socket with the Linux kernel, bypassing the firewalld rules, so it is not necessary to allow the service/port. Be sure the resulting /etc/dhcp/dhcpd.conf file is only configured for the desired interface(s) for hosting DHCP.

• serverfault & stackexchange sources.

Reconfigure enp0s8 to operate in the new firewalld zone

nmcli con mod enp0s8 connection.zone cobbler

As defined above, this zone will NOT permit inbound ICMP traffic like ping

For simplicity, enp0s3 can be left in the default firewalld zone (verify the current active zones with firewall-cmd --get-active-zones)

Service confinement

On the topic of network security, we should also restrict our Cobbler services to only listen on the PXE server interface.

• configure HTTP to only listen on the PXE server interface:

sed -i 's,^Listen.*,Listen 10.0.0.10:80,' /etc/httpd/conf/httpd.conf

• Create a local systemd service for tftpd, modified from the original config to only listen on a specific IP address:

systemctl stop tftp.service
systemctl stop tftp.socket

cp /lib/systemd/system/tftp.service /etc/systemd/system/.
sed 's,ListenDatagram=69,ListenDatagram=10.0.0.10:69,' /lib/systemd/system/tftp.socket | tee /etc/systemd/system/tftp.socket

• now, reload the systemd configuration and restart services

systemctl daemon-reload
systemctl restart tftp httpd

Feel free to comment with additional security hardening measures you would employ.

Tips & Troubleshooting

1. Cobbler’s official docs advise that installing and running Cobbler in a virtual environment is not possible, but I have had no issues related to virtualization.

2. Some dependencies are missing from the official Cobbler docs. Have a look at the dockerfile located in cobbler’s source code for a more complete list.

3. It’s always a good idea to manually install the desired distro (Fedora 34 or 37 Server in the case of this guide) to the desired hardware (VirtualBox VM in this case) before attempting a network installation, to validate compatability.

   • Any Fedora installation, whether installed manually or automatically, should always contain the file /root/anaconda-ks.cfg, which is the kickstart file used to install that particular system.

   • Depending on your deployed system of choice (Fedora 34 or 37 Server here) if you are having issues, you may want to familiarize yourself with the installation system environment.

     1. With most distributions, while the installer is progressing and outputing logs to the monitor (over TTY/GUI, not serial or IPMI session), you can press ALT+F2 or ALT+F3 or a similar key combo to access a privileged shell to troubleshoot.

     2. some commands you may want to try on the installer system include:

         dmesg
         cat /proc/cmdline
         less /var/log/syslog
         mount
         ls -la
         ls -la /tmp
         ls -la /
         cpuinfo
         lsblk
         lsmod
         uname -a
         cat /etc/os-release
         lsb_release -a
         echo $SHELL
        

4. The PXE client software that comes with VirtualBox VM firmware DOES NOT support HTTP as a download protocol, so it may be best to leave iPXE disabled in /etc/cobbler/settings.yaml

   • HTTP can always be utilized if you boot the PXE client from CD using the latest ipxe.iso

5. By default, Cobbler’s DHCP server attempts to detect the PXE client’s boot firmware type in order to provide a BIOS or UEFI compatible bootloader (BIOS: “grub.0” - UEFI: “grubx64.efi”). If using BIOS firmware for the PXE client VM, the Cobbler server’s DHCP service may not provide the right bootloader.

   1. manually set the bootloader for a Cobbler Profile or System with the following command (the filename value corresponds to the filename variable for the Cobbler System in the generated dhcpd.conf file, and is a local system path to the desired bootloader file relative to Cobbler’s tftp root directory: /var/lib/tftpboot):

      • cobbler <profile|system> edit --name <name> --filename grub/grub.0

   2. The PXE client software that comes with VirtualBox VM BIOS & EFI firmware DOES NOT support HTTP as a download protocol, so it may be best to leave iPXE disabled in /etc/cobbler/settings.yaml

      • HTTP can always be utilized if you boot the PXE client from CD using the latest ipxe.iso

6. The autoinstall template files “Fedora34.ks” “Fedora37.ks” & “sample.ks” located in /var/lib/cobbler/templates are used to dynamically generate the specific kickstart files for either a Cobbler Profile or System at the time of installation

   1. These templates use variables local to the Cobbler server to compose a specific configuration for the desired installation, but you can always craft an explicit kickstart file (without variables or snippets) manually and copy it to the same /var/lib/cobbler/templates directory and assign a Cobbler Profile or System to use that specific file with the command: cobbler <profile|system> edit --name <name> --autoinstall <file> just be sure to escape any dollar-signs ($) with back-slashes (\), otherwise they may be interpretted as variables by Cobbler

   2. The final autoinstall files (kickstart for RHEL distros, preseed for Debian, cloud-init for Ubuntu, autoyast for OpenSUSE, answerfile for Windows as well as XCP-ng, although dissimilar schemas) are hosted over HTTP by Cobbler and, by default, assigned to a Cobbler System or Profile via the bootloader config (GRUB or pxelinux) under /var/lib/tftpboot/grub/system or /var/lib/tftpboot/pxelinux.cfg directories, or in /var/lib/tftpboot/grub/x86_64_menu_items.cfg file.

       curl http://localhost/cblr/svc/op/autoinstall/profile/Fedora34-x86_64
       curl http://localhost/cblr/svc/op/autoinstall/profile/Fedora37-x86_64
       curl http://localhost/cblr/svc/op/autoinstall/system/Fedora34
       curl http://localhost/cblr/svc/op/autoinstall/system/Fedora37
      

   3. There are several template files and accompanying snippets that come default with Cobbler which you can look at if you have any questions about how the Cobbler templates (Python Cheetah templates) work. Located in /var/lib/cobbler/templates & /var/lib/cobbler/snippets

7. Optionally, remove the deprecated “tree” option from the Cobbler Distro kernel-options which can be used in the kickstart file instead. This option will be ignored anyway by the installer.

    cobbler distro edit --name Fedora34-x86_64 --kernel-options
   

   • cobbler still appropriately sets the “inst.repo” kernel-option in the GRUB or PXE kernel-options. This is likey managed by the Cobbler Profile’s “tree” variable (seen with cobbler profile report above) or the Profile’s config file in /var/www/cobbler/distro_mirror/config directory

here are some helpful troubleshooting commands on the Cobbler server

man cobbler
man dhcpd
man tftp
man httpd
man selinux
man firewalld

cobbler distro report
cobbler profile report
cobbler system report

cobbler --help
cobbler distro edit --help
cobbler profile edit --help
cobbler system edit --help

systemctl status cobblerd
systemctl status dhcpd
systemctl status tftp
systemctl status httpd

journalctl -eu cobblerd
journalctl -eu dhcpd
journalctl -eu tftp
journalctl -eu httpd

ls /etc/cobbler
cat /etc/cobbler/settings.yaml
ls /var/lib/cobbler
ls /var/www/cobbler
ls /var/lib/tftpboot/grub/system
ls /var/lib/tftpboot/pxelinux.cfg

cat /var/log/httpd/error_log
cat /var/log/audit/audit.log | grep AVC

cat /etc/dhcp/dhcpd.conf

cat /var/lib/tftpboot/grub/x86_64_menu_items.cfg
cat /var/lib/tftpboot/pxelinux.cfg/default

tftp 10.0.0.10
tftp 127.0.0.1

cat /etc/httpd/conf/httpd.conf
cat /etc/httpd/conf.d/cobbler.conf