View Issue Details

IDProjectCategoryView StatusLast Update
0000065AlmaLinux-8-OTHERpublic2021-12-10 12:12
ReporterZnuff Assigned Tosfokin  
PrioritynormalSeverityminorReproducibilityalways
Status assignedResolutionopen 
Summary0000065: virt-sysprep created images do not run the --firstboot script
DescriptionAs per title, we usually create our own templates/images for deployment.

Normally we just install the OS minimally, then install the packages that we need, dump to an image to a file, then run `virt-sysprep -a file.img --firstboot our-script.sh`.

This works fine with CentOS 8, Ubuntu 20 and so on -- as expected, on first boot the script runs via the installed `guestfs-firstboot.service` without issues.

On AlmaLinux 8.3 the initial script doesn't run at all.

I'm not sure what prevents it from running. I've run `virt-sysprep -v` against both CentOS 8 and AlmaLinux 8.3, and the output is the same in regards to the `--firstboot` part.
Steps To Reproduce1. Install AlmaLinux as a guest in a VM
2. Dump it to an image file (ie: qemu-img convert -O raw /block/device almalinux.img)
3. Run virt-sysprep on it -- virt-sysprep -a almalinux.img --firstboot our-script.sh
4. Deploy the newly created image in your environment
5. Expect our-script.sh to run via the installed guestfs-firstboot.service
6. Script doesn't run on first boot
Additional Informationhttps://libguestfs.org/virt-sysprep.1.html#firstboot-vs-script
TagsNo tags attached.
abrt_hash
URL

Activities

sfokin

2021-06-24 13:34

developer   ~0000281

Could you please help me with reproducing bug. I've had a problem on step 3.
Ive tried on Centos 8.4
===================
# virt-sysprep -a /media/sf_mount/almalinux.img --firstboot our-script.sh
[ 0.0] Examining the guest ...
virt-sysprep: error: libguestfs error: could not create appliance through
libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct

Original error from libvirt: internal error: qemu unexpectedly closed the
monitor: 2021-06-24T11:59:11.781802Z qemu-kvm: -blockdev
{"driver":"file","filename":"/media/sf_mount/almalinux.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}:
Could not open '/media/sf_mount/almalinux.img': Permission denied [code=1
int1=-1]

If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:

  virt-sysprep -v -x [...]
===================

i add LIBGUESTFS_BACKEND=direct and get
===================
# LIBGUESTFS_BACKEND=direct virt-sysprep -a /media/sf_mount/almalinux.img --firstboot our-script.sh
[ 0.0] Examining the guest ...
virt-sysprep: warning: mount_options: mount_options_stub:
/dev/mapper/cl-root: expecting a device name (ignored)
[ 21.2] Performing "abrt-data" ...
virt-sysprep: error: libguestfs error: glob_expand: glob_expand_stub: you
must call 'mount' first to mount the root filesystem

If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:

  virt-sysprep -v -x [...]
===================

Znuff

2021-12-09 04:10

reporter   ~0000439

Sorry this has been taking me so long to get back to this.

I'm not sure about your issue with virt-sysprep, mine just works without having done anything special.

It does seem that the mount that contains the disk image doesn't seem to be a local one and libvirt has issues accessing it?

---

That being said, the issue is still present. It seems to be something about systemd not liking... something. Again, the issue is **not** present on CentOS8.

There's a discussion on the systemd issue tracker at https://github.com/systemd/systemd/issues/6334 that might seem relevant.

Also, worth mentioning that this is not only happening on Alma Linux, but it's also happening on Rocky Linux 8.4.

Relevant snippets from the virt-sysprep log:


[ 9.5] Installing firstboot script: our-script.sh

commandrvf: ln -sf -- /usr/lib/systemd/system/guestfs-firstboot.service /sysroot/etc/systemd/system/multi-user.target.wants

In the output image the file is present in the proper systemd directory:


[[email protected] multi-user.target.wants]# cat /usr/lib/systemd/system/guestfs-firstboot.service
[Unit]
Description=libguestfs firstboot service
After=network.target
Before=prefdm.service

[Service]
Type=oneshot
ExecStart=/usr/lib/virt-sysprep/firstboot.sh start
RemainAfterExit=yes
StandardOutput=journal+console
StandardError=inherit

[Install]
WantedBy=multi-user.target
[[email protected] multi-user.target.wants]# systemctl status guestfs-firstboot.service
● guestfs-firstboot.service - libguestfs firstboot service
   Loaded: loaded (/usr/lib/systemd/system/guestfs-firstboot.service; bad; vendor preset: disabled)
   Active: inactive (dead)


And the symlink is present:

[[email protected] multi-user.target.wants]# stat $(pwd)/guestfs-firstboot.service
  File: /etc/systemd/system/multi-user.target.wants/guestfs-firstboot.service -> /usr/lib/systemd/system/guestfs-firstboot.service
  Size: 49 Blocks: 0 IO Block: 4096 symbolic link
Device: fd01h/64769d Inode: 140550 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:unlabeled_t:s0
Access: 2021-12-09 06:02:49.352448191 +0200
Modify: 2021-12-09 05:05:40.532000000 +0200
Change: 2021-12-09 05:05:40.532000000 +0200
 Birth: 2021-12-09 05:05:40.532000000 +0200

systemd reports the unit "bad" though, and I can't figure out why:

[[email protected] multi-user.target.wants]# systemctl status guestfs-firstboot.service
● guestfs-firstboot.service - libguestfs firstboot service
   Loaded: loaded (/usr/lib/systemd/system/guestfs-firstboot.service; bad; vendor preset: disabled)
   Active: inactive (dead)

The default target seems to be reached:

[[email protected] multi-user.target.wants]# systemctl status default.target
● multi-user.target - Multi-User System
   Loaded: loaded (/usr/lib/systemd/system/multi-user.target; bad; vendor preset: disabled)
   Active: active since Thu 2021-12-09 07:10:03 EET; 1h 0min left
     Docs: man:systemd.special(7)

Dec 09 07:10:03 test.web-host.ro systemd[1]: Reached target Multi-User System.


So I'm not sure at this point what else is preventing this from being run.

Znuff

2021-12-10 12:12

reporter   ~0000440

Hello again,

Going back to this, the issue was SELinux.

Running:

virt-sysprep -a almalinux.img --firstboot our-script.sh --selinux-relabel

Fixes the problem.

Seems that our reference CentOS image was an 8.1, and between 8.1 > 8.4 some SELinux stuff changed.

This can be closed now.

Issue History

Date Modified Username Field Change
2021-04-21 16:14 Znuff New Issue
2021-04-24 09:26 alukoshko Assigned To => sfokin
2021-04-24 09:26 alukoshko Status new => assigned
2021-06-24 13:34 sfokin Note Added: 0000281
2021-12-09 04:10 Znuff Note Added: 0000439
2021-12-10 12:12 Znuff Note Added: 0000440