View Issue Details

IDProjectCategoryView StatusLast Update
0000359AlmaLinux-9kernelpublic2023-11-20 22:27
Reportertchristian Assigned Toalukoshko  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformDell Optiplex 7050OSAlmaLinuxOS Version9.1
Summary0000359: Reboot/Poweroff hangs with "block device autoloading is deprecated and will be removed" message
DescriptionContext:
AlmaLinux 9.1 updated to latest version (all packages updated, including kernel).
Boot and OS installed on a md raid (mdadm) using an SSD (sata) and a NVME module (M2).
MDADM is not broken, but it does not behave as expected.

Behavior:
System hangs for a few minutes repeating this message several times: "block device autoloading is deprecated and will be removed". After a while, the system proceeds to reboot/poweroff.

Apparently, there was a modification in kernel (current version of Alma) that was fixed in further versions of kernel (6.x). There is a kernel patch to support block device autoloading (assuming that mdadm will not remove device autoloading very soon - last updates on mdadm code is 1-2 years old). I see the kernel patch as the only alternative for the moment.
The patch should enable BLOCK_LEGACY_AUTOLOAD in kernel.
Steps To ReproduceInstall Alma 9.1 (latest) on one or more mdadm partitions (both boot and root). Reboot system
Additional InformationThis behavior was observed on multiple linux distros - all were fixed by either applying the patch or updating kernel to 5.18+ or 6.x
Tagskernel, mdadm, patch, poweroff, reboot, system hangs

Activities

toracat

2023-01-22 11:08

reporter   ~0000788

Maybe related to:

https://forums.rockylinux.org/t/shutdown-gets-into-an-infinite-loop-rocky-linux-9-1/8106
https://lkml.org/lkml/2022/8/26/846

Then the patch will be:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4c66a326b5ab784cddd72de07ac5b6210e9e1b06

Is this the one you applied?

tchristian

2023-01-22 13:56

reporter   ~0000789

Thanks for the updates !

As I see it, there are 2 ways to deal with this issue:
1. Either enable BLOCK_LEGACY_AUTOLOAD in kernels 5.14 to 5.18 (since 5.18 the mainstream of kernel reverted to this option to be ON/YES).
2. Apply the patch you mentioned to not freeze the queue even if the legacy option is not ON.

Since the kernel folks re-enabled this option, I would go for the first option.

The main problem here is that Alma is still using kernel 5.14 - this will be resolved by itself once Alma will reach kernel 5.18+

As a workaround, i would suggest re-enabling the option for kernels 5.14-5.18 directly in the kernel configure/build of Alma

What do you think ?

toracat

2023-01-22 17:59

reporter   ~0000790

In Alma 9.1 (kernel-5.14.0-162.6.1.el9_1), BLOCK_LEGACY_AUTOLOAD is indeed enabled.

* Thu Jun 30 2022 Scott Weaver <scweaver@redhat.com> [5.14.0-120.82.el9]
rhel: configs: enable BLOCK_LEGACY_AUTOLOAD (Ming Lei) [2083917]

* Mon Jun 27 2022 Patrick Talbert <ptalbert@redhat.com> [5.14.0-120.el9]
- block: default BLOCK_LEGACY_AUTOLOAD to y (Ming Lei) [2083917]

In the RHEL (therefore Alma) kernels, the version number can be misleading as a great number of patches are backported.

toracat

2023-01-23 10:47

reporter   ~0000791

I now have a 9.1 kernel set that was built with the patch. It is available here:

https://toracat.org/test/kernel/4c66a32/

Please test if you can. Note that they are not signed and are offered for testing purposes only.

tchristian

2023-01-30 20:06

reporter   ~0000809

Works perfectly fine! Tested with 5.14.0-162.12.1.4c66a32.el9_1 !

First, I updated all the packages, it ended up with the same version of kernel (mainstream, without the patch)... then i installed the rpms provided by you and I confirm that the behavior is as expected, the system does NOT hang anymore on reboot.

Any chance this BLOCK_LEGACY_AUTOLOAD will be set to N in the kernels provided with Alma from now on ? (mainstream).. or at least up to 5.18 ?

Thanks again, great work !

toracat

2023-01-30 20:23

reporter   ~0000810

@tchristian

Great news. Thanks for testing the patched kernel.

Now, the kernel you have tested with success has a patch that was referenced in comment 788 above. The only way to get the fix in the Alma kernel is to file a bug report with Red Hat. AlmaLinux is supposed to be a bug-for-bug rebuild of RHEL, so any fix must come from the upstream kernel.

Likewise, kernel config options cannot be modified in the Alma kernel.

I wonder if you'll be able to file a bug with RH. Here's the link:

https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%209&version=CentOS%20Stream

If you are not comfortable doing this, I could do it. However it will be best if someone who can run a test do the submission.

stijnhoop

2023-02-18 10:33

reporter   ~0000811

I see this same problem in my test VM and also confirm that the kernel provided by @toracat above solves the issue.

toracat

2023-02-18 18:13

reporter   ~0000813

@stijnhoop

Thanks for your report.

If no one is going to file a bug with RH, I'll try doing it. That is the only way to get the patch info downstream projects like AlmaLinux.

toracat

2023-02-20 02:24

reporter   ~0000816

RHBZ filed:

https://bugzilla.redhat.com/show_bug.cgi?id=2171340

toracat

2023-03-06 19:24

reporter   ~0000835

@tchristian and @stijnhoop

The current kernel with the patch (kernel-5.14.0-162.18.1.4c66a32.el9_1.x86_64) is in https://toracat.org/test/kernel/4c66a32/ if you need to update.

toracat

2023-05-18 17:33

reporter   ~0000899

This issue was fixed in the EL 9.2 GA kernel.

The ticket can be closed as 'resolved'.

stijnhoop

2023-05-20 09:16

reporter   ~0000901

I can confirm that AlmaLinux 9.2 images do not exhibit this problem anymore in my test setup. Many thanks @toracat for providing the workaround in the meantime!

toracat

2023-05-21 09:01

reporter   ~0000903

@stijnhoop

You are quite welcome.

Issue History

Date Modified Username Field Change
2023-01-18 11:17 tchristian New Issue
2023-01-18 11:17 tchristian Tag Attached: kernel
2023-01-18 11:17 tchristian Tag Attached: mdadm
2023-01-18 11:17 tchristian Tag Attached: patch
2023-01-18 11:17 tchristian Tag Attached: poweroff
2023-01-18 11:17 tchristian Tag Attached: reboot
2023-01-18 11:17 tchristian Tag Attached: system hangs
2023-01-22 11:08 toracat Note Added: 0000788
2023-01-22 13:56 tchristian Note Added: 0000789
2023-01-22 17:59 toracat Note Added: 0000790
2023-01-23 10:47 toracat Note Added: 0000791
2023-01-30 20:06 tchristian Note Added: 0000809
2023-01-30 20:23 toracat Note Added: 0000810
2023-02-18 10:33 stijnhoop Note Added: 0000811
2023-02-18 18:13 toracat Note Added: 0000813
2023-02-20 02:24 toracat Note Added: 0000816
2023-02-20 19:08 alukoshko Assigned To => alukoshko
2023-02-20 19:08 alukoshko Status new => confirmed
2023-03-06 19:24 toracat Note Added: 0000835
2023-05-18 17:33 toracat Note Added: 0000899
2023-05-20 09:16 stijnhoop Note Added: 0000901
2023-05-21 09:01 toracat Note Added: 0000903
2023-11-20 22:27 alukoshko Status confirmed => resolved
2023-11-20 22:27 alukoshko Resolution open => fixed