View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000432AlmaLinux-8kernelpublic2023-10-13 17:522023-10-13 17:52
Reportersbhat Assigned To 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformLinuxOSAlmalinux8 and 9OS Version8.8 9.1
Summary0000432: Intermittent I/O block, timeout of disks in ZFS on Almalinux8
DescriptionThis is a continuation of https://bugs.almalinux.org/view.php?id=142 -

We had ordered around 4 https://www.supermicro.com/en/products/accessories/addon/AOM-S3216-L-2.php hardwares and installed Almalinux 8 and 9 to Setup a ZFS pool for our Storage server need.

However, the pool was failing with read and write errors exactly like previous bug:
```
[root@zfs-backup5:/backups/blahblah]$ zpool status | awk '$4 > 0'
status: One or more devices has experienced an unrecoverable error. An
        attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
        NAME STATE READ WRITE CKSUM
            sde ONLINE 2 302 0
            sdk ONLINE 3 234 0
errors: No known data errors
[root@zfs-backup5:/backups/blahblah]$
```

This is how errors look in dmesg:
```
Oct 4 12:52:55 zfs-backup5 kernel: sd 6:0:10:0: [sdk] tag#4183 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=30s
Oct 4 12:52:55 zfs-backup5 kernel: sd 6:0:10:0: [sdk] tag#4183 CDB: Write(16) 8a 00 00 00 00 00 1d ca 72 d8 00 00 08 00 00 00
Oct 4 12:52:55 zfs-backup5 kernel: blk_update_request: I/O error, dev sdk, sector 499806936 op 0x1:(WRITE) flags 0x700 phys_seg 94 prio class 0
Oct 4 12:52:55 zfs-backup5 kernel: zio pool=backups vdev=/dev/sdk1 error=5 type=2 offset=255900102656 size=1048576 flags=40080c80
Oct 4 12:52:55 zfs-backup5 kernel: sd 6:0:10:0: attempting task abort!scmd(0x00000000000fbe3a), outstanding for 30395 ms & timeout 30000 ms
Oct 4 12:52:55 zfs-backup5 kernel: sd 6:0:10:0: [sdk] tag#4120 CDB: Write(16) 8a 00 00 00 00 00 1d ca 82 d8 00 00 08 00 00 00
Oct 4 12:52:55 zfs-backup5 kernel: scsi target6:0:10: handle(0x001b), sas_address(0x5000c500ec42c8dd), phy(5)
Oct 4 12:52:55 zfs-backup5 kernel: scsi target6:0:10: enclosure logical id(0x500304802ce92e00), slot(10)
Oct 4 12:52:55 zfs-backup5 kernel: scsi target6:0:10: enclosure level(0x0000), connector name( )
Oct 4 12:52:55 zfs-backup5 kernel: sd 6:0:10:0: task abort: SUCCESS scmd(0x00000000000fbe3a)
Oct 4 12:52:55 zfs-backup5 kernel: sd 6:0:10:0: [sdk] tag#4120 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=30s
Oct 4 12:52:55 zfs-backup5 kernel: sd 6:0:10:0: [sdk] tag#4120 CDB: Write(16) 8a 00 00 00 00 00 1d ca 82 d8 00 00 08 00 00 00
Oct 4 12:52:55 zfs-backup5 kernel: blk_update_request: I/O error, dev sdk, sector 499811032 op 0x1:(WRITE) flags 0x700 phys_seg 99 prio class 0
Oct 4 12:52:55 zfs-backup5 kernel: zio pool=backups vdev=/dev/sdk1 error=5 type=2 offset=255902199808 size=1048576 flags=40080c80
Oct 4 12:53:25 zfs-backup5 kernel: sd 6:0:10:0: attempting task abort!scmd(0x00000000797d5399), outstanding for 60067 ms & timeout 60000 ms
Oct 4 12:53:25 zfs-backup5 kernel: sd 6:0:10:0: [sdk] tag#4454 CDB: Log Sense 4d 00 4d 00 00 00 00 00 04 00
Oct 4 12:53:25 zfs-backup5 kernel: scsi target6:0:10: handle(0x001b), sas_address(0x5000c500ec42c8dd), phy(5)
Oct 4 12:53:25 zfs-backup5 kernel: scsi target6:0:10: enclosure logical id(0x500304802ce92e00), slot(10)
Oct 4 12:53:25 zfs-backup5 kernel: scsi target6:0:10: enclosure level(0x0000), connector name( )
Oct 4 12:53:25 zfs-backup5 kernel: sd 6:0:10:0: task abort: SUCCESS scmd(0x00000000797d5399)
```

1. We cleaned up zfs pool, zfs packages and built a mdraid using all the HDDs and started hammering the storage using a cp command that would copy high number of 1Gb files into these storages. Everything completed fine, no errors in the EXT4 filesystem.
2. We cleaned up mdraid array and individually mounted each HDD as mount and started hammering the individual disk mounts using same method as #1. Everything completed fine, no errors in the EXT4 filesystem.
3. We rebuilt zfs pool without arc/log l2 cache setup and started storage test, we started seeing same errors.
4. We cleaned up zpool and rebuilt again and added arc/log l2 cache setup and started storage test, we started seeing the errors.
Note:
4.1. While we were testing all 4 of above, we did change IO scheduler to `none` - we noticed only improvements of write speed on storage array but we still noticed errors on zfs pool
4.2 While we were testing, we did try various kernel versions, installed ZFS supported / recommended Kernel versions as well. Upgraded and Downgraded ZFS versions as well.
5. We disabled the pool sync setting (zfs set sync=disabled backups) and tried to test storage and read/write errors still occurred. Same when we disabled this interrupt setting (/sys/firmware/acpi/interrupts/gpe6C) no improvements.
6. We confirmed pool ashift setting (12) is good and is recommended by checking the disk (zdb -e backups /dev/sda)
7. We installed kernel-ml kernel but a suitable ZFS package or a zfs_kmod were not available to add to the kernel - we read its not worth using ZFS on kernel-ml on ML or LT. So we wanted to go for Alma9 testing.
8. ZFS version we tried with Alma9 and 9 are 2.1.12-1 and 2.1.13-1.

All 4 HW we got had ` LSI SAS3216 PCI-Express Fusion-MPT SAS-3 (rev 01)` controllers in them.

It looks like to me, Almalinux mpt3sas driver is not working correctly with Broadcom Controllers and there seem to be stability issues.
Steps To Reproduce1. Get a Hardware with LSI SAS3216 PCI-Express Fusion-MPT SAS-3 (rev 01) controller.
2. Install Alma8 or 9 and install ZFS, create a zpool.
3. Do as much as copies of data.

You will see IO blocked and controller freaking out.
Additional InformationWe installed Ubuntu 22 and ZFS and everything worked perfectly fine. No errors. We are planning to adopt Ubuntu + ZFS setup for our backup infra. However, as a big RHEL/Almalinux/opensource fan, I wanted to report this so someone would try debug, improve the drivers or work with ZFS may be so AlmaLinux could also be a playground for ZFS lovers and ZFS could be a storage opportunity for AlmaLinux fans.
Tagsalma8, alma9, cmdblock, controllers, lsi, zfs
abrt_hash
URL

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2023-10-13 17:52 sbhat New Issue
2023-10-13 17:52 sbhat Tag Attached: alma8
2023-10-13 17:52 sbhat Tag Attached: alma9
2023-10-13 17:52 sbhat Tag Attached: cmdblock
2023-10-13 17:52 sbhat Tag Attached: controllers
2023-10-13 17:52 sbhat Tag Attached: lsi
2023-10-13 17:52 sbhat Tag Attached: zfs