View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000397AlmaLinux-9kernelpublic2023-05-23 13:352024-07-26 08:29
Reporterws-ab Assigned To 
PriorityhighSeveritymajorReproducibilityrandom
Status newResolutionopen 
PlatformVM in ProxmoxOSAlmaLinuxOS Version9.2
Summary0000397: Kernel bug with cephfs after update
DescriptionAfter the update from 9.1 to 9.2 we encounter a, seemingly, random kernel bug with cephfs.
Steps To ReproduceWe can't reproduce it willingly, but it will happen sooner or later.
After we updated one of our machines yesterday, it already happened 3 times.

ceph-base-17.2.3-2.el9s.x86_64
kernel: 5.14.0-284.11.1.el9_2.x86_64
Additional InformationWe already asked our ceph support, they recommended to open up a case here.
Tagsalmalinux9, Bug, ceph, kernel

Relationships

has duplicate 0000398 closedmetalefty Kernel bug with cephfs after update 

Activities

ws-ab

2023-05-23 13:38

reporter   ~0000905

I am sorry, somehow we got an Error 503 and opened up the case two times.

I just wanted to provide a file with the messages.

May 22 12:55:18 host kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
May 22 12:55:18 host kernel: #PF: supervisor instruction fetch in kernel mode
May 22 12:55:18 host kernel: #PF: error_code(0x0010) - not-present page
May 22 12:55:18 host kernel: PGD 0 P4D 0
May 22 12:55:18 host kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI
May 22 12:55:18 host kernel: CPU: 26 PID: 237932 Comm: kworker/u64:6 Not tainted 5.14.0-284.11.1.el9_2.x86_64 #1
May 22 12:55:18 host kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
May 22 12:55:18 host kernel: Workqueue: ceph-inode ceph_inode_work [ceph]
May 22 12:55:18 host kernel: RIP: 0010:0x0
May 22 12:55:18 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
May 22 12:55:18 host kernel: RSP: 0018:ffff9b640077bb80 EFLAGS: 00010246
May 22 12:55:18 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015
May 22 12:55:18 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffffc0f338637680
May 22 12:55:18 host kernel: RBP: ffff9b640077bd48 R08: 0000000000000078 R09: 00000000ffffffff
May 22 12:55:18 host kernel: R10: ffffffffffffffff R11: ffff8cbf3ffd5d80 R12: 0000000000000001
May 22 12:55:18 host kernel: R13: ffff9b640077bc90 R14: ffffc0f338637680 R15: 0000000000000000
May 22 12:55:18 host kernel: FS: 0000000000000000(0000) GS:ffff8cbf3fc80000(0000) knlGS:0000000000000000
May 22 12:55:18 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 22 12:55:18 host kernel: CR2: ffffffffffffffd6 CR3: 000000101f826005 CR4: 0000000000770ee0
May 22 12:55:18 host kernel: PKRU: 55555554
May 22 12:55:18 host kernel: Call Trace:
May 22 12:55:18 host kernel: <TASK>
May 22 12:55:18 host kernel: ceph_writepages_start+0x8bf/0x19b0 [ceph]
May 22 12:55:18 host kernel: ? find_busiest_group+0x43/0x190
May 22 12:55:18 host kernel: do_writepages+0xcf/0x1d0
May 22 12:55:18 host kernel: ? newidle_balance+0x2e5/0x400
May 22 12:55:18 host kernel: ? update_load_avg+0x7e/0x730
May 22 12:55:18 host kernel: filemap_fdatawrite_wbc+0x66/0x90
May 22 12:55:18 host kernel: filemap_fdatawrite+0x4f/0x70
May 22 12:55:18 host kernel: ceph_inode_work+0x28/0xb0 [ceph]
May 22 12:55:18 host kernel: process_one_work+0x1e8/0x3c0
May 22 12:55:18 host kernel: ? rescuer_thread+0x3a0/0x3a0
May 22 12:55:18 host kernel: worker_thread+0x50/0x3b0
May 22 12:55:18 host kernel: ? rescuer_thread+0x3a0/0x3a0
May 22 12:55:18 host kernel: kthread+0xd9/0x100
May 22 12:55:18 host kernel: ? kthread_complete_and_exit+0x20/0x20
May 22 12:55:18 host kernel: ret_from_fork+0x22/0x30
May 22 12:55:18 host kernel: </TASK>
May 22 12:55:18 host kernel: Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nft_counter nfsv3 nfs_acl nfs lockd grace xt_addrtype nft_compat br_netfilter bridge stp llc ceph libceph dns_resolver fscache netfs rfkill nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject overlay nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 isofs ip_set nf_tables nfnetlink binfmt_misc vfat fat intel_rapl_msr intel_rapl_common qxl drm_ttm_helper ttm kvm_amd drm_kms_helper ccp syscopyarea kvm sysfillrect irqbypass virtio_balloon i2c_piix4 sysimgblt pcspkr fb_sys_fops joydev sch_fq tcp_bbr drm auth_rpcgss xfs libcrc32c sr_mod cdrom ata_generic nvme_tcp nvme_fabrics nvme sd_mod nvme_core sg ata_piix nvme_common crct10dif_pclmul t10_pi crc32_pclmul libata crc32c_intel virtio_net net_failover virtio_console ghash_clmulni_intel failover virtio_scsi serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod fuse
May 22 12:55:18 host kernel: CR2: 0000000000000000
May 22 12:55:18 host kernel: ---[ end trace e4febc6ade47e8ca ]---
May 22 12:55:18 host kernel: RIP: 0010:0x0
May 22 12:55:18 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
May 22 12:55:18 host kernel: RSP: 0018:ffff9b640077bb80 EFLAGS: 00010246
May 22 12:55:18 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015
May 22 12:55:18 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffffc0f338637680
May 22 12:55:18 host kernel: RBP: ffff9b640077bd48 R08: 0000000000000078 R09: 00000000ffffffff
May 22 12:55:18 host kernel: R10: ffffffffffffffff R11: ffff8cbf3ffd5d80 R12: 0000000000000001
May 22 12:55:18 host kernel: R13: ffff9b640077bc90 R14: ffffc0f338637680 R15: 0000000000000000
May 22 12:55:18 host kernel: FS: 0000000000000000(0000) GS:ffff8cbf3fc80000(0000) knlGS:0000000000000000
May 22 12:55:18 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 22 12:55:18 host kernel: CR2: ffffffffffffffd6 CR3: 000000101f826005 CR4: 0000000000770ee0
May 22 12:55:18 host kernel: PKRU: 55555554

May 22 13:07:05 host kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
May 22 13:07:05 host kernel: #PF: supervisor instruction fetch in kernel mode
May 22 13:07:06 host kernel: #PF: error_code(0x0010) - not-present page
May 22 13:07:06 host kernel: PGD 0 P4D 0
May 22 13:07:06 host kernel: Oops: 0010 [0000002] PREEMPT SMP NOPTI
May 22 13:07:06 host kernel: CPU: 24 PID: 249565 Comm: kworker/u64:5 Tainted: G D -------- --- 5.14.0-284.11.1.el9_2.x86_64 #1
May 22 13:07:06 host kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
May 22 13:07:06 host kernel: Workqueue: ceph-inode ceph_inode_work [ceph]
May 22 13:07:06 host kernel: RIP: 0010:0x0
May 22 13:07:06 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
May 22 13:07:06 host kernel: RSP: 0018:ffff9b6402043b80 EFLAGS: 00010246
May 22 13:07:06 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015
May 22 13:07:06 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffffc0f3243e3400
May 22 13:07:06 host kernel: RBP: ffff9b6402043d48 R08: 0000000000000078 R09: 00000000ffffffff
May 22 13:07:06 host kernel: R10: ffffffffffffffff R11: ffff8cbf3ffd5d80 R12: 0000000000000001
May 22 13:07:06 host kernel: R13: ffff9b6402043c90 R14: ffffc0f3243e3400 R15: 0000000000000000
May 22 13:07:06 host kernel: FS: 0000000000000000(0000) GS:ffff8cbf3fc00000(0000) knlGS:0000000000000000
May 22 13:07:06 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 22 13:07:06 host kernel: CR2: ffffffffffffffd6 CR3: 00000006ce530004 CR4: 0000000000770ee0
May 22 13:07:06 host kernel: PKRU: 55555554
May 22 13:07:06 host kernel: Call Trace:
May 22 13:07:06 host kernel: <TASK>
May 22 13:07:06 host kernel: ceph_writepages_start+0x8bf/0x19b0 [ceph]
May 22 13:07:06 host kernel: ? __blk_mq_try_issue_directly+0x168/0x1e0
May 22 13:07:06 host kernel: ? wb_calc_thresh+0x4f/0x70
May 22 13:07:06 host kernel: ? __blk_flush_plug+0x102/0x160
May 22 13:07:06 host kernel: ? blk_finish_plug+0x25/0x40
May 22 13:07:06 host kernel: do_writepages+0xcf/0x1d0
May 22 13:07:06 host kernel: ? newidle_balance+0x2e5/0x400
May 22 13:07:06 host kernel: ? update_load_avg+0x7e/0x730
May 22 13:07:06 host kernel: filemap_fdatawrite_wbc+0x66/0x90
May 22 13:07:06 host kernel: filemap_fdatawrite+0x4f/0x70
May 22 13:07:06 host kernel: ceph_inode_work+0x28/0xb0 [ceph]
May 22 13:07:06 host kernel: process_one_work+0x1e8/0x3c0
May 22 13:07:06 host kernel: worker_thread+0x50/0x3b0
May 22 13:07:06 host kernel: ? rescuer_thread+0x3a0/0x3a0
May 22 13:07:06 host kernel: kthread+0xd9/0x100
May 22 13:07:06 host kernel: ? kthread_complete_and_exit+0x20/0x20
May 22 13:07:06 host kernel: ret_from_fork+0x22/0x30
May 22 13:07:06 host kernel: </TASK>
May 22 13:07:06 host kernel: Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nft_counter nfsv3 nfs_acl nfs lockd grace xt_addrtype nft_compat br_netfilter bridge stp llc ceph libceph dns_resolver fscache netfs rfkill nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject overlay nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 isofs ip_set nf_tables nfnetlink binfmt_misc vfat fat intel_rapl_msr intel_rapl_common qxl drm_ttm_helper ttm kvm_amd drm_kms_helper ccp syscopyarea kvm sysfillrect irqbypass virtio_balloon i2c_piix4 sysimgblt pcspkr fb_sys_fops joydev sch_fq tcp_bbr drm auth_rpcgss xfs libcrc32c sr_mod cdrom ata_generic nvme_tcp nvme_fabrics nvme sd_mod nvme_core sg ata_piix nvme_common crct10dif_pclmul t10_pi crc32_pclmul libata crc32c_intel virtio_net net_failover virtio_console ghash_clmulni_intel failover virtio_scsi serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod fuse
May 22 13:07:06 host kernel: CR2: 0000000000000000
May 22 13:07:06 host kernel: ---[ end trace e4febc6ade47e8cb ]---
May 22 13:07:06 host kernel: RIP: 0010:0x0
May 22 13:07:06 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
May 22 13:07:06 host kernel: RSP: 0018:ffff9b640077bb80 EFLAGS: 00010246
May 22 13:07:06 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015
May 22 13:07:06 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffffc0f338637680
May 22 13:07:06 host kernel: RBP: ffff9b640077bd48 R08: 0000000000000078 R09: 00000000ffffffff
May 22 13:07:06 host kernel: R10: ffffffffffffffff R11: ffff8cbf3ffd5d80 R12: 0000000000000001
May 22 13:07:06 host kernel: R13: ffff9b640077bc90 R14: ffffc0f338637680 R15: 0000000000000000
May 22 13:07:06 host kernel: FS: 0000000000000000(0000) GS:ffff8cbf3fc00000(0000) knlGS:0000000000000000
May 22 13:07:06 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 22 13:07:06 host kernel: CR2: ffffffffffffffd6 CR3: 00000006ce530004 CR4: 0000000000770ee0
May 22 13:07:06 host kernel: PKRU: 55555554

May 23 09:45:33 host kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
May 23 09:45:33 host kernel: #PF: supervisor instruction fetch in kernel mode
May 23 09:45:33 host kernel: #PF: error_code(0x0010) - not-present page
May 23 09:45:33 host kernel: PGD 161766067 P4D 161766067 PUD 161767067 PMD 0
May 23 09:45:33 host kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI
May 23 09:45:33 host kernel: CPU: 17 PID: 1244585 Comm: kworker/u64:0 Not tainted 5.14.0-284.11.1.el9_2.x86_64 #1
May 23 09:45:33 host kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
May 23 09:45:33 host kernel: Workqueue: ceph-inode ceph_inode_work [ceph]
May 23 09:45:33 host kernel: RIP: 0010:0x0
May 23 09:45:33 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
May 23 09:45:33 host kernel: RSP: 0018:ffffa648023b7b80 EFLAGS: 00010246
May 23 09:45:33 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015
May 23 09:45:33 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: fffff72068be0480
May 23 09:45:33 host kernel: RBP: ffffa648023b7d48 R08: 0000000000000078 R09: 00000000ffffffff
May 23 09:45:33 host kernel: R10: ffffffffffffffff R11: ffff95093ffd5d80 R12: 0000000000000001
May 23 09:45:33 host kernel: R13: ffffa648023b7c90 R14: fffff72068be0480 R15: 0000000000000000
May 23 09:45:33 host kernel: FS: 0000000000000000(0000) GS:ffff9508ffa40000(0000) knlGS:0000000000000000
May 23 09:45:33 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 23 09:45:33 host kernel: CR2: ffffffffffffffd6 CR3: 0000000161730003 CR4: 0000000000770ee0
May 23 09:45:33 host kernel: PKRU: 55555554
May 23 09:45:33 host kernel: Call Trace:
May 23 09:45:33 host kernel: <TASK>
May 23 09:45:33 host kernel: ceph_writepages_start+0x8bf/0x19b0 [ceph]
May 23 09:45:33 host kernel: ? fprop_reflect_period_percpu.isra.0+0x7b/0xc0
May 23 09:45:33 host kernel: do_writepages+0xcf/0x1d0
May 23 09:45:33 host kernel: ? newidle_balance+0x2e5/0x400
May 23 09:45:33 host kernel: ? update_load_avg+0x7e/0x730
May 23 09:45:33 host kernel: filemap_fdatawrite_wbc+0x66/0x90
May 23 09:45:33 host kernel: filemap_fdatawrite+0x4f/0x70
May 23 09:45:33 host kernel: ceph_inode_work+0x28/0xb0 [ceph]
May 23 09:45:33 host kernel: process_one_work+0x1e8/0x3c0
May 23 09:45:33 host kernel: worker_thread+0x50/0x3b0
May 23 09:45:33 host kernel: ? rescuer_thread+0x3a0/0x3a0
May 23 09:45:33 host kernel: kthread+0xd9/0x100
May 23 09:45:33 host kernel: ? kthread_complete_and_exit+0x20/0x20
May 23 09:45:33 host kernel: ret_from_fork+0x22/0x30
May 23 09:45:33 host kernel: </TASK>
May 23 09:45:33 host kernel: Modules linked in: tls xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfsv3 nft_counter nfs_acl nfs lockd grace xt_addrtype nft_compat br_netfilter bridge stp llc ceph libceph dns_resolver fscache netfs rfkill overlay nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack isofs nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink binfmt_misc vfat fat intel_rapl_msr intel_rapl_common kvm_amd qxl ccp drm_ttm_helper ttm kvm drm_kms_helper syscopyarea sysfillrect irqbypass sysimgblt i2c_piix4 virtio_balloon pcspkr fb_sys_fops joydev sch_fq tcp_bbr drm auth_rpcgss xfs libcrc32c sr_mod sd_mod cdrom sg ata_generic nvme_tcp nvme_fabrics nvme nvme_core nvme_common ata_piix t10_pi crct10dif_pclmul crc32_pclmul libata crc32c_intel virtio_net net_failover ghash_clmulni_intel virtio_console failover virtio_scsi serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod fuse
May 23 09:45:33 host kernel: CR2: 0000000000000000
May 23 09:45:33 host kernel: ---[ end trace d3df7407caf4e3a6 ]---
May 23 09:45:33 host kernel: RIP: 0010:0x0
May 23 09:45:33 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
May 23 09:45:33 host kernel: RSP: 0018:ffffa648023b7b80 EFLAGS: 00010246
May 23 09:45:33 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015
May 23 09:45:33 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: fffff72068be0480
May 23 09:45:33 host kernel: RBP: ffffa648023b7d48 R08: 0000000000000078 R09: 00000000ffffffff
May 23 09:45:33 host kernel: R10: ffffffffffffffff R11: ffff95093ffd5d80 R12: 0000000000000001
May 23 09:45:33 host kernel: R13: ffffa648023b7c90 R14: fffff72068be0480 R15: 0000000000000000
May 23 09:45:33 host kernel: FS: 0000000000000000(0000) GS:ffff9508ffa40000(0000) knlGS:0000000000000000
May 23 09:45:33 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 23 09:45:33 host kernel: CR2: ffffffffffffffd6 CR3: 0000000161730003 CR4: 0000000000770ee0
May 23 09:45:33 host kernel: PKRU: 55555554

Issue History

Date Modified Username Field Change
2023-05-23 13:35 ws-ab New Issue
2023-05-23 13:35 ws-ab Tag Attached: almalinux9
2023-05-23 13:35 ws-ab Tag Attached: Bug
2023-05-23 13:35 ws-ab Tag Attached: ceph
2023-05-23 13:35 ws-ab Tag Attached: kernel
2023-05-23 13:38 ws-ab Note Added: 0000905
2024-07-26 08:29 metalefty Relationship added has duplicate 0000398