View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000397 | AlmaLinux-9 | kernel | public | 2023-05-23 13:35 | 2023-05-23 13:42 |
| Reporter | ws-ab | Assigned To | |||
| Priority | high | Severity | major | Reproducibility | random |
| Status | new | Resolution | open | ||
| Platform | VM in Proxmox | OS | AlmaLinux | OS Version | 9.2 |
| Summary | 0000397: Kernel bug with cephfs after update | ||||
| Description | After the update from 9.1 to 9.2 we encounter a, seemingly, random kernel bug with cephfs. | ||||
| Steps To Reproduce | We can't reproduce it willingly, but it will happen sooner or later. After we updated one of our machines yesterday, it already happened 3 times. ceph-base-17.2.3-2.el9s.x86_64 kernel: 5.14.0-284.11.1.el9_2.x86_64 | ||||
| Additional Information | We already asked our ceph support, they recommended to open up a case here. | ||||
| Tags | almalinux9, Bug, ceph, kernel | ||||
|
|
I am sorry, somehow we got an Error 503 and opened up the case two times. I just wanted to provide a file with the messages. May 22 12:55:18 host kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000 May 22 12:55:18 host kernel: #PF: supervisor instruction fetch in kernel mode May 22 12:55:18 host kernel: #PF: error_code(0x0010) - not-present page May 22 12:55:18 host kernel: PGD 0 P4D 0 May 22 12:55:18 host kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI May 22 12:55:18 host kernel: CPU: 26 PID: 237932 Comm: kworker/u64:6 Not tainted 5.14.0-284.11.1.el9_2.x86_64 #1 May 22 12:55:18 host kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 May 22 12:55:18 host kernel: Workqueue: ceph-inode ceph_inode_work [ceph] May 22 12:55:18 host kernel: RIP: 0010:0x0 May 22 12:55:18 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6. May 22 12:55:18 host kernel: RSP: 0018:ffff9b640077bb80 EFLAGS: 00010246 May 22 12:55:18 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015 May 22 12:55:18 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffffc0f338637680 May 22 12:55:18 host kernel: RBP: ffff9b640077bd48 R08: 0000000000000078 R09: 00000000ffffffff May 22 12:55:18 host kernel: R10: ffffffffffffffff R11: ffff8cbf3ffd5d80 R12: 0000000000000001 May 22 12:55:18 host kernel: R13: ffff9b640077bc90 R14: ffffc0f338637680 R15: 0000000000000000 May 22 12:55:18 host kernel: FS: 0000000000000000(0000) GS:ffff8cbf3fc80000(0000) knlGS:0000000000000000 May 22 12:55:18 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 22 12:55:18 host kernel: CR2: ffffffffffffffd6 CR3: 000000101f826005 CR4: 0000000000770ee0 May 22 12:55:18 host kernel: PKRU: 55555554 May 22 12:55:18 host kernel: Call Trace: May 22 12:55:18 host kernel: <TASK> May 22 12:55:18 host kernel: ceph_writepages_start+0x8bf/0x19b0 [ceph] May 22 12:55:18 host kernel: ? find_busiest_group+0x43/0x190 May 22 12:55:18 host kernel: do_writepages+0xcf/0x1d0 May 22 12:55:18 host kernel: ? newidle_balance+0x2e5/0x400 May 22 12:55:18 host kernel: ? update_load_avg+0x7e/0x730 May 22 12:55:18 host kernel: filemap_fdatawrite_wbc+0x66/0x90 May 22 12:55:18 host kernel: filemap_fdatawrite+0x4f/0x70 May 22 12:55:18 host kernel: ceph_inode_work+0x28/0xb0 [ceph] May 22 12:55:18 host kernel: process_one_work+0x1e8/0x3c0 May 22 12:55:18 host kernel: ? rescuer_thread+0x3a0/0x3a0 May 22 12:55:18 host kernel: worker_thread+0x50/0x3b0 May 22 12:55:18 host kernel: ? rescuer_thread+0x3a0/0x3a0 May 22 12:55:18 host kernel: kthread+0xd9/0x100 May 22 12:55:18 host kernel: ? kthread_complete_and_exit+0x20/0x20 May 22 12:55:18 host kernel: ret_from_fork+0x22/0x30 May 22 12:55:18 host kernel: </TASK> May 22 12:55:18 host kernel: Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nft_counter nfsv3 nfs_acl nfs lockd grace xt_addrtype nft_compat br_netfilter bridge stp llc ceph libceph dns_resolver fscache netfs rfkill nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject overlay nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 isofs ip_set nf_tables nfnetlink binfmt_misc vfat fat intel_rapl_msr intel_rapl_common qxl drm_ttm_helper ttm kvm_amd drm_kms_helper ccp syscopyarea kvm sysfillrect irqbypass virtio_balloon i2c_piix4 sysimgblt pcspkr fb_sys_fops joydev sch_fq tcp_bbr drm auth_rpcgss xfs libcrc32c sr_mod cdrom ata_generic nvme_tcp nvme_fabrics nvme sd_mod nvme_core sg ata_piix nvme_common crct10dif_pclmul t10_pi crc32_pclmul libata crc32c_intel virtio_net net_failover virtio_console ghash_clmulni_intel failover virtio_scsi serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod fuse May 22 12:55:18 host kernel: CR2: 0000000000000000 May 22 12:55:18 host kernel: ---[ end trace e4febc6ade47e8ca ]--- May 22 12:55:18 host kernel: RIP: 0010:0x0 May 22 12:55:18 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6. May 22 12:55:18 host kernel: RSP: 0018:ffff9b640077bb80 EFLAGS: 00010246 May 22 12:55:18 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015 May 22 12:55:18 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffffc0f338637680 May 22 12:55:18 host kernel: RBP: ffff9b640077bd48 R08: 0000000000000078 R09: 00000000ffffffff May 22 12:55:18 host kernel: R10: ffffffffffffffff R11: ffff8cbf3ffd5d80 R12: 0000000000000001 May 22 12:55:18 host kernel: R13: ffff9b640077bc90 R14: ffffc0f338637680 R15: 0000000000000000 May 22 12:55:18 host kernel: FS: 0000000000000000(0000) GS:ffff8cbf3fc80000(0000) knlGS:0000000000000000 May 22 12:55:18 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 22 12:55:18 host kernel: CR2: ffffffffffffffd6 CR3: 000000101f826005 CR4: 0000000000770ee0 May 22 12:55:18 host kernel: PKRU: 55555554 May 22 13:07:05 host kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000 May 22 13:07:05 host kernel: #PF: supervisor instruction fetch in kernel mode May 22 13:07:06 host kernel: #PF: error_code(0x0010) - not-present page May 22 13:07:06 host kernel: PGD 0 P4D 0 May 22 13:07:06 host kernel: Oops: 0010 [0000002] PREEMPT SMP NOPTI May 22 13:07:06 host kernel: CPU: 24 PID: 249565 Comm: kworker/u64:5 Tainted: G D -------- --- 5.14.0-284.11.1.el9_2.x86_64 #1 May 22 13:07:06 host kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 May 22 13:07:06 host kernel: Workqueue: ceph-inode ceph_inode_work [ceph] May 22 13:07:06 host kernel: RIP: 0010:0x0 May 22 13:07:06 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6. May 22 13:07:06 host kernel: RSP: 0018:ffff9b6402043b80 EFLAGS: 00010246 May 22 13:07:06 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015 May 22 13:07:06 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffffc0f3243e3400 May 22 13:07:06 host kernel: RBP: ffff9b6402043d48 R08: 0000000000000078 R09: 00000000ffffffff May 22 13:07:06 host kernel: R10: ffffffffffffffff R11: ffff8cbf3ffd5d80 R12: 0000000000000001 May 22 13:07:06 host kernel: R13: ffff9b6402043c90 R14: ffffc0f3243e3400 R15: 0000000000000000 May 22 13:07:06 host kernel: FS: 0000000000000000(0000) GS:ffff8cbf3fc00000(0000) knlGS:0000000000000000 May 22 13:07:06 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 22 13:07:06 host kernel: CR2: ffffffffffffffd6 CR3: 00000006ce530004 CR4: 0000000000770ee0 May 22 13:07:06 host kernel: PKRU: 55555554 May 22 13:07:06 host kernel: Call Trace: May 22 13:07:06 host kernel: <TASK> May 22 13:07:06 host kernel: ceph_writepages_start+0x8bf/0x19b0 [ceph] May 22 13:07:06 host kernel: ? __blk_mq_try_issue_directly+0x168/0x1e0 May 22 13:07:06 host kernel: ? wb_calc_thresh+0x4f/0x70 May 22 13:07:06 host kernel: ? __blk_flush_plug+0x102/0x160 May 22 13:07:06 host kernel: ? blk_finish_plug+0x25/0x40 May 22 13:07:06 host kernel: do_writepages+0xcf/0x1d0 May 22 13:07:06 host kernel: ? newidle_balance+0x2e5/0x400 May 22 13:07:06 host kernel: ? update_load_avg+0x7e/0x730 May 22 13:07:06 host kernel: filemap_fdatawrite_wbc+0x66/0x90 May 22 13:07:06 host kernel: filemap_fdatawrite+0x4f/0x70 May 22 13:07:06 host kernel: ceph_inode_work+0x28/0xb0 [ceph] May 22 13:07:06 host kernel: process_one_work+0x1e8/0x3c0 May 22 13:07:06 host kernel: worker_thread+0x50/0x3b0 May 22 13:07:06 host kernel: ? rescuer_thread+0x3a0/0x3a0 May 22 13:07:06 host kernel: kthread+0xd9/0x100 May 22 13:07:06 host kernel: ? kthread_complete_and_exit+0x20/0x20 May 22 13:07:06 host kernel: ret_from_fork+0x22/0x30 May 22 13:07:06 host kernel: </TASK> May 22 13:07:06 host kernel: Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nft_counter nfsv3 nfs_acl nfs lockd grace xt_addrtype nft_compat br_netfilter bridge stp llc ceph libceph dns_resolver fscache netfs rfkill nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject overlay nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 isofs ip_set nf_tables nfnetlink binfmt_misc vfat fat intel_rapl_msr intel_rapl_common qxl drm_ttm_helper ttm kvm_amd drm_kms_helper ccp syscopyarea kvm sysfillrect irqbypass virtio_balloon i2c_piix4 sysimgblt pcspkr fb_sys_fops joydev sch_fq tcp_bbr drm auth_rpcgss xfs libcrc32c sr_mod cdrom ata_generic nvme_tcp nvme_fabrics nvme sd_mod nvme_core sg ata_piix nvme_common crct10dif_pclmul t10_pi crc32_pclmul libata crc32c_intel virtio_net net_failover virtio_console ghash_clmulni_intel failover virtio_scsi serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod fuse May 22 13:07:06 host kernel: CR2: 0000000000000000 May 22 13:07:06 host kernel: ---[ end trace e4febc6ade47e8cb ]--- May 22 13:07:06 host kernel: RIP: 0010:0x0 May 22 13:07:06 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6. May 22 13:07:06 host kernel: RSP: 0018:ffff9b640077bb80 EFLAGS: 00010246 May 22 13:07:06 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015 May 22 13:07:06 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffffc0f338637680 May 22 13:07:06 host kernel: RBP: ffff9b640077bd48 R08: 0000000000000078 R09: 00000000ffffffff May 22 13:07:06 host kernel: R10: ffffffffffffffff R11: ffff8cbf3ffd5d80 R12: 0000000000000001 May 22 13:07:06 host kernel: R13: ffff9b640077bc90 R14: ffffc0f338637680 R15: 0000000000000000 May 22 13:07:06 host kernel: FS: 0000000000000000(0000) GS:ffff8cbf3fc00000(0000) knlGS:0000000000000000 May 22 13:07:06 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 22 13:07:06 host kernel: CR2: ffffffffffffffd6 CR3: 00000006ce530004 CR4: 0000000000770ee0 May 22 13:07:06 host kernel: PKRU: 55555554 May 23 09:45:33 host kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000 May 23 09:45:33 host kernel: #PF: supervisor instruction fetch in kernel mode May 23 09:45:33 host kernel: #PF: error_code(0x0010) - not-present page May 23 09:45:33 host kernel: PGD 161766067 P4D 161766067 PUD 161767067 PMD 0 May 23 09:45:33 host kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI May 23 09:45:33 host kernel: CPU: 17 PID: 1244585 Comm: kworker/u64:0 Not tainted 5.14.0-284.11.1.el9_2.x86_64 #1 May 23 09:45:33 host kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 May 23 09:45:33 host kernel: Workqueue: ceph-inode ceph_inode_work [ceph] May 23 09:45:33 host kernel: RIP: 0010:0x0 May 23 09:45:33 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6. May 23 09:45:33 host kernel: RSP: 0018:ffffa648023b7b80 EFLAGS: 00010246 May 23 09:45:33 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015 May 23 09:45:33 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: fffff72068be0480 May 23 09:45:33 host kernel: RBP: ffffa648023b7d48 R08: 0000000000000078 R09: 00000000ffffffff May 23 09:45:33 host kernel: R10: ffffffffffffffff R11: ffff95093ffd5d80 R12: 0000000000000001 May 23 09:45:33 host kernel: R13: ffffa648023b7c90 R14: fffff72068be0480 R15: 0000000000000000 May 23 09:45:33 host kernel: FS: 0000000000000000(0000) GS:ffff9508ffa40000(0000) knlGS:0000000000000000 May 23 09:45:33 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 23 09:45:33 host kernel: CR2: ffffffffffffffd6 CR3: 0000000161730003 CR4: 0000000000770ee0 May 23 09:45:33 host kernel: PKRU: 55555554 May 23 09:45:33 host kernel: Call Trace: May 23 09:45:33 host kernel: <TASK> May 23 09:45:33 host kernel: ceph_writepages_start+0x8bf/0x19b0 [ceph] May 23 09:45:33 host kernel: ? fprop_reflect_period_percpu.isra.0+0x7b/0xc0 May 23 09:45:33 host kernel: do_writepages+0xcf/0x1d0 May 23 09:45:33 host kernel: ? newidle_balance+0x2e5/0x400 May 23 09:45:33 host kernel: ? update_load_avg+0x7e/0x730 May 23 09:45:33 host kernel: filemap_fdatawrite_wbc+0x66/0x90 May 23 09:45:33 host kernel: filemap_fdatawrite+0x4f/0x70 May 23 09:45:33 host kernel: ceph_inode_work+0x28/0xb0 [ceph] May 23 09:45:33 host kernel: process_one_work+0x1e8/0x3c0 May 23 09:45:33 host kernel: worker_thread+0x50/0x3b0 May 23 09:45:33 host kernel: ? rescuer_thread+0x3a0/0x3a0 May 23 09:45:33 host kernel: kthread+0xd9/0x100 May 23 09:45:33 host kernel: ? kthread_complete_and_exit+0x20/0x20 May 23 09:45:33 host kernel: ret_from_fork+0x22/0x30 May 23 09:45:33 host kernel: </TASK> May 23 09:45:33 host kernel: Modules linked in: tls xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfsv3 nft_counter nfs_acl nfs lockd grace xt_addrtype nft_compat br_netfilter bridge stp llc ceph libceph dns_resolver fscache netfs rfkill overlay nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack isofs nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink binfmt_misc vfat fat intel_rapl_msr intel_rapl_common kvm_amd qxl ccp drm_ttm_helper ttm kvm drm_kms_helper syscopyarea sysfillrect irqbypass sysimgblt i2c_piix4 virtio_balloon pcspkr fb_sys_fops joydev sch_fq tcp_bbr drm auth_rpcgss xfs libcrc32c sr_mod sd_mod cdrom sg ata_generic nvme_tcp nvme_fabrics nvme nvme_core nvme_common ata_piix t10_pi crct10dif_pclmul crc32_pclmul libata crc32c_intel virtio_net net_failover ghash_clmulni_intel virtio_console failover virtio_scsi serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod fuse May 23 09:45:33 host kernel: CR2: 0000000000000000 May 23 09:45:33 host kernel: ---[ end trace d3df7407caf4e3a6 ]--- May 23 09:45:33 host kernel: RIP: 0010:0x0 May 23 09:45:33 host kernel: Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6. May 23 09:45:33 host kernel: RSP: 0018:ffffa648023b7b80 EFLAGS: 00010246 May 23 09:45:33 host kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0017ffffc0002015 May 23 09:45:33 host kernel: RDX: 0000000000001000 RSI: 0000000000000000 RDI: fffff72068be0480 May 23 09:45:33 host kernel: RBP: ffffa648023b7d48 R08: 0000000000000078 R09: 00000000ffffffff May 23 09:45:33 host kernel: R10: ffffffffffffffff R11: ffff95093ffd5d80 R12: 0000000000000001 May 23 09:45:33 host kernel: R13: ffffa648023b7c90 R14: fffff72068be0480 R15: 0000000000000000 May 23 09:45:33 host kernel: FS: 0000000000000000(0000) GS:ffff9508ffa40000(0000) knlGS:0000000000000000 May 23 09:45:33 host kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 23 09:45:33 host kernel: CR2: ffffffffffffffd6 CR3: 0000000161730003 CR4: 0000000000770ee0 May 23 09:45:33 host kernel: PKRU: 55555554 |
