View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000649 | AlmaLinux-9 | kernel | public | 2026-07-13 12:42 | 2026-07-13 12:42 |
| Reporter | aslefnad7 | Assigned To | |||
| Priority | high | Severity | crash | Reproducibility | have not tried |
| Status | new | Resolution | open | ||
| Platform | R740, R750xa, R640 | OS | AlmaLinux | OS Version | 9.8 |
| Summary | 0000649: kernel BUG in folio_end_writeback via cifs_writepages on close() | ||||
| Description | -------------------------------------------------------------------------------- SUMMARY -------------------------------------------------------------------------------- 5.14.0-687.15.1.el9_8 (also reproduced on -687.17.1 and -687.20.1) — kernel BUG at mm/filemap.c:1593 in folio_end_writeback, reached from cifs_writepages during a synchronous writeback flush issued by the CIFS code itself: in most dumps from cifs_flush on close(), in two dumps from cifs_open on open(). Seen in the same event are a refcount_warn_saturate (refcount underflow / use-after-free) in the CIFS async write path and a socket send failure (Error -32 / EPIPE) to the server. The crash(8) disassembly confirms the BUG is the "if (!__folio_end_writeback(folio)) BUG();" case — i.e. writeback is ended on a folio whose writeback flag was already clear (a double-completion of writeback). The same underlying corruption also manifests as a second signature: a general protection fault on a non-canonical address with RIP in unlock_page+0x5, again directly under cifs_writepages, with the identical refcount-WARN + Error -32 preamble (4 of 17 dumps). The crash reproduces on fully UNTAINTED kernels: two of the four affected machines carry no out-of-tree modules at all ("Not tainted" in the dumps). Severity: crash / kernel panic Category: kernel -------------------------------------------------------------------------------- FOUR INDEPENDENT MACHINES, THREE KERNEL BUILDS, IDENTICAL SIGNATURE -------------------------------------------------------------------------------- The panic has now been captured on four physically different servers (three different Dell models) across three el9_8 kernel builds. 17 kdumps total. The call traces match down to the same symbol offsets. hostA Server : Dell PowerEdge R740, BIOS 2.26.1 Kernels : 5.14.0-687.15.1.el9_8 Comm : dask worker [tc (e.g. PID 70950, CPU 22) Hardware: 40 CPUs, 766.6 GB RAM CIFS : cache=none Taint : P OE (NVIDIA out-of-tree module loaded) Dumps : 5 (2026-06-20 .. 2026-06-23) hostB Server : Dell PowerEdge R750xa, BIOS 1.20.2 Kernels : 5.14.0-687.15.1.el9_8 and 5.14.0-687.17.1.el9_8 Comm : pt_main_thread / python Hardware: 24 CPUs, 383.2 GB RAM CIFS : cache=strict Taint : P OE (NVIDIA out-of-tree module loaded) Dumps : 6 (2026-06-20 .. 2026-07-11; the three July dumps are on -687.17.1) hostC Server : Dell PowerEdge R740, BIOS 2.26.1 Kernels : 5.14.0-687.15.1.el9_8 and 5.14.0-687.20.1.el9_8 Comm : dask worker [tc Taint : NOT TAINTED — no out-of-tree modules Dumps : 3 (all 2026-07-03; one of them on -687.20.1) hostD Server : Dell PowerEdge R640, BIOS 2.26.1 Kernels : 5.14.0-687.20.1.el9_8 Comm : python Taint : NOT TAINTED — no out-of-tree modules Dumps : 3 (all 2026-07-03) Kernel build string (hostA/hostB June dumps): Linux version 5.14.0-687.15.1.el9_8.x86_64 (mockbuild@x64-builder01.almalinux.org) ... #1 SMP PREEMPT_DYNAMIC Thu Jun 11 08:51:45 EDT 2026 Four different servers of three models crashing at the identical instruction with the identical call chain points to a software defect, not a per-machine hardware fault. -------------------------------------------------------------------------------- NOT AN OUT-OF-TREE-MODULE ISSUE — UNTAINTED REPRODUCTIONS -------------------------------------------------------------------------------- Earlier dumps (hostA/hostB) carried the "P ... OE" taint from the NVIDIA driver. The six hostC/hostD dumps rule that out as a factor: all of them crash with CPU: 10 PID: 97317 Comm: dask worker [tc Kdump: loaded Not tainted 5.14.0-687.15.1.el9_8.x86_64 #1 (hostC) CPU: 12 PID: 6801 Comm: python Kdump: loaded Not tainted 5.14.0-687.20.1.el9_8.x86_64 #1 (hostD) On the subsequent BUG oops line these hosts show "Tainted: G W" — the W is set by the refcount_warn_saturate WARNING that fires immediately before the BUG in the very same event; G means no proprietary/out-of-tree modules were ever loaded. The bug therefore reproduces on a pristine distribution kernel. -------------------------------------------------------------------------------- WHAT THE DUMPS SHOW -------------------------------------------------------------------------------- In all 17 dumps the crash is a two-part event on the same CPU/PID. The two parts are shown below in the order the report discusses them; the order in which they were actually printed to the kernel log varies (see "A note on ordering" at the end of this section). 1) refcount underflow in the CIFS async write path refcount_t: underflow; use-after-free. WARNING: CPU: <cpu> PID: <pid> at lib/refcount.c:28 refcount_warn_saturate+0xba/0x110 ... Tainted: P OE 5.14.0-687.15.1.el9_8.x86_64 (hostA/hostB) ... Not tainted 5.14.0-687.20.1.el9_8.x86_64 (hostC/hostD) Hardware name: Dell Inc. PowerEdge R740, BIOS 2.26.1 RIP: 0010:refcount_warn_saturate+0xba/0x110 Call Trace: refcount_warn_saturate+0xba/0x110 cifs_call_async+0x1ee/0x330 [cifs] smb2_async_writev+0x473/0x6f0 [cifs] ? __pfx_cifs_writedata_release+0x10/0x10 [cifs] cifs_writepages+0x52f/0xbc0 [cifs] do_writepages+0xd5/0x1b0 cifs_flush+0x73/0x120 [cifs] __x64_sys_close+0x2e/0x80 The release function on the stack is cifs_writedata_release — i.e. the object whose refcount saturates/underflows is the CIFS writedata associated with this writeback. 2) socket send failure, and the BUG CIFS: VFS: \\<fileserver> Error -32 sending data on socket to server ------------[ cut here ]------------ kernel BUG at mm/filemap.c:1593! invalid opcode: 0000 [#1] PREEMPT SMP NOPTI ... Tainted: P W OE 5.14.0-687.15.1.el9_8.x86_64 Hardware name: Dell Inc. PowerEdge R740, BIOS 2.26.1 RIP: 0010:folio_end_writeback+0x7d/0x80 Call Trace: folio_end_writeback+0x7d/0x80 cifs_writepages+0x998/0xbc0 [cifs] do_writepages+0xd5/0x1b0 filemap_fdatawrite_wbc+0x66/0x90 filemap_write_and_wait_range+0x3e/0xb0 cifs_flush+0x73/0x120 [cifs] __x64_sys_close+0x2e/0x80 BUG at mm/filemap.c:1593 inside folio_end_writeback means writeback is being ended on a folio that is no longer flagged as under writeback — the folio's writeback state has already been completed/torn down once. Combined with the refcount underflow on the CIFS writedata and the socket send failure (Error -32 / EPIPE) seen in the same event, the signature is a use-after-free / double-completion of the writeback in the SMB2 async write path, occurring during a synchronous CIFS flush. We read the failed/aborted send during writeback as the likely trigger, but state this as a hypothesis rather than a proven sequence (see "A note on ordering"). Matching offsets on all machines: cifs_call_async+0x1ee, smb2_async_writev+0x473, cifs_writepages+0x52f (send path) and +0x998 (writeback-end path), folio_end_writeback+0x7d, BUG at mm/filemap.c:1593. Socket error code: across all 17 dumps the ONLY error ever logged is "Error -32 sending data on socket to server" (20 occurrences total). No other error code appears. A note on ordering The three symptoms — the refcount underflow (refcount_warn_saturate), the socket send failure (CIFS: VFS: ... Error -32 sending data on socket), and the BUG in folio_end_writeback — appear together in every dump, but their PRINTED order is not stable (it differs between hosts and even between different dumps of the same host). We therefore do NOT claim a proven cause-and-effect order between the send failure and the refcount underflow. printk ordering on a many-CPU box under load does not necessarily reflect the true event timeline, and the send-error and refcount messages originate in different contexts. What is consistent across all dumps is the end state: a CIFS writeback whose state is ended twice (__folio_end_writeback() returns false -> BUG), in a synchronous CIFS flush path, in the presence of a failing/aborted SMB2 send. The "failed send during writeback triggers a bad writeback completion" reading is our best-supported hypothesis, not a fact the logs establish on their own. -------------------------------------------------------------------------------- SECOND MANIFESTATION — GPF ON A NON-CANONICAL ADDRESS IN unlock_page -------------------------------------------------------------------------------- 4 of the 17 dumps (hostB x3, hostA x1) do not reach the BUG in folio_end_writeback but die a few instructions earlier with a general protection fault, dereferencing a garbage folio pointer: general protection fault, probably for non-canonical address 0xd0bbf42d90685ec1: 0000 [#1] PREEMPT SMP NOPTI RIP: 0010:unlock_page+0x5/0x40 Call Trace: ? unlock_page+0x5/0x40 cifs_writepages+0x585/0xbc0 [cifs] do_writepages+0xd5/0x1b0 filemap_fdatawrite_wbc+0x66/0x90 __filemap_fdatawrite_range+0x54/0x80 filemap_write_and_wait_range+0x3e/0xb0 cifs_flush+0x73/0x120 [cifs] filp_flush+0x2e/0x80 __x64_sys_close+0x2e/0x80 Preamble (refcount_warn_saturate + Error -32) and entry path are identical to the BUG variant; only the point of death differs (unlock_page on an already-freed/poisoned folio vs. folio_end_writeback on a folio whose writeback bit is already clear). Both are consistent with the same use-after-free / double-completion of the CIFS writeback state. -------------------------------------------------------------------------------- NOT ONLY close() — SECOND SYSCALL ENTRY PATH VIA open() -------------------------------------------------------------------------------- 15 of the 17 dumps enter the fatal writeback via the flush on close() (cifs_flush -> filp_flush -> __x64_sys_close). Two hostC dumps (both 2026-07-03, one of them on an untainted -687.15.1, comm "dask worker") enter it via open() instead: do_writepages+0xd5/0x1b0 filemap_fdatawrite_wbc+0x66/0x90 __filemap_fdatawrite_range+0x54/0x80 filemap_write_and_wait_range+0x3e/0xb0 cifs_open+0x56e/0xc70 [cifs] do_dentry_open+0x14f/0x440 do_open+0x219/0x400 path_openat+0x110/0x280 do_filp_open+0xb0/0x160 do_sys_openat2+0x96/0xd0 __x64_sys_openat+0x53/0xa0 The defect is therefore not tied to the close() syscall specifically, but to the synchronous filemap_write_and_wait_range() flushes that the CIFS code issues itself (so far observed from cifs_flush and cifs_open). We have NOT observed the crash from fsync/msync, background writeback (kworker/flush) or memory reclaim in any dump. -------------------------------------------------------------------------------- BOTH CACHE MODES AFFECTED -------------------------------------------------------------------------------- hostA and hostB run different CIFS cache modes, and each crashed with the same confirmed signature: - hostA (R740): cache=none - hostB (R750xa): cache=strict This is visible in the stacks themselves. On hostB (cache=strict), the writeback path goes through cifs_strict_writev: cifs_writepages+0x52f/0xbc0 [cifs] ? generic_perform_write+0x14c/0x210 do_writepages+0xd5/0x1b0 ? cifs_strict_writev+0x1c4/0x330 [cifs] <-- strict path ... cifs_writepages+0x998/0xbc0 [cifs] ? cifs_strict_writev+0x1c4/0x330 [cifs] <-- strict path do_writepages+0xd5/0x1b0 cifs_flush+0x73/0x120 [cifs] The cifs_strict_writev frame is present on hostB and absent on hostA, confirming the two crashes reach the same folio_end_writeback double-completion through two different CIFS write paths (strict vs. none). The bug therefore does NOT depend on a single cache mode; it is in the writeback-completion handling common to both. -------------------------------------------------------------------------------- crash(8) ANALYSIS — VERIFIED SOURCE LINE AND FULL KERNEL STACK -------------------------------------------------------------------------------- crash(8) sessions over the full vmcores of hostA and hostB (with vmlinux debuginfo for 5.14.0-687.15.1.el9_8) confirm the source line and the complete in-kernel stack that the redacted dmesg only shows by offset. The disassembly of folio_end_writeback shows the BUG() at line 1593 is reached directly from the "if (!__folio_end_writeback(folio)) BUG();" path — i.e. __folio_end_writeback() returned false because the folio's PG_writeback flag was already clear: 1591 folio_get(folio); 1592 if (!__folio_end_writeback(folio)) ... call __folio_end_writeback ; test %al,%al ; je <+125> 1593 BUG(); 0xffffffff8137921d <+125>: ud2 Resolved kernel-side backtrace (hostA; hostB is identical bar addresses): [exception RIP: folio_end_writeback+125] 0000008 cifs_writepages [cifs] 0000009 do_writepages 0000010 filemap_fdatawrite_wbc 0000011 __filemap_fdatawrite_range 0000012 filemap_write_and_wait_range #13 cifs_flush [cifs] 0000014 filp_flush 0000015 __x64_sys_close 0000016 do_syscall_64 0000017 entry_SYSCALL_64_after_hwframe This matches the dmesg-derived signature exactly and pins the BUG to the __folio_end_writeback()-returns-false (writeback already ended) case. Note for reviewers: in the slab PAGE lines of the attachments, MAPPING values of the form dead0000000004xx / dead000000000001 are the normal kernel slab page-poison marker (page->mapping for a slab page), NOT corruption and NOT a redaction. -------------------------------------------------------------------------------- CIFS MOUNT OPTIONS (anonymized) -------------------------------------------------------------------------------- Requested options (autofs map; same on all hosts): sec=krb5,cruid=${UID},uid=${UID},gid=${GID},forceuid,forcegid, iocharset=utf8,vers=3.0,nosuid,nodev,dir_mode=0700,file_mode=0700, noserverino,cache=none Effective options after expansion (from the hostA/hostB dumps), anonymized: hostA (R740) — cache=none: rw,nosuid,nodev,relatime,vers=3.0,sec=krb5,cruid=<uid>,cache=none, upcall_target=app,username=<username>,uid=<uid>,forceuid,gid=<gid>, forcegid,addr=<fileserver-ip>,file_mode=0700,dir_mode=0700, iocharset=utf8,soft,nounix,mapposix,reparse=nfs,nativesocket, symlink=native,rsize=4194304,wsize=4194304,bsize=1048576, echo_interval=60,actimeo=1,closetimeo=1 hostB (R750xa) — cache=strict: rw,nosuid,nodev,relatime,vers=3.0,sec=krb5,cruid=<uid>,cache=strict, upcall_target=app,username=<username>,uid=<uid>,forceuid,gid=<gid>, forcegid,addr=<fileserver-ip>,file_mode=0700,dir_mode=0700, iocharset=utf8,soft,nounix,mapposix,reparse=nfs,nativesocket, symlink=native,rsize=4194304,wsize=4194304,bsize=1048576, echo_interval=60,actimeo=1,closetimeo=1 Anonymized fields: cruid/uid -> <uid>, gid -> <gid>, username -> <username>, addr -> <fileserver-ip>. All other values are verbatim. The two hosts differ only in the cache mode (none vs. strict). Why these may matter to maintainers (context, not a diagnosis): - cache mode. Same signature under cache=none (hostA) and cache=strict (hostB), so the bug is not specific to one writeback path; see "Both cache modes affected". - vers=3.0 (not 3.1.1) and sec=krb5 narrow down which SMB2/3 code path and auth path are in use. - soft, actimeo=1, closetimeo=1. A soft mount with very short attribute/close timeouts means operations abort with errors sooner under network pressure — consistent with the Error -32 / "send fails mid-writeback" condition seen in the dumps. - rsize/wsize=4194304 (4 MiB). Large I/O sizes, relevant to how writeback is chunked. We note these as potentially relevant context; we are not claiming any single option is the proven cause. -------------------------------------------------------------------------------- UPSTREAM CONTEXT (known class of bug; exact fix status for el9 is an open question) -------------------------------------------------------------------------------- This is consistent with a previously reported class of CIFS writeback-completion bugs rather than a brand-new defect: - On the linux-cifs list (Feb 2024), Steve French reported the same call path (cifs_flush -> filemap_write_and_wait_range -> ... -> folio_end_writeback) hit under xfstests when writes time out due to repeatedly failing server reconnects. Matthew Wilcox's reading was that the bug is something failing to call folio_end_writeback() when it should — i.e. a mismatch in writeback-end accounting, which is exactly the __folio_end_writeback()-returns-false signature seen here. Our trigger (Error -32 / EPIPE on the socket send during the close() flush) is the same "send/reconnect fails mid-writeback" condition. Thread: https://www.spinics.net/lists/linux-cifs/msg31067.html - The CIFS writeback path saw significant churn upstream: a two-phase cifs_writepages_begin / cifs_extend_writeback design (around f3dc1bdb6b0b "cifs: Fix writeback data corruption"), later a cut-over to netfslib (3ee1a1fc3981), plus follow-up fixes (e.g. a per-cifs_writepages_begin folio leak fix by Yang Erkun). Several of these touch exactly the folio/writeback refcounting on the write path. We have NOT been able to identify a single specific upstream commit and confirm whether it is present in or missing from the el9_8 kernels. Note that the crash reproduces unchanged on 5.14.0-687.20.1.el9_8 (the newest build we have in production), so whatever fix is needed is still missing as of that build. Determining whether el9's backport of the CIFS writeback path already contains the relevant fix (or needs one) is the key question for the maintainers, and requires checking the el9 kernel changelog against the upstream CIFS/netfs writeback fixes above. We're happy to provide the full vmcore(s) and test further. -------------------------------------------------------------------------------- ATTACHMENTS -------------------------------------------------------------------------------- 14 files — for each of seven representative dumps a vmcore-dmesg (.txt) and the matching kexec/kdump boot log (.log). Naming: <host>-<kernel>-<signature>, where 151/171/201 = 5.14.0-687.15.1/-687.17.1/-687.20.1.el9_8, "bug" = kernel BUG at mm/filemap.c:1593, "gpf" = GPF variant in unlock_page, "openat" = BUG variant entered via open() instead of close(): hostA-151-bug.txt/.log R740, tainted P OE, BUG, close(), 2026-06-23 hostB-151-bug.txt/.log R750xa, tainted P OE, BUG, close(), 2026-06-21 hostB-171-bug.txt/.log R750xa, BUG on -687.17.1, close(), 2026-07-11 hostB-171-gpf.txt/.log R750xa, GPF in unlock_page, close(), 2026-07-09 hostC-151-openat.txt/.log R740, NOT TAINTED, BUG, open() entry, 2026-07-03 hostC-201-bug.txt/.log R740, NOT TAINTED, BUG on -687.20.1, 2026-07-03 hostD-201-bug.txt/.log R640, NOT TAINTED, BUG on -687.20.1, 2026-07-03 Full vmcores for all 17 dumps are retained and available on request. | ||||
| Steps To Reproduce | 1. Mount SMB/CIFS shares from a file server (//<fileserver>/<share>) with the options listed under "CIFS mount options" above (vers=3.0, sec=krb5, soft). Seen under both cache=none (hostA) and cache=strict (hostB). 2. Run a userspace workload that writes heavily to files on the CIFS mount and then triggers a synchronous flush of those files. In 15 of 17 dumps the flush is the one issued on close(); in 2 dumps (hostC) it is the flush that cifs_open() itself issues on open(). Crashing processes are dask worker, pt_main_thread and python; all hosts were under load, hostB notably so (load average ~14-15). 3. In the flush path (__x64_sys_close -> cifs_flush -> do_writepages -> cifs_writepages, or __x64_sys_openat -> ... -> cifs_open -> filemap_write_and_wait_range -> do_writepages -> cifs_writepages), in the presence of an SMB2 async write that hits a socket send failure (CIFS: VFS: ... Error -32 sending data on socket), the dumps show a CIFS writedata refcount underflow (refcount_warn_saturate) followed by one of two fatal outcomes inside cifs_writepages: a) folio_end_writeback() on a folio whose writeback has already been ended (__folio_end_writeback() returns false) -> BUG at mm/filemap.c:1593 (13 of 17 dumps), or b) unlock_page() on an already-freed folio pointer -> general protection fault on a non-canonical address, RIP unlock_page+0x5 (4 of 17 dumps). Either way the machine panics. (See "A note on ordering": the precise sequence of the send failure vs. the refcount underflow is not established by the logs.) | ||||
| Additional Information | - 17 crashes across four servers (three Dell models: R740, R750xa, R640) and three kernel builds: 5.14.0-687.15.1.el9_8 (built Thu Jun 11 08:51:45 EDT 2026, per the dump banner), 5.14.0-687.17.1.el9_8 and 5.14.0-687.20.1.el9_8. The newest build we run (-687.20.1) is still affected. - Two of the four machines (hostC, hostD) run NO out-of-tree modules and crash with "Not tainted" kernels — the NVIDIA taint on hostA/hostB is not a factor. - The fault is in the SMB client (cifs) async write / writeback-completion path; the mm BUG (and, in 4 dumps, the GPF in unlock_page) is the symptom of the writeback being completed twice / the writeback state being freed while still referenced. - The crashes span both CIFS cache modes (none and strict), two syscall entry points into the flush (close() and open()), three server models and untainted plus tainted kernels — which argues against a per-machine, per-cache-mode or third-party-module cause. - Across all 17 dumps the only socket error ever logged is "Error -32 sending data on socket to server" (EPIPE; 20 occurrences). - We have NOT observed the crash from fsync/msync, background writeback (kworker/flush) or memory reclaim; every dump enters the fatal writeback through a synchronous flush issued by the CIFS code itself. - Redacted vmcore-dmesg.txt + kexec-dmesg.log for seven representative dumps (see ATTACHMENTS) and redacted crash(8) analyses for hostA/hostB are attached. Full vmcore images for all 17 crashes can be provided privately on request. | ||||
| Tags | No tags attached. | ||||
| Attached Files | |||||
