View Issue Details

IDProjectCategoryView StatusLast Update
0000176AlmaLinux-8rsyncpublic2022-01-24 10:05
Reporterslayerduck Assigned To 
PriorityurgentSeveritymajorReproducibilityalways
Status newResolutionopen 
OSAlmalinuxOS Version8.5 
Summary0000176: Rsync memleak percpu
DescriptionI’m using lsyncd to keep folders up to date between servers. This calls upon rsync and thus I’m doing a lot of rsyncs, a couple 1000 per hour. After upgrading from C8 to Almalinux my ram is filling up on destination servers and that eventually crashes the entire server. When i check /proc/meminfo the “percpu” is the one that fills up.

I tried swtiching kernel to elrepo’s and this did not fix the issue. It has to be some lib that was replaced when i converted. I can reproduce this on all servers i update to alma, the more rsyncs i do the faster it fills up. Cloudlinux also has the issue.

Munin mem attached of two production servers, one was actually crashed the other one with 256GB ram nearly when this issue was found.
Steps To Reproduce#! /bin/bash
# Example test for percpu/rsync memleak. Needs two servers, source can be any linux with rsync. Destination with CD or Alma will leak percpu.
# Can reproduce in Cloudlinux and AlmaLinux

# STEP 1: Generate random files for rsync. CD into any empty DIR
workdir=/opt/test3/
cd $workdir

for n in {1..5000}; do
    dd if=/dev/urandom of=file$( printf %03d "$n" ).bin bs=1 count=$(( RANDOM + 1024 ))
done

# STEP 2: Generate SSH keys and make sure you can SSH into remote machine (Alma or Cloud)
ls $workdir | xargs -n1 -P10 -I% rsync -Pa % 192.168.0.212:$workdir

# Open new window, check: cat /proc/meminfo | grep Percpu
# running rm -rf on dir with tranfered files will lower Percpu again. Otherwise no other way i know of so far to reduce mem besides reboot. Dropping caches does not work.
TagsNo tags attached.
abrt_hash
URL

Activities

slayerduck

2022-01-22 18:11

reporter  

memory-week.png (48,859 bytes)   
memory-week.png (48,859 bytes)   
memory-week-crashed.png (54,778 bytes)   
memory-week-crashed.png (54,778 bytes)   

alukoshko

2022-01-24 08:40

administrator   ~0000480

Hello. Have you tried the same with RHEL or CentOS 8.5?

slayerduck

2022-01-24 10:05

reporter   ~0000481

I have today, and rhel 8.5 is showing the same symptoms. If i replace rsync with scp the bug occurs still to. So this doesn't seem like an rsync issue.
It also seems like it takes some time for it to go all the way up, if i try to do it on a small VM with 2gb ram it fills up the whole server then somehow does drops back down. So it might only be an issue with machines with larger amounts of ram or over a longer time.

scp or rsyncing 100k files/2gb data with 8 threads, server with 128GB ram: Percpu: 8447616 kB

Issue History

Date Modified Username Field Change
2022-01-22 18:11 slayerduck New Issue
2022-01-22 18:11 slayerduck File Added: memory-week.png
2022-01-22 18:11 slayerduck File Added: memory-week-crashed.png
2022-01-24 08:40 alukoshko Note Added: 0000480
2022-01-24 10:05 slayerduck Note Added: 0000481