OOM Killer caused by disk I/O during file transfer
Hello, it's Munou.
A few days have passed since I built a home server cheaply in my new house. A problem I'd been casually wondering about for a while occurred even with plenty of memory.
What happened?
My current environment is a bit special, or rather, the memory capacity is an odd 16GB + 8GB, but please understand that I built it with leftover parts.
The thing is, with 24GB of RAM, which might be common now, I personally thought it was quite a rich state and that OOM wouldn't occur, but an OOM (Out Of Memory) event, like a scolding from the Linux kernel, occurred.
Applications are just Apache, php-fpm, Redis, etc., and the memory each of them allocates is negligible...
Troubled by File Transfers
It happened when I tried to fetch files via scp from my home server still located at my parents' house, or when I was batch-saving images to a 2TB HDD using aria2c to store a certain amount of images.
Now, why would OOM occur with just this write process? This is because files are first received in memory and then written to the storage area, so to speed things up, they need to be accepted by the memory as a cache first.
It seems that when transferring files of several hundred GBs using scp -r, especially if those files involve random I/O operations, it really consumes resources.
And since this isn't considered per-process memory usage, it can't be confirmed by checking with `top` or similar tools for a single process; it's purely a Linux kernel-side issue, making it quite difficult to pinpoint.
Simply put, it seems this happens because the amount of data temporarily stored in memory due to I/O wait can't keep up with the writing process, leading to a gradual depletion of resources.
Modifiable Kernel Parameters
As something the kernel can do, it seems possible to configure how much to cache, etc., and while I tried adjusting it, I didn't feel much effect.
2.2. VFS チューニングオプション: リサーチと実験 Red Hat Product Documentation
Linuxページキャッシュの設定を変更してWrite I/Oをチューニングしたメモ- YOMON8.NET
The phenomenon is similar to what this person experienced.
OOM Killer invoked due to slow disk I/O - #17 by Whis-key - troubleshooting - Storj Community Forum (official)
I saw the word Storj after a long time... though my free tier was deleted without my permission...
I can't believe I'm struggling with memory exhaustion just from writing to an HDD... This is also where the common saying about memory comes in: "Linux is optimistic about RAM, Windows is pessimistic." While I understand the intention to fully utilize memory for more efficient performance, I really want to find a way to deal with this problem caused by I/O wait.