Ticket #641 (new enhancement)
rtorrent should be redesigned to not use a chunk's worth of virtual memory for each download/upload connection
|Reported by:||email@example.com||Owned by:||rakshasa|
The current rtorrent design will keep an entire chunk mapped in memory for each download or upload or hash check of a chunk.
Unfortunately, torrents may have 4 MB chunks (and perhaps more), which means that a single peer connection may require 4 MB + 4 MB + 4 MB = 12 MB of virtual address space for upload, download and hash checking.
On a 32-bit machine, at most 2 GB can be likely used effectively, and even assuming only 8 MB per connection, this results in rtorrent supporting at most 256 active download+upload connections before throwing "Cannot allocate memory".
This is a serious problem, because assuming a 2 KB/s average download or upload rate, this results in a (2KB/s + 2KB/s) * 256 = 1MB/s transfer limit, which is barely enough to saturate a 10mbps link and insufficient for college/business/fiber-to-the-home 100mbps and 1gbps links.
Furthermore, in addition to the transfer speed problem, curtailing the number of open sockets allows to connect to less peers and not curtailing may result in rtorrent using up all memory, possibly causing problems or misprioritization of torrents.
Solving the issue requires a partial redesign of rtorrent. I suggest the following two approaches:
- Have a "global memory chunk size limit" with a user-settable limit and have a "per-torrent memory chunk size" that would be the minimum of the global limit and the torrent chunk size. Chunks can then be mapped in "memory chunk size" sized pieces.
This does not "fix" the issue, but, for example, setting a 64 KB memory chunk size can allow to multiply by 64 (4 MB -> 64 KB) the current limits (leading to 16K connections and 512mbps throughput with 2KB/s per-connection speed, which may be enough for most).
Furthermore, each TCP connection already uses kernel memory buffer, so there is a limit anyway (unless the kernel can put TCP buffers in highmem and the machine has a lot of ram).
- Have a mode that does not map upload chunks in memory but uses sendfile() (or read+write if sendfile is unavailable). This should be a good idea anyway.
- Have a mode that does not map download chunks in memory but uses read()+hash check+write()
- Investigate whether splice/tee/vmsplice can help.
- On 64-bit platforms, set the default address space limit to a much higher value than 4 GB. A good choice could be to use the actual limit of the operating system, detected at runtime (it was 512 GB on past x86-64 Linux kernels, not sure if it has improved).
Of course, if the virtual address space is larger than a small multiple of the available disk space on rtorrent's partition, there is no problem (unless the virtual adddress space gets badly fragmented). BTW, on 64-bit machines it may be interesting to try to keep the whole files always mmaped.