rtorrent should be redesigned to not use a chunk's worth of virtual memory for each download/upload connection

[luca.barbieri@gmail.com]

The current rtorrent design will keep an entire chunk mapped in memory for each download or upload or hash check of a chunk.

Unfortunately, torrents may have 4 MB chunks (and perhaps more), which means that a single peer connection may require 4 MB + 4 MB + 4 MB = 12 MB of virtual address space for upload, download and hash checking.

On a 32-bit machine, at most 2 GB can be likely used effectively, and even assuming only 8 MB per connection, this results in rtorrent supporting at most 256 active download+upload connections before throwing "Cannot allocate memory".

This is a serious problem, because assuming a 2 KB/s average download or upload rate, this results in a (2KB/s + 2KB/s) * 256 = 1MB/s transfer limit, which is barely enough to saturate a 10mbps link and insufficient for college/business/fiber-to-the-home 100mbps and 1gbps links.

Furthermore, in addition to the transfer speed problem, curtailing the number of open sockets allows to connect to less peers and not curtailing may result in rtorrent using up all memory, possibly causing problems or misprioritization of torrents.

Solving the issue requires a partial redesign of rtorrent. I suggest the following two approaches:

1. Have a "global memory chunk size limit" with a user-settable limit and have a "per-torrent memory chunk size" that would be the minimum of the global limit and the torrent chunk size. Chunks can then be mapped in "memory chunk size" sized pieces.

This does not "fix" the issue, but, for example, setting a 64 KB memory chunk size can allow to multiply by 64 (4 MB -> 64 KB) the current limits (leading to 16K connections and 512mbps throughput with 2KB/s per-connection speed, which may be enough for most).

Furthermore, each TCP connection already uses kernel memory buffer, so there is a limit anyway (unless the kernel can put TCP buffers in highmem and the machine has a lot of ram).

2. Have a mode that does not map upload chunks in memory but uses sendfile() (or read+write if sendfile is unavailable). This should be a good idea anyway.

3. Have a mode that does not map download chunks in memory but uses read()+hash check+write()

4. Investigate whether splice/tee/vmsplice can help.

5. On 64-bit platforms, set the default address space limit to a much higher value than 4 GB. A good choice could be to use the actual limit of the operating system, detected at runtime (it was 512 GB on past x86-64 Linux kernels, not sure if it has improved).

Of course, if the virtual address space is larger than a small multiple of the available disk space on rtorrent's partition, there is no problem (unless the virtual adddress space gets badly fragmented). BTW, on 64-bit machines it may be interesting to try to keep the whole files always mmaped.

[rakshasa]

On x86_64 you have the option to increase the VM size, I made it smaller than necessary by default to ensure it worked. For x86, I'll consider something in the future, not really a priority.

Though I doubt splitting the chunks up by too much would be a clean solution. And the problem with splice or that other linux system call i can't remember the name of, is that they might not be as efficient when transferring just 16kb at a time.

[anonymous]

I came across this while trying to find how much ram rtorrent was allocating. When downloading/uploading many files from many peers it was using a lot of ram (>200mb). Is this the method that other torrent clients use? Mem usage is one of the reasons I started using utorrent on my windows box. I don't know how it allocates memory, but it is better than every other client I've used and I hope something similar can be implemented in rtorrent.

[rakshasa]

AFAIK, windows doesn't count the equivalent of mmaped memory, so µtorrent doesn't show the right amount of memory usage. And other clients use a less efficient way by duplicating data in memory. You are just seeing the disk cache in action.

[anonymous]

other torrent clients have to use some other method, I connect to over 1000 peers using other clients, which is not possible on rtorrent. I hope this issue is fixed soon, because at the moment I cannot use rtorrent for my purposes

[Thehound]

I do just fine running rtorrent on a machine setup as a server with a high speed line(100/100). 512 MB Ram, Intel Pentium 4@2 GHz. VM Size is not actual application memory consumption, it's more like a max including cache. The fact it caches is good! It saves wear and tear on your drive. Cache can be aggressively swapped out when need be for another application under Linux. If this occurs slowly, you might adjust max.vm.swappiness parameter in sysctl.conf. I seemed to be connecting to few peers until I adjust the min peers and numwant parameters in rtorrent. You might want to try that, It's helped rtorrent connect to enough peers to make use of my line :) Good luck!

[Thehound]

vm.swappiness is the parameter. Sorry for double post. I just felt the need to be technically correct, I hope this helps people as I do think rtorrent manages memory quite well on all my machines ranging from a Pentium 3 to an Athlon FX-62.

[libtorrent@terra.chuui.jp]

just adding that I would like to see something done about this too.. and it doesn't seem like anyone has put any effort into resolving this.

Thehound: linux caches on it's own, and there is no reason for rtorrent to be using over 200MB of ram at anytime.

On my machine, which has 1GB of ram and 482MB of ram in use before rtorrent.. it is unacceptable for rtorrent to be using enough memory to force my machine to start using the swap just for torrenting. I go above 600 peers regularly on torrents (using Azureus). for rtorrent to do this successfully on a torrent with only 512K it requires 1.5MB per peer it would need 900MB of ram. Azureus on the other hand, even with it's java yuckyness, can do this with a mere 200 or so MB of ram (I have cache turned way up on Azureus.)

[Carl]

I made a post about this as well some days ago - it was removed, probably due this "thread" being active. I have the exact same problem; my machine has a normal memory usage of about 40mb prior to getting rtorrent up and running, and after a while almost all 512mb of the machine are used up, and swap-memory is put to use. I've never seen this massive memory use in any other torrent client, really. It simply makes rtorrent unusable, which is a shame considering how great a client it is.

[Incognito <in.incognito@gmail.com>]

I have the same problem. I end up restarting rtorrent every few hours because of this.

Current upload speed 17kb/s, 0kb/s download.

Its using 900mb of virtual mem, 80 shared, res is 350mb.

This is serious and should be a priority.

[rakshasa]

Replying to libtorrent@terra.chuui.jp:

Thehound: linux caches on it's own, and there is no reason for rtorrent to be using over 200MB of ram at anytime.

You are in fact seeing the 'linux cache', since rtorrent uses mmap.

On my machine, which has 1GB of ram and 482MB of ram in use before rtorrent.. it is unacceptable for rtorrent to be using enough memory to force my machine to start using the swap just for torrenting. I go above 600 peers regularly on torrents (using Azureus). for rtorrent to do this successfully on a torrent with only 512K it requires 1.5MB per peer it would need 900MB of ram. Azureus on the other hand, even with it's java yuckyness, can do this with a mere 200 or so MB of ram (I have cache turned way up on Azureus.)

Which means Azureus is using what rtorrent uses + 200MB of ram... Only thing that makes the azureus case acceptable at all, is that it doesn't touch the pages in the kernel file cache so they get thrown out quicker. (Which also means you get more disk activity)

A sane kernel will notice that stuff rtorrent maps, only gets touched once or twice, thus throwing it out before something important.

[anonymous]

So only Linux is sane, and all three BSD's and OS X is "insane"? Something tells me it's your method that's not the best approach, as no other torrent client out there devours RAM like rtorrent does. Transmission ( http://transmission.m0k.org) appears to have a different approach that is far more efficient and conservative with memory usage - I've used it on FreeBSD, Linux and OS X, and I can vouch that it does not hog like rtorrent does. It's also open source, so perhaps some refreshing ideas would shows up from checking the code out.

I hope this gets rectified soon so I can go back to using rtorrent again...

[anonymous]

Related, I guess: this behaviour also show up with torrent-hash'ing alone, without any connections being established. I just added a completed torrent of 1.4gb size for seeding. At the end of the hash'ing, before any peers were connected, almost all 512mb of RAM were used up, and an additional 14mb of swap memory had been put to use. These 14mb that had been swapped out belonged not just to rtorrent, but largely to other applications running - killing rtorrent released only 2mb of the swapped content. That's just not right.

[anonymous]

Apparently all you commenters don't understand how open source software works. He's not your bitch and is not going to fix all your problems right now because you are whining about your little torrent problem. Shut up and code, or shut up and give him time to look into it. Enough said.

[anonymous]

Replying to anonymous:

Apparently all you commenters don't understand how open source software works. He's not your bitch and is not going to fix all your problems right now because you are whining about your little torrent problem. Shut up and code, or shut up and give him time to look into it. Enough said.

Wow, what a clever addition... I suppose everyone should just stop feedbacking about bugs and err behaviour in order to perfectly align with your view of open source collaboration -- oh, wait, that'd actually steer towards closed source in a very backward kind of way, you Luddite.

[anonymous]

if you mean a feature is a bug then just dont use it.

[rakshasa]

If you see things get swapped out by the kernel after hashing, then it is the _kernel_ that messed up by not being able to figure out that the data was mostly read-once. And how do you know that what got swapped out is in fact _useful_ data?

This does not how ever have anything to do with the original ticket, so please stay on topic.

[anonymous]

it would be alot more efficient if memory was used on per piece basis instead of per peer.

[rakshasa]

It is used on a per piece basis...

[anonymous]

i might be wrong about this but, say 2-3 peers in a swarm, same torrent, request the same piece. The cache will contain 2-3 copies of that piece wouldn't it? It would be more efficient if it was just 1 copy.

[rakshasa]

Yes, you are wrong about it.

[anonymous]

It starts out looking very efficient on memory but creeps up and up and up during use. If one starts several torrents (some large) and goes to bed, the machine is nearly frozen in the morning due to memory usage. This is sad, as this program is excellent on the CPU side of the equation.

I tried to "ulimit" it and/or to use the "max_memory" rc setting (figure nearly double the actual usage you set there, by the way), but this just generates errors when rtorrent can't fulfill its insatiable thirst for more and more RAM. ("File chunk write error: Cannot allocate memory.")

[Marcus]

Any progress on this? I'd really like to get back to using rTorrent...

[Weedy <weedy2887@gmail.com>]

Ok am i missing something here? Cause I really don't see an issue.

$ ls -l torrents/watch/ |wc -l
46

top - 12:58:02 up 8 days, 17:16,  2 users,  load average: 0.00, 0.04, 0.03
Tasks:  85 total,   1 running,  84 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.3%sy,  0.0%ni, 97.0%id,  2.0%wa,  0.3%hi,  0.2%si,  0.0%st
Mem:   2075492k total,  2022672k used,    52820k free,    17900k buffers
Swap:  2048276k total,     5364k used,  2042912k free,  1696864k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 8927 weedy     20   0 37524  21m  12m S    0  1.0   5:13.52 rtorrent

                            *** rTorrent 0.8.2/0.12.2 - Chii_v2:8927 ***
[Throttle off/off KB] [Rate 511.6/ 29.6 KB] [Port: 37401] [U 16/0] [D 5/0] [H 0/32] [S 1/47/768] [F 125/128]

[anonymous]

You probably don't have any active downloads with a large (2MB+) chunk size.

[Weedy <weedy2887@gmail.com>]

2 torrents over 5 gigs each 4mb chunk size.

[anonymous]

To call posix_fadvise (fd, chunk_start, chunk_len, POSIX_FADV_DONTNEED) after the chunk is saved, should help from thrashing memory by rtorrent. I don't see the reason why a newly downloaded chunk should be kept in memory evicting some other useful pages from cache.

[anonymous]

It worked perfectly for me.  tupperware -  hoosier

[Kate Hanley <submission@wewanttraffic.com>]

I really liked your post, it really added a great perspective on the matter. Thank you.

 Exhibition Planning Management |  real estate in Dubai |  buy Dubai property |  Dubai apartments |  Dubai property

[Kate Hanley <submission@wewanttraffic.com>]

Its wonderful, looking at the time and effort you put into your weblog and detailed information you provide. I'll bookmark your weblog and visit it weekly for your new posts.

 abu dhabi villa |  Web Designing in Dubai |  SEO in Dubai