rtorrent crashes while checking hashes on large files

[cxd@maximilianeum.ch]

rtorrent crashes when checking the hashes on large torrents with the following error:

[ :0] Hashing: Checking hash [30%]                                                                                                                                                  
Caught Bus error, dumping stack:0/  0.0 KB] [Port: 9082]                                                                             [U 0/6] [D 0/0] [H 0/32] [S 0/3/768] [F 22/128]
Stack dump not enabled.                                                                                                                                                             
A bus error probably means you ran out of diskspace.                                                                                                                                
Aborted

However the disks still have plenty of space left (~15GB on the disk rtorrent is running on). The only way to fix this problem is by deleting the torrent, which is really bad.

[josef]

Apparently your system's freeaddrinfo implementation doesn't accept NULL arguments. However since that isn't standards-compliant usage anyway, it should be considered a bug in rtorrent.

Try this patch, it should fix it:  http://ovh.ttdpatch.net/~jdrexler/rt/fix_scgi_crash.diff

[josef]

Crap, replied to the wrong ticket. Disregard the above.

[josef]

What rtorrent/libtorrent version? What machine? What OS? What compiler was used to compile rtorrent/libtorrent? How big is the torrent?

While it's hashing, observe the memory usage in the download info. Is it stable or does it keep rising? If it's stable, check again with `top'.

[anonymous]

I seem to have run into the same problem. I am running rtorrent/libtorrent from svn (build 1087) compiled with
gcc (g++) (Ubuntu 4.3.2-1ubuntu12) 4.3.2 on Ubuntu 8.10, kernel 2.6.27-7.16.

After a start of a single torrent of size 4378.6MB rtorrent aborted with the following
message:

[ :0] Hashing: Checking hash [ 8%]

Caught Bus error, dumping stack:0/  0.0 KB] [Port: 6890]

Stack dump not enabled.

A bus error probably means you ran out of diskspace.

Aborted

On a second attempt rtorrent crashed again with a slightly more verbose output:

[ :0] Hashing: Checking hash [ 9%]

Caught Bus error, dumping stack:1/  0.1 KB] [Port: 6890]

0 /usr/local/bin/rtorrent [0x807dc81]

1 /usr/local/bin/rtorrent [0x8082380]

2 [0xb8089400]

3 /usr/lib/i686/cmov/libcrypto.so.0.9.8(sha1_block_asm_data_order+0x25) [0xb7c87e25]

A bus error probably means you ran out of diskspace.

Aborted

Here is (a part of) what the info page showed at the time of crash:

File stats:         single 1 files

Chunks:             0/1095*4194304

Priority:           2

Peer exchange:      enabled active (0/8)

State changed:      0:05:01

Memory usage:       120.0MB

Max memory usage:   819.2MB

Free diskspace:   55517.9MB

Safe diskspace:     632.0MB

Note the value of "Memory usage". Normally, with min/max = 5/30 I would not get more than 10.0-12.0 MB here. And obviously, there is plenty of free diskspace available (an HFS file system on an external usb hard drive).

Is this something to do with the values of "hash_read_ahead", "hash_interval", "hash_max_tries", parameters? What are their default values or how are they being computed? (The values of these were NOT set in my .rtorrent.rc.) I am relatively new to rtorrent and I have not have time to look at the code closely.

[anonymous]

HFS or HFS+? Because HFS only supports files up to 2GB in size.

And memory usage increases with chunk size (the 4MB chunks here are on the large side), so it's possible that you're running out of memory, or perhaps hitting a ulimit.

[anonymous]

My apology. It is HFS+, non-journaled. The memory hypothesis will need more investigation.

[rakshasa]

Probably hitting one of the bugs in openssl.

[anonymous]

I had this same problem. I found out that it was bad blocks on my USB external drive. Try running fsck -c /dev/[your torrent partition].

[anonymous]

Just ran into the same problem, the torrent was > 50 GB, multiple files inside though.

badblocks and xfs_check reported no problems. It seems to happen only with really huge files. The system has 4 GB of RAM available and the disk is almost empty too.

Ubuntu 10.04 with rtorrent 0.8.6.

[anonymous]

SIGBUS can also happen with temporary hardware problems, so that you don't need to have actual bad blocks to get SIGBUS.

Check whether the kernel itself was aware of any problems/unusuals at that time (e.g. kernel syslogs including debug level).

[DarianaMitch <solxsir@tungstencarbideweddingbands.org>]

Replying to anonymous:

My apology. It is HFS+, non-journaled. The memory hypothesis will need more investigation.  dresses for prom online

[Geoffrey <geoffrey@bedl.am>]

I'm running into a similar problem as the OP. - Ubuntu 10.10 - External HSF+ USB drives - rtorrent crashes while performing hashes of large (2gb+) files

If it helps, there is no output in my kernel syslog, and it appears that the issue (for me at least) is down to hashing files on hfs+ file systems. In other words, if I hash a file on the local filesystem, it does not crash.

Additionally, rtorrent will crash when not hashing files, ie when downloading/writing the file normally. When this happens I have to perform a: sudo fsck.hfsplus /dev/[device_id] in order to "clean" it.

Thanks for all the hard work that goes into rtrorrent!

[claes]

Same problem - get about 10% into a 4GB file on ubuntu 10.10, downloading to HFS+ non-journaled volume, caught bus error...

[claes]

Replying to claes:

Same problem - get about 10% into a 4GB file on ubuntu 10.10, downloading to HFS+ non-journaled volume, caught bus error...

correction, 10% on 4.5GB file, latest rtorrent and openssl in repositories

[rakshasa]

SIGBUS on linux indicates that it's more likely a problem with your system, e.g. no more disk space or file system not supporting filesizes > 4GB.

[anonymous]

In my case the partition is formatted as HFS+ (maximum file size is 8 EiB) and has 1.4 TB of available space on an HFS+ disk.

Used fsck.hfs and found errors on the large file, fsck fixed them, tried again and got the same error about 10% into hashing.

I then deleted my session folder and downloaded a smaller torrent (1.2 GB) to see if it was working and all went well. Added the 4.5 GB torrent and got the same error.

I'm going to turn on logging now and see what it says... any other tests/troubleshooting I might do?

More specs: Athlon XP 1440+ 512 MB RAM Disk being written to is connected via FW400, rtorrent is run off of local PATA disk (Ext4) Xubuntu 10.10, if that makes a difference, rtorrent running via screen with wtorrent

[anonymous]

guess there's no more execute_log option!

Replying to anonymous:

In my case the partition is formatted as HFS+ (maximum file size is 8 EiB) and has 1.4 TB of available space on an HFS+ disk. ... I'm going to turn on logging now and see what it says... any other tests/troubleshooting I might do?

[claes]

Sorry, last two messages were from me. Quick update: tried encoding_list = utf-8, safe_sync = yes, and receive and send buffers at 8. Tried options separately and together and no luck.

[claes]

After checking my logs and not finding anything, trying with transmission to see if I got errors (I did, generic I/O errors), and then using dd to make sure it wasn't a bug with HFS+ on linux (it worked), I deleted my rtorrent session directory, deleted the problem torrent's files, ran fsck.hfs, and added the following to my .rtorrent.rc:

encoding_list = UTF-8 safe_sync = yes receive_buffer_size = 8K send_buffer_size = 8K max_memory_usage = 64M

I then restarted rtorrent and added the problem torrent. The torrent started downloading until I got "file chunk write error: cannot allocate memory" repeatedly and rtorrent crashed. Restarted and it hashed all the way through (yay!) but was still getting file chunk errors. I then stopped rtorrent, changed max_memory_usage = 256M and it finished downloading, although I haven't tried hashing it yet.

I imagine the encoding and memory usage parameters are the ones that did the trick? I read about safe_sync in the manpage but don't know anything about memory mapping, Ubuntu, and HFS+. Should I change these at all? How will these parameters effect rtorrent's performance? Are they useful in dealing with large files/chunks? Thanks!

[claes]

Hashed successfully last night, though I don't have nearly as many torrents running as I did initially. Hopefully those parameters help someone else. Any input on my .rtorrent.rc adjustments is more than welcome!