Ticket #495 (closed defect: invalid)

Opened 5 years ago

Last modified 2 years ago

[File chunk read error: Cannot allocate memory]

Reported by: jdriggers@gmail.com Owned by: rakshasa
Priority: high Component: libtorrent
Version: Severity: critical
Keywords: Cc:

Description

with 0.10.3 / 0.6.3 I'm rather consistently getting Storage error: [File chunk read error: Cannot allocate memory]..

Did not have this problem with 0.10.2 / 0.6.2

Change History

  Changed 5 years ago by JDriggers@gmail.com

When attempting to force re-hash (r), follow error is encountered given the that the torrent returns : [File chunk read error: Cannot allocate memory].. * Hashing: Storage error: [Hash checker was unable to map chunk: Cannot allocate memory]

  Changed 5 years ago by steevel

I have the exact same problem.

follow-up: ↓ 4   Changed 5 years ago by JDriggers

Seems to occur whenever the max_memory is exceeded when hashing. Memory does not seem to be released after a hash is completed.

in reply to: ↑ 3   Changed 5 years ago by anonymous

Replying to JDriggers:

Seems to occur whenever the max_memory is exceeded when hashing. Memory does not seem to be released after a hash is completed.

Re-compiled with gcc 3.4.6 and problems went away..

follow-up: ↓ 6   Changed 5 years ago by rakshasa

Yeah, I assume that was 4.1.x you used? At least that seems to be the common factor between those who've reported this problem, though I'm having trouble determining if it is a compiler bug or a bug being triggered specifically by that compiler version.

in reply to: ↑ 5   Changed 5 years ago by anonymous

Replying to rakshasa:

Yeah, I assume that was 4.1.x you used? At least that seems to be the common factor between those who've reported this problem, though I'm having trouble determining if it is a compiler bug or a bug being triggered specifically by that compiler version.

Yeah, 4.1.1 on Fedora core 5, i686..

follow-up: ↓ 8   Changed 5 years ago by rakshasa

There is a suspicion of the compiler producing wrong code on -march=i686, try 4.1.1 with -march=i386.

in reply to: ↑ 7 ; follow-ups: ↓ 9 ↓ 10   Changed 5 years ago by lusitania

Replying to rakshasa:

There is a suspicion of the compiler producing wrong code on -march=i686, try 4.1.1 with -march=i386.

I actually suspect 4.1.1 is producing bad code with -02, not -march=i686. I've been testing on x86 with vanilla GCC 4.1.1.

"-O2 -march=pentium4" was originally crashing for me, so I tried "-O0 -march=i386" and it worked fine. So I tried "-O2 -march=i386" and it crashed.

I have since sucessfully tried: "-O1 -march=pentium4" "-O1 -march=i386" "-O0 -march=pentium4" "-O0 -march=i386"

(Not specifying any -O flag will of course imply -O0)

in reply to: ↑ 8   Changed 5 years ago by sthamel

Replying to lusitania:

Replying to rakshasa:

There is a suspicion of the compiler producing wrong code on -march=i686, try 4.1.1 with -march=i386.

I actually suspect 4.1.1 is producing bad code with -02, not -march=i686. I've been testing on x86 with vanilla GCC 4.1.1. "-O2 -march=pentium4" was originally crashing for me, so I tried "-O0 -march=i386" and it worked fine. So I tried "-O2 -march=i386" and it crashed. I have since sucessfully tried: "-O1 -march=pentium4" "-O1 -march=i386" "-O0 -march=pentium4" "-O0 -march=i386" (Not specifying any -O flag will of course imply -O0)

Yup, same here... Going with "-O0 -march=pentiumII" did the trick even with gcc 4.1.1-r1. It also fixed the huge memory allocation by rtorrent, even after the app was terminated.

in reply to: ↑ 8   Changed 5 years ago by anonymous

Replying to lusitania:

Replying to rakshasa:

There is a suspicion of the compiler producing wrong code on -march=i686, try 4.1.1 with -march=i386.

I actually suspect 4.1.1 is producing bad code with -02, not -march=i686. I've been testing on x86 with vanilla GCC 4.1.1. "-O2 -march=pentium4" was originally crashing for me, so I tried "-O0 -march=i386" and it worked fine. So I tried "-O2 -march=i386" and it crashed. I have since sucessfully tried: "-O1 -march=pentium4" "-O1 -march=i386" "-O0 -march=pentium4" "-O0 -march=i386" (Not specifying any -O flag will of course imply -O0)

I can confirm the success of "-O1 -march=pentium4" on a gentoo machine running 4.1.1. Force re-hash works fine, and it seems to be running well.

  Changed 5 years ago by anonymous

I just want to report that I am successfully running 0.6.3/0.10.3 with gcc 4.1.1 and flags "-O2 -pipe -march=k8". It has been running stable for 3+ days now, with at least one 3gb+ torrent downloaded / hashed / seeding without restart. I have also manually forced hash check twice on a 16gb torrent while checking memory usage and don't see any problem.

With same compiler and settings 0.6.2/0.10.2 did not work (memory errors), though 0.6.1/0.10.1 did. Apparently one of the changes in 0.10.3 did fix the problem for me.

  Changed 5 years ago by rakshasa

In 0.10.3 I moved around and divided some of the suspected code, so that probably avoided some of the compiler bugs from 0.10.2.

  Changed 5 years ago by epchris

Don't know if it's much help, but I'm getting the same error consistantly on OpenSuSE 10.1 with gcc == 4.1.0. It usually occurrs after rtorrent has been running for a while.

Any suggestions?

  Changed 5 years ago by rakshasa

  • status changed from new to closed
  • resolution set to worksforme

Use any flags but -O2, like f.ex. -O3 or -Os.

  Changed 5 years ago by epchris

Maybe -O2 shouldn't be the default then? (Seems to be for me anyway).

  Changed 5 years ago by anonymous

That's because -O2 works fine with everything except gcc 4.1

  Changed 5 years ago by anonymous

The build in fedora-extras yum repo still has this problem. If someone in this group is responsible for uploading new builds please do so!

  Changed 4 years ago by bash-fool@bluebottle.com

  • status changed from closed to reopened
  • resolution worksforme deleted

On Gentoo, i tried both (without xmlrpc support): x86: 0.7.8/0.11.8 , and; ~x86: 0.7.9/0.11.9

with C/CXXFLAGS: "-march=pentium-mmx -O3 -pipe -fomit-frame-pointer"

and GCC version: 3.4.6 (Gentoo Hardened 3.4.6-r2 p1.5, ssp-3.4.6-1.0, pie-8.7.10)

and i keep getting: "Hashing: Storage error: [Hash checker was unable to map chunk: Cannot allocate memory]"

regardless of tweaking of following options in .rtorrent.rc: hash_read_ahead hash_interval hash_max_tries max_open_files max_memory_usage

weird thing is i got this error yesterday when 3 torrents neared their completion, then i got it to go away by pausing some of them. but now i can't restart even one of them without giving me allocation errors. i thought there might be a problem with reading the files all together, but the launching user can read, write (and delete) in the downloads dir just fine.

  Changed 4 years ago by anonymous

I am using the newest versions of rtorrent/libtorrent, but compilied from the source. I tried using the -Os or O3, but I still have hashing errors, and cannot allocate memory errors. I am also using gcc 4. I wasn't sure how to use those flags. I exported CFLAGS and CXXFLAGS to -Os. Then did ./configure, make, make install. I am also using CentOS 5 if that matters.

  Changed 4 years ago by anonymous

This problem happens on OS X too.

  Changed 2 years ago by anonymous

Seems error caused by iowait values and general os slowdown.

Run any linux in VM with 1 Gb ram. Run 2 copies of rtorrent with rehashing something big.

Third rtorrent with download will spam that error.

  Changed 2 years ago by rakshasa

  • status changed from reopened to closed
  • resolution set to invalid

Don't reopen, use a different GCC version.

Note: See TracTickets for help on using tickets.