Memory Management with huge text files

[foozball3000]

We're trying to write a small program that removes rogue End Of File Characters from text data files. The code works fine, but shortly after execution our laptops (4GB Ram, Vista 32bit) run out of memory.

We're coding in VB.Net with Visual Studio 2008.

How can we work our way around this?

 

[dequadin]

Don't read the entire text file in before processing, rather batch it into say 200 lines at a time or something similar.

Also you're obviously looping this thing do don't re-declare any big objects in the loop as the GC takes a bit of time to free up this memory. Declare them outside the loop and reassign them in the loop. Don't forget to flush your buffers

 

[foozball3000]

Don't read the entire text file in before processing, rather batch it into say 200 lines at a time or something similar.

Also you're obviously looping this thing do don't re-declare any big objects in the loop as the GC takes a bit of time to free up this memory. Declare them outside the loop and reassign them in the loop. Don't forget to flush your buffers

We're thinking down the line of batches as well.

Just to mention, it seems that the out of memory error happens when the program's ram usage grow to about 500mb.
Task manager says that idle, VISTA is using 1.2 GB Ram, when the usage spikes to almost 1.7 it crashes.

 

[semaphore]

Its because you are allocating space beyond the LOH (large object heap), you are trying allocate to contiguous memory, that is why you are getting a stack dump. Also the problem is your varibles might not have been promoted to a Gen 1 GC clean up.

You will need to load it into a batch

 

[Mike_De_Lange]

Off Topic: I assume you would be looking to open large files in the manner that LTF Viewer (Large Text File Viewer or something) does? Check it out. No idea how they do it though sorry.

http://www.swiftgear.com/ltfviewer/features.html