Best Way to Process Large Text Files

Hello,

I am tasked with writing an application to process some large text
files, i.e. > 1 GB. The input will be csv and the output will be in the
format of an IIS web server log.

I've done this sort of thing before. In the past, I've just
brute-forced it, with a BufferedReader and BufferedWriter handling the
input/output line by line.

I have a little time to complete this project and I'd like to build
something more efficient, that won't peg the CPU for an hour.

My thought was to have a read thread and a write thread and create a
buffer into which some amount of input would be written; and then, when
a threshold was reached, the data would be written out.

Is this a good idea? Are there better ways to manage this?

And finally, I need pointers as to how I would create such a buffer.
The threaded read/write part I can do.

Thanks for any help.

mp

--
Michael Powe michael@trollope.org Naugatuck CT USA
Re graphics: A picture is worth 10K words -- but only those to describe
the picture. Hardly any sets of 10K words can be adequately described
with pictures.

On Wed, 10 Feb 2010 06:28:14 -0500, Michael Powe
<michael+gnus@trollope.org> wrote:

>Hello,
>
>I am tasked with writing an application to process some large text
>files, i.e. > 1 GB. The input will be csv and the output will be in the
>format of an IIS web server log.
>
>I've done this sort of thing before. In the past, I've just
>brute-forced it, with a BufferedReader and BufferedWriter handling the
>input/output line by line.
>
>I have a little time to complete this project and I'd like to build
>something more efficient, that won't peg the CPU for an hour.
>
>My thought was to have a read thread and a write thread and create a
>buffer into which some amount of input would be written; and then, when
>a threshold was reached, the data would be written out.
>
>Is this a good idea? Are there better ways to manage this?
>
>And finally, I need pointers as to how I would create such a buffer.
>The threaded read/write part I can do.
>
>Thanks for any help.
>
>mp
If the innput is a CSV file then the logical unit is presumably a
record, either as a line of text or (partly) processed.

Create a queue. The read process adds records to the queue. The
write process pulls records off the queue.

rossum

On Wed, 10 Feb 2010, Michael Powe wrote:

> My thought was to have a read thread and a write thread and create a
> buffer into which some amount of input would be written; and then, when
> a threshold was reached, the data would be written out.
>
> Is this a good idea?

I'm slightly skeptical. If the processing is simple, then most of the time
will be spend doing IO even with a simple implementation. Adding threads
to overlap IO and processing might not be a big win. You could try writing
a sequential version of the program (with sufficiently large buffers - a
few megabytes, maybe?), then measuring how fast it runs - if the total
input and output data rate is close to your storage subsystem's capacity,
then no amount of programming cleverness will make it much faster.

If, OTOH, there's significant headroom above the rate you reach, then
using threads as you describe would be a good thing to try. Either that or
non-blocking IO via the NIO package, but i think you'd get decent results
from threads.

> And finally, I need pointers as to how I would create such a buffer. The
> threaded read/write part I can do.

You could try java.io.PipedInputStream and PipedOutputStream. If you want
a bigger buffer, you could grab the code for these from OpenJDK and modify
it. Mind you, circular buffers are a pretty standard bit of programming,
so there will be dozens of other implementations and descriptions out
there on the web.

tom

--
It's rare that you're simply presented with a knob whose only two
positions are "Make History" and "Flee Your Glorious Destiny." --
Tycho Brahae

On Wed, 10 Feb 2010 06:28:14 -0500, Michael Powe
<michael+gnus@trollope.org> wrote, quoted or indirectly quoted someone
who said :

>I've done this sort of thing before. In the past, I've just
>brute-forced it, with a BufferedReader and BufferedWriter handling the
>input/output line by line.

There is quite a bit of CPU work parsing a CSV file. Try
http://mindprod.com/products1.html#CSV
and give it a 64K buffer before you go to a lot of work cooking up
something exotic.
--
Roedy Green Canadian Mind Products
http://mindprod.com

Every compilable program in a sense works. The problem is with your unrealistic expections on what it will do.

On 10/02/2010 10:28 PM, Michael Powe wrote:
> I have a little time to complete this project and I'd like to build
> something more efficient, that won't peg the CPU for an hour.

Fix your code. It only takes a few seconds to read a file of practically
any size. In my experience the only way you can take an hour to process
any file on modern equipment is if you read the whole file into memory
via concatenation of Strings and then process it, which is the wrong
approach from every possible point of view. Process a line at a time.

On 10-02-2010 18:59, EJP wrote:
> On 10/02/2010 10:28 PM, Michael Powe wrote:
>> I have a little time to complete this project and I'd like to build
>> something more efficient, that won't peg the CPU for an hour.
>
> Fix your code. It only takes a few seconds to read a file of practically
> any size. In my experience the only way you can take an hour to process
> any file on modern equipment is if you read the whole file into memory
> via concatenation of Strings and then process it, which is the wrong
> approach from every possible point of view. Process a line at a time.

I agree completely with your point.

Huge files may still take time to read from the disk though.

Arne

Arne Vajhøj wrote:
> On 10-02-2010 18:59, EJP wrote:
>> On 10/02/2010 10:28 PM, Michael Powe wrote:
>>> I have a little time to complete this project and I'd like to build
>>> something more efficient, that won't peg the CPU for an hour.
>>
>> Fix your code. It only takes a few seconds to read a file of practically
>> any size. In my experience the only way you can take an hour to process
>> any file on modern equipment is if you read the whole file into memory
>> via concatenation of Strings and then process it, which is the wrong
>> approach from every possible point of view. Process a line at a time.
>
> I agree completely with your point.
>
> Huge files may still take time to read from the disk though.


The OP said > 1 GB, so we don't know if he meant up to 2 GB or if he's
talking about 10 GB or 100 GB or 1000 GB. So a little clarification
here would help, I think.

On Wed, 10 Feb 2010 06:28:14 -0500, Michael Powe
<michael+gnus@trollope.org> wrote, quoted or indirectly quoted someone
who said :

>
>I am tasked with writing an application to process some large text
>files, i.e. > 1 GB. The input will be csv and the output will be in the
>format of an IIS web server log.

Do a little benchmark where you do nothing but read the giant file.

If all the time is spend processing the file, there not much point in
fancy stuff to read the file files.

Usually when things slow to a crawl it is because you have filled RAM
with objects you don't need, and that forces very frequent GC.

Before you start optimising, you first have to prove where the
bottlenecks are.
--
Roedy Green Canadian Mind Products
http://mindprod.com

Every compilable program in a sense works. The problem is with your unrealistic expections on what it will do.

On Feb 10, 9:07*pm, Arne Vajhøj <a...@vajhoej.dk> wrote:
> Huge files may still take time to read from the disk though.
A lot of time. I tried my skills in Netflix $1,000,000 contest... on
my computer it took 15 minutes to read their entire data (for example
to do some calculation). I had compressed it to the zip archive and
then reading it and uncompress with the same calculation took only 3
minutes.

On 2/10/2010 3:28 AM, Michael Powe wrote:
> Hello,
>
> I am tasked with writing an application to process some large text
> files, i.e.> 1 GB. The input will be csv and the output will be in the
> format of an IIS web server log.
>
> I've done this sort of thing before. In the past, I've just
> brute-forced it, with a BufferedReader and BufferedWriter handling the
> input/output line by line.
>
> I have a little time to complete this project and I'd like to build
> something more efficient, that won't peg the CPU for an hour.
>
> My thought was to have a read thread and a write thread and create a
> buffer into which some amount of input would be written; and then, when
> a threshold was reached, the data would be written out.
>
> Is this a good idea? Are there better ways to manage this?
>
> And finally, I need pointers as to how I would create such a buffer.
> The threaded read/write part I can do.
>
> Thanks for any help.
>
> mp
>
Depending on how processor intensive the transformation is, you might
not gain anything from threading.

If you are using regex to parse, you may be better off optimizing your
regexs, or using hand-coded parsing instead. A naive regex which "works"
may have some performance problems. Use greedy matching where
appropriate is one way to improve performance.

--
Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>