AWK experts - how would I code around this in awk...

[Steve Flynn]

On Mon, Feb 22, 2010 at 10:10 PM, Karl Auer <kauer at biplane.com.au> wrote:

> No - that's why I said "may be". I don't know how big a Perl variable
> can be, but the OP mentioned multi-gigabyte input files. Conventional
> wisdon=m says "no limits" - my experience is that everything has limits.

For info, the migration I'm performing looks like it's going to run to
around 3.8 terabytes of data in total. I'm looking at some big files.
Really big files.

> I just wrote a little test program that started swapping at 1 gigabyte
> (64-bit laptop with 4 gig, but only about 1 gig available at the time).
>
> A real test would be to see how big Steve Flynn's variables can get :-)

Fnarr! :)

As this lot is running under AIX, I really have no idea of any system
enforced limits. Interesting exercise in finding out though.

As a case in point, this afternoon I ran across an Oracle limit in
that VARCHAR2 fields can only be 4000 bytes period, it seems. When I
was defining the table, I though "4k... who writes 4k sql
statements.... bound to be big enough." One arrive today weighing in
at 5780 bytes in size which means I now have to store this field as a
CLOB and it also appears that one cannot execute MAX() or EXECUTE
IMMEDIATE on a value plucked from  a CLOB.... /sigh

All by the by for this problem but I'm already hitting limits on
things I'd never even though about.

I'll have a play (and also get some timings from the various (great!) offerings.

> Just mentioning. It's the sort of thing that gets forgotten - an edge
> case. The likelihood is that a partial record is not wanted anyway.

Nope, it's not, and if the spec runs true there shouldn't be any
partial records, period.

We've all seen specs like that however.

> Fun?!? :-)

:) I get paid for the fun. <*Beam*>

-- 
Steve
When one person suffers from a delusion it is insanity. When many
people suffer from a delusion it is called religion.

09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0