On 04/06/2007 03:27 PM, Bill Stephenson wrote:
> Perl crashes when using the included script for large data files. How
> can I prevent this from happening?
>
What do you mean by crashes? Does it abort with an error message placed
in the web-server log file? If so, that's not a crash; it's just an
error abort of the script.
> Kindest Regards,
>
> --
> Bill Stephenson
>
>
> Sample data source is a Delimited text file with the following fields:
>
> FMT:STORE:TIKEY:TITLE:QTY:DC:COMPANY:ZONE
>
> DV:043712:20882903:DVD-ALL THE KING'S M('06/WS/2D:8.00000:ECDC:Hlyw:4
> DV:043712:20883071:DVD-ALTERED (2006/WS):8.00000:ECDC:Hlyw:4
> DV:043712:20883070:DVD-AMERICAN PIE 5-****(UR/WS:32.00000:ECDC:Hlyw:4
> DV:043712:20883253:DVD-BROOKLYN LOBSTER (WS):2.00000:ECDC:Hlyw:4
> DV:043704:20882903:DVD-ALL THE KING'S M('06/WS/2D:16.00000:WCDC:Hlyw:4
> DV:043704:20883071:DVD-ALTERED (2006/WS):8.00000:WCDC:Hlyw:4
> DV:043693:20882903:DVD-ALL THE KING'S M('06/WS/2D:24.00000:ECDC:Hlyw:4
> DV:043693:20883071:DVD-ALTERED (2006/WS):8.00000:ECDC:Hlyw:4
> DV:043693:20883070:DVD-AMERICAN PIE 5-****(UR/WS:24.00000:ECDC:Hlyw:4
> DV:043693:20883092:DVD-ZOMBIE NATION (2006/WS):4.00000:ECDC:Hlyw:4
> DV:043670:20882903:DVD-ALL THE KING'S M('06/WS/2D:8.00000:ECDC:Hlyw:4
>
> <code>
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
>
> # set-up our variable....
>
> my $source = "dataSource.txt";
> my $destination = "testOUTPUT.txt";
>
> my $fmt = "";
> my $storeno = "";
> my $co = "";
> my $dc = "";
> my $zone = "";
> my $store = "";
> my $tikey = "";
> my $title = "";
> my $qty = "";
> my $text1 = "";
> my $text2 = "";
> my $output = "";
>
> my @[EMAIL PROTECTED]
= "";
This creates an array with one element, an empty string. Usually people
just want to create an empty array which can be created like so:
my @[EMAIL PROTECTED]
= ();
>
>
> # start the work...
>
> open my $IN, $source or die "Can't read source file $source: $!\n";
>
> # put all the stores into an array (@[EMAIL PROTECTED]
)...
> my @[EMAIL PROTECTED]
= <$IN>;
>
> foreach my $line(@[EMAIL PROTECTED]
) {
>
This reads all of the lines into memory at once. Whoa! You almost
certainly want to use a while loop to read the lines one-by-one:
my @[EMAIL PROTECTED]
(my $line = <$IN>) {
Or even better, use Tie::File to treat the lines in the file as an
array. That would almost certainly suit you better since your code uses
an array of lines that you search through more than once.
tie my @[EMAIL PROTECTED]
'Tie::File', $source
or die "Can't read file $source: $!\n";
Read the manual pages for Tie::File and perltie:
Start->Run->"perldoc Tie::File"
Start->Run->"perldoc perltie"
> ($fmt, $store, $tikey, $title, $qty, $dc, $co, $zone) =
> split(/:/,$line);
>
> push @[EMAIL PROTECTED]
$store;
>
> }
>
>
> # remove duplicate stores from the list...
> my %seen = ();
> my @[EMAIL PROTECTED]
= ();
>
> foreach my $item (@[EMAIL PROTECTED]
) {
>
> unless ($seen{$item}) {
> # if we get here, we have not seen it before
> $seen{$item} = 1;
> push(@[EMAIL PROTECTED]
$item);
> }
> }
>
> # for each store in our @[EMAIL PROTECTED]
store list we look for
> # lines in our file that have a matching store
>
> foreach my $uniqStore (@[EMAIL PROTECTED]
) {
>
> foreach my $line(@[EMAIL PROTECTED]
) {
>
> ($fmt, $store, $tikey, $title, $qty, $dc, $co, $zone) =
> split(/:/,$line);
>
> # if they match we create our entry for the destination file....
> if ($store eq $uniqStore) {
>
> $text1= "Street Date $source \n". "Company $co \n".
> "Distribution Center $dc \n". "Zone $zone \n". "Store Number $store
\n\n";
>
> $text2 .= "$tikey \t $title \t $qty\n";
>
> }
>
> }
>
> # then add it to the final $output.
> if ($text1 ne "") {
> $output .=
> "$text1$text2\n####################\n####################\n\n\f\n";
> $text2 = "";
> }
> }
>
> # and finally print all out
>
> # print $output; #for debugging
>
> open(my $DATA, "> $destination")
> or die (" Error 1.3 Couldn't file:: $destination for writing: \n");
>
> print $DATA ($output);
>
> close $DATA;
>
>
Once you start using Tie::File, your program will be half its current
size and much faster.
|
