Re: Splitting huge XML Files into fixsized wellformed parts

Malapha wrote:
> 
> As I come from the VBA world - I tried to get familiar with awk. What
> I do have is theoretical solution in form of a structured process
> diagram :-)
> 
> Copy Header and Footer from Original to Var
> Set Start_Offer = First Offer (from <Offer> to </Offer>)
> Set End_Transaction = 0
> Set Part = 0
> Set FileSize = 0
> Set MaxFileSize = 250
> while not Start_Offer < EOF(OriginalXMLFile)
>      Part=part+1
>      Open NewFile OriginalXMLFileName + Part + ".xml"
>      Paste Header from Var to NewFile
>      While filesize(NewFile)<MaxFileSize do
>          Copy Offer (Start_Offer) from OriginalXMLDatei to NewFile
>          Start_Offer=Start_Offer + 1
>      wend
>      Paste Footer from Var to NewFile
> wend
> 
> I am right now trying to translate this into awk.. Please dont ask me
> how far i am, its frustrating :-)
> 
> 

Below one solution for splitting in well-formed chunks, here: 100 
OfferInfos each.  There might be better solutions (I just don't know 
them ;-) It only works if the XML data is in "pretty print format", as 
the sample data you posted.


$ cat split_bigfile.awk

BEGIN {	new_chunk = 1 ; size = 100 }

NR == 1 { header = $0 ; next }
NR == 2 { header = header ORS $0 ; footer = "</" substr($1,2) ">" ; next }

$0 !~ footer {
	if (new_chunk) {
		outfile = "chunk" sprintf("%07d", num) ".xml"
		print header > outfile
		new_chunk = 0
	}
	print > outfile
}

/<\/OfferInfo>/ {
	num = int(count++/size)
	if (num > prev_num) {
		print footer > outfile
		new_chunk = 1
	}
	prev_num = num
}

END { if (!new_chunk) print footer > outfile }