I have a crawler program, it fetches html on the internet and a
parser will parse the page.
Because of slow networks, I use many robots to crawl the pages and
one parser.
The program is multi-threaded.
There is a pagestore object used as the bridge between robot and
parser, it's has a List structure,
the parser will remove the item it has parsed.
But as the program runs, its memory keep on inscreasing. I don't know
why. What I see in the taskmgr shows that most memory is in the
virtual memory, and the actual memory occupation is not much.
So what could possibly be the reasons?
Joe
