RE: report on running QE with a large index

["Doug Olson" <dlolson@lbl.gov>]

Wonderful!  This is certainly in good shape for MDC2.
It also leaves us with some future work to do to
handle 10**9 evts.
Doug

> -----Original Message-----
> From: Henrik Nordberg [mailto:hnordberg@lbl.gov]
> Sent: Thursday, February 18, 1999 5:03 PM
> To: gcdev
> Subject: report on running QE with a large index
>
>
> Hello,
>
> I took the hijing data (db_30022) and appended the ASCII file
> index file to itself 700 times, creating a 1e7+ event file.
> I used starsu00.
> That file is roughly 1 GB.
> It took 57 minutes to build the index.
> The size of the index is 1.3 GB.
> The bitmap index is 137 MB.
> Creating the bitmap index from the binary index took half an
> hour (or 1854s 410ms to be exact).
>
> I started QE (~10 minutes) and ran 2 queries:
>
> (starsu00) 16:33 client > inq -c "comp1" -q "num_Pion_p<10" ../gc.config
> Using configuration file "../gc.config".
> [36ccb1a3-4fefcd] New query token: 36ccb1a3-4fefcd
> [36ccb1a3-4fefcd] Quick estimate gives: 0 <= num events <= 2097200
> [36ccb1a3-4fefcd] Estimated completion time Thu Feb 18 16:37:10 1999
> [36ccb1a3-4fefcd] Number of events: 8400
> [36ccb1a3-4fefcd] succeed
> (starsu00) 16:36 client > inq -c "comp1;comp0;comp2;comp3;comp4;"
> -q "num_Pion_p>-1" ../gc.config -v
> <index info snipped>
> [36ccb271-71d187] New query token: 36ccb271-71d187
> [36ccb271-71d187] Quick estimate gives: 10027500 <= num events <= 10027500
> [36ccb271-71d187] Estimated completion time Thu Feb 18 16:43:28 1999
> [36ccb271-71d187] Number of events: 10027500
> [36ccb271-71d187] succeed
> (starsu00) 16:45 client >
>
> CPU usage of QE after loading the index and these two queries
> was 14 minutes. The total resident memory usage was >800 MB.
>
> (but it didn't crash)
>
> Cheers, Henrik
> _________________________________________
> Henrik Nordberg       <hnordberg@lbl.gov>
> Scientific Data Management Research Group
> Lawrence Berkeley National Laboratory
>