report on running QE with a large index

[Henrik Nordberg <hnordberg@lbl.gov>]

Hello,

I took the hijing data (db_30022) and appended the ASCII file 
index file to itself 700 times, creating a 1e7+ event file. 
I used starsu00.
That file is roughly 1 GB. 
It took 57 minutes to build the index. 
The size of the index is 1.3 GB. 
The bitmap index is 137 MB.
Creating the bitmap index from the binary index took half an
hour (or 1854s 410ms to be exact).

I started QE (~10 minutes) and ran 2 queries:

(starsu00) 16:33 client > inq -c "comp1" -q "num_Pion_p<10" ../gc.config 
Using configuration file "../gc.config".
[36ccb1a3-4fefcd] New query token: 36ccb1a3-4fefcd
[36ccb1a3-4fefcd] Quick estimate gives: 0 <= num events <= 2097200
[36ccb1a3-4fefcd] Estimated completion time Thu Feb 18 16:37:10 1999
[36ccb1a3-4fefcd] Number of events: 8400
[36ccb1a3-4fefcd] succeed
(starsu00) 16:36 client > inq -c "comp1;comp0;comp2;comp3;comp4;" -q "num_Pion_p>-1" ../gc.config -v
<index info snipped>
[36ccb271-71d187] New query token: 36ccb271-71d187
[36ccb271-71d187] Quick estimate gives: 10027500 <= num events <= 10027500
[36ccb271-71d187] Estimated completion time Thu Feb 18 16:43:28 1999
[36ccb271-71d187] Number of events: 10027500
[36ccb271-71d187] succeed
(starsu00) 16:45 client > 

CPU usage of QE after loading the index and these two queries
was 14 minutes. The total resident memory usage was >800 MB.

(but it didn't crash)

Cheers, Henrik
_________________________________________
Henrik Nordberg       <hnordberg@lbl.gov>
Scientific Data Management Research Group
Lawrence Berkeley National Laboratory