Tests for GC-MDC1

[Arie Shoshani <arie@math.lbl.gov>]

Enclosed is a summary of test we plan to run in the next few
days.  It represents discussions I recently had with Henrik, 
Alex and Doug.

Dave, Dave, and Jeff,

Please take the time to look at that and give us comments 
(if any) as soon as you can.  We'll start some of these tests
tomorrow.

Arie.
------------------------------------------------------------------
TESTS PLANNED FOR GC-MDC1

To figure out the scale of the tests, we use the following
notation:

N - total number of events in the system
S  - average size of event
F - number of files in the system
F_size - average size of each file
T_capacity - tape capacity in GB
N_tapes - number of tapes to store the files
N_event_per_file = F_size \ S

For the tests we currently have:

N = 5,000 - 10,000 (we use 10,000)
S  = 2 MB
F_size = 250 MB
F = 10,000*2 / 250 = 40
T_capacity = 25 GB
N_tapes = 1 (but data was spread out on 4 tapes)
N_event_per_file = 250 / 2 = 125

Scale down factor:

Assume a total of 15.10**6 event/year
fraction = 15.10**6 / 10,000 = 1,500 (we'll use 1000)

Cache size for tests: 
1000 GB /1000 = 1GB 
(We'll use initially 2 GBs - or 8 files cache)

Query size for tests:
In real situations we expect queries of size: 10,000 - 50,000 events
scale down by 1000 gives: 10 - 50 events
(We'll use initially 100 events to see effects with more files
cached)

TESTS SETUP

To describe queries and their relationship we use the following
parameters:

n - number of events that qualify for a query
f - number of files that contain the events that qualified
CF = clustering_factor = n / f = average number of events that qualify
per file
                    (this factor is between 1 and N_event_per_file)
OF = overlap factor = % of files that overlap between any 2 queries on
the average
                    (this factor is between 0 and 100%)

Parameters we can set for each test:

(Q1, Q2, ..., Qn) - ordered set of queries to run
PT - processing time per event (in seconds)
CT - time to cache a file (in seconds)
AT - arrival time: time delay between queries (in seconds)
NC - number of times to cycle over the set of queries

example: to run the same query 5 times spaced every 10 sec, with
processing
time of 8 sec/event.
(Q1)
PT = 8
CT = F_size / stage speed + dismount/mount = 250 / 5 + 15 = 75 sec
AT = 10 sec
NC = 5

BASELINE TESTS

Baseline test checking are for getting minimum time per query, i.e.
running
the query by itself (stand alone).  This should help us determine if
the system
behaves as expected.

TEST A.  make caching time dominate

Assume a query has:
n - 100
f - 20
CF = n / f = 5
PT = 4 sec
PT_per_file = 4*5 = 20

Since CT = 75, total time will be dominated by caching
We should expect:  20*75= 1500 sec = about 0.5 hour

TEST A1: run above for each query with SII

TEST A2: run above for each query with full system
    (if time is significantly > TEST A1, check transfer time
      from Objectivity to User Code)

TEST B.  Make processing time dominate

Assume the same query as above, but increase processing time:
n - 100
f - 20
CF = n / f = 5
PT = 30 sec
PT_per_file = 30*5 = 150

Since PT_per_file > CT, processing time dominates.
We should expect: 20*150= 3000 sec = about 1 hour

TEST B1: run above for each query with SII

TEST B2: run above for each query with full system


TEST C:  to see that all files are readable

Use a query that reads all files;
Set PT very small = 0.1 sec, thus PT_per_file = 125*0.1 = 12.5 sec
Measure total time:
expect: 40*75 = 3000 sec


TESTS THAT SHOW GAIN IN PERFORMANCE

TEST D: to show effect of overlap

Need: ability to turn off overlap policy by QM.
           When this is turned off, the QM schedules file
           according to the order they were submitted.
           (another possibility to be determined later:
             choose random order for file scheduling)

Each test below is run with and without caching policy.

TEST D1: full overlap

Run a single query 10 time,
Vary arrival time (AT) till AT > PT_per_file
Measure effect with AT variation

TEST D2: partial overlap

Run all queries is some order;
Vary AT;  measure effect
Change queries to vary Overlap Factor (OF): 10%, 25%, 50%, 75%;
      Keep AT short, measure effect


TEST E: to show effect of clustering
 
TEST E1:  vary clustering factor (CF)

Each test below should be run with caching policy on.
(We may want to try the same with caching policy off).

Check CF of the given queries;
Change queries so that CF is doubled, tripled, etc.
leave arrival time small;
measure effect.

TEST E2:  Vary AT with a few selected CFs
See effects.


TEST F: to show effect of order-optimization

QE gives QO list of OIDs (full estimate).
QO scrambles order, and request each OID as a separate query.
Measure effect.


WHAT WILL BE  MEASURED

As was mentioned in a previous message from Henrik, various things
will be logged.  Our goal is to detect any anomalies, and then measure
the effect of various tests in terms of total time to perform each
test,
and breakdown of times for various components of the system.

Henrik covered in his message the measurements we expect to extract 
from the logs. 
----------------------------------------------------------------------