measurement plan summary for MDC1

[Arie Shoshani <arie@math.lbl.gov>]

Enclosed is a summary of what we discussed (with some refinement)
during the meeting at BNL on what GCA measurement we should plan to
perform during MDC1.  I took the time to spell this out in enough
detail that it can be understood by people outside GC as well.

I hope I captured this accurately.

Arie.
----------------------------------------------------------------

Measurements plans for MDC1

1.  10-20 typical queries

We hope to get a set of typical queries (and perhaps some analysis
user code as well) to apply for our measurements.  The queries will be
in the form of conjunctive or disjunctive ranges over properties in
the tagDB.  We may want to associate with each query a "frequency
likelihood" measure that tells us how likely it is that the query will
be submitted (perhaps by different users).  We may chose 1-2 most
likely queries for our "same query" tests (see below in part 5).

2.  Before environment.

We need a way to test the performance before involving GCA machinery.
In particular, we need to imitate what will be done if GC did not
exist.  

It was agreed that a realistic environment we should compare to is one
that includes the existence of Objectivity and a tag database that is
disk resident.  In this scenario, the user code will obtain a list of
event OIDs by querying the tag database, and request (from
Objectivity) the event information it needs for these OIDs in an
arbitrary order.

We also assume that there will be some mechanism that will bring into
disk cache files that Objectivity needs, and are not in the cache at
that time.  This mechanism will also decide what to have in cache and
what to remove from cache.  We hope to be able to use the AMS server
being developed by Babar (Andy Anishevsky).

3.  "After environment"

The "after environment" is, of course, the GCA machinery.  However, we
want to distinguish in our measurements the time used by the Storage
Manager from the time used by the user code, and the time used by
Objectivity.

Thus, we plan to have 3 types of experiments done, on the same set of
queries:

a) User code does nothing, and Objectivity is not invoked (what Dave
Malon called: "no touch" option);
b) User code does significant work (if necessary we introduce
artificial delays) and Objectivity is not invoked;
c) User code does nothing, and Objectivity is invoked to get the event
data to the user code (the "touch" option).


4.  What will be measured

We agreed that for global effect measures it is sufficient to measure
overall times of the system and the Storage Manager.  We may want, in
addition, to measure the time spent in individual components for our
own debugging purposes.

Specifically, the following 3 overall time measurements are planned:

a) Total "wall clock" time;
b) Total CPU time;
c) Total "storage caching" time per query.

The total "storage caching" time per query is the total time from the
first EI request of a file for a query, to the time that a "done" is
issued by the EI for the last file of that query.  Note that adding
these time maybe longer than the total clock time in case of query
overlap.


5.  What tests are planned

We wish to see the effect of event-order optimization and the
file-overlap optimization.  For each of these we plan to perform the
"before" and "after" tests.  The "after" tests may include the 3 test
types mentioned above (with/without user code and Objectivity).

The event-order optimization tests will check the effect of reading
all events out of the file that was cached at the same time.  In that
case, the selected OIDs are requested in the order they stored in
files.  The file-overlap optimization tests will check the effect of
overlapping the reading of files in cache by multiple simultaneous
queries.  

Three types of tests are planned:

a)  Sequential execution of the typical queries.

The purpose of this test is to see the effect of event-order
optimization.  Since we wish to see the effect of running each query
in isolation, we start each query with an empty cache.  Of course, if
the cache is large enough to hold all the files needed by the query,
no effect will be detected.  So, we need to limit this to queries that
the total file size to be cached far exceeds the size of the cache to
notice the effect.

b)  Simultaneous execution of the same query.

The purpose is to check the file-overlap optimization at the extreme.
In this case, we the same query several times in simultaneously.  This
test should show full overlap.  

c)  Simultaneous execution of same query with induced delays.

In this case we do the same as above, but the queries are initiated in
a staggered fashion, with some time distance in between them.  As the
time distance increases we should see less overlap.

A variation on this that we may test is the introduction of random
delays in the user code for each query.  This imitates a real
situation better, were different analysis codes take different amount
of time to analyze each event.

d)  Simultaneous execution of the typical queries.

In this case we run the query mix of typical queries all started
simultaneously.  As a variation on that, we may want to duplicate
queries in the mix that have a higher "frequency likelihood" measure.
This test should provide a realistic overall comparison of "before"
and "after" use of the GCA that includes both event-order and
file-overlap optimizations.

6.  Set up of disk cache size

Since at the time of MDC1 the number of files will be relatively
small, we need to limit the size of the disk cache for this
measurement tests to emulate a realistic future scenario.  A crude but
reasonable way to do that is to allocate the same ratio of total files
size to cache size as expected in an operational system.

Assuming that after a year there will be a total of 300 TB, and the
cache size will be about 10 TB, the ration is 1/30.  Assuming that by
the time of MDC1 there will be about 50,000 simulated events, each
having a total of 5 MBs, the total size will be 250 GB.  Thus, the
cache size we should use is about 10 GB.  That seem plausible if each
query will end up needing to touch 10-20  1GB files.