ROOT file_ids and Bundle_ids

[Arie Shoshani <arie@math.lbl.gov>]

This message is maily for Dave M., but it makes assumptions about the 
directory structure for MDC2, so it is relevant to others as well.

Arie.


We discussed the file_id issues and here is what we concluded.

In the long run, we'd like to have the Storage Manager software 
unafected by the type of files we deal with - Objectivity files, ROOT 
files, or any other kind.  To accommodate that, we'll have to use a 
consistent set of file_IDs, regardless of where they come from.  I'll 
elaborate on how we plan to do that later.

As for what we pass to the Event Iterator (EI) for multiple components, 
we concluded that we'd like to pass a bundle_id, and when the EI is 
done with the bundle, it should use this bundle_id when releasing the 
bundle.  However, if there is a value to passing the file_ids (or 
file_names) we could do that also at a later time (after MDC2).

So, here is what we plan for MDC2 and after MDC2.

For MDC2:

1)  Both Objectivity files and ROOT files will be pre-assigned 
Objectivity file-ids, and those will be used by the index.

2)  All file names (including path_names if more that a single 
directory is used) will be retrieved from the Objectivity catalog.

3)  We assume files can be in different directories.  However, the same 
directory structure will exist beyond a root-path in HPSS as in Unix.
(e.g. suppose the Objectivity root-path is o1/o2/o3 and HPSS root-path is 
h1/h2, then a file in Objectivity directory o1/o2/03/x/y/z/file_name, 
will be in the HPSS directory as h1/h2/x/y/z/file_name.)  Thus, given the 
objectivity path, the HPSS path can be generated.

4)  The QM will pass to the EI only the bundle_id and the set of 
event_ids that qualify for this bundle.  The EI will return this 
bundle_id when releasing the bundle.


After MDC2:

1)  ROOT files (or any other Unix files) will be assigned a 32 bit 
file_ids by the QE at the time of index creation.  Objectivity IDs will 
also be 32 bits to accommodate future objectivity file_per_container 
strategy.  Internally to the SM, we'll distinguish between these two 
file types, so each type will used the full 32 bits for ids.

2)  The SM will maintain the map between full file_names and the 32-bit 
internal file_id.

3)  HPSS file_names will be maintained in the same map for use by the 
Cache Manager.  In general, it will possible for the HPSS directory 
structure to be different than the Unix file structure.

4)  As above, the QM will pass to the EI the bundle_id and the set of 
event_ids that qualify for this bundle.  But, in addition, the QM will 
pass an-ordered list of file_ids (or file_names in case of ROOT files) 
if this is deemed useful.  The EI will only return the bundle_id when 
releasing the bundle.

5)  This scheme will allow the usage of a mixed types of files in the 
same "federation" if that will be found to be useful.


Comments?

Arie.