RE: What else to log

["Doug Olson" <dlolson@lbl.gov>]

I agree that we should log a lot more.  My comment was really just
about the presentation in the netlogger display. There can easily be
thousands of items to log from the iterator for each file.
If you try to plot that in the netlogger display you showed,
the storage manager items in the display will be compressed to a
single line at the bottom of the picture.
Ideally one would like to be able to measure how many bytes
from each file are actually used.  This can be from
measuring how many event components are accessed in a file plus
how much of each component is actually used.  I don't know
exactly how to get this information, but if we did it would
be a lot.

Doug

> -----Original Message-----
> From: Arie Shoshani [mailto:arie@math.lbl.gov]
> Sent: Wednesday, December 16, 1998 8:56 PM
> To: gcdev@sseos.lbl.gov; dlolson@lbl.gov
> Subject: RE: What else to log
> 
> 
> 
> > Maybe ploting when the OOI returns the first and last EI for each
> > file is interesting as a kind of consistency check.  Plotting
> > an "event" (vertical axis item) for every EI returned by the OOI
> > won't work because there are easily too many for that type of plot.
> > I am curious what Dave M thinks.
> > 
> > Doug
> 
> Logging first and last EI for each file is OK, but it is not enough 
> to undestand why things are not what we expect for user code 
> behavior.  To understand if it is the fault of the network or 
> something else, more detailed logging is needed.   At least during 
> testing, it may be worth recording what's happening in more detail.
> 
> Comments?
> 
> Arie.
> 
> 
> > > -----Original Message-----
> > > From: Arie Shoshani [mailto:arie@math.lbl.gov]
> > > Sent: Tuesday, December 15, 1998 1:55 PM
> > > To: DLOlson@lbl.gov; malon@anl.gov
> > > Cc: gcdev@sseos.lbl.gov
> > > Subject: What else to log
> > > 
> > > 
> > > Doug,
> > > 
> > > I saw in the minutes of the last conference call that you 
> > > said:
> > > 
> > > "It will be good to think about what information might be 
> > > available in the iterator that could be added to these plots 
> > > as well."
> > > 
> > > I discussed this with Luis and here is what we think:
> > > 
> > > When we plot log information now, we sometimes see long 
> > > processing time.  See for example:
> > > http://gizmo.lbl.gov/sm/netlogger.images/oct4th_run1.gif
> > > 
> > > The long purple lines show that there is a long delay 
> > > between the time that a file was given to the EI 
> > > (file_pushed) and the time it started to use it 
> > > (file_retrieve).  In addition, there are unusually long 
> > > processing times for each file after it is retrieved.  From 
> > > this plot alone we have no idea why.
> > > 
> > > Thus, I think it will be useful to log what the EI is doing: 
> > > the time that each event is read and passed to the User Code 
> > > (UC), the time that the UC is done processing each event, 
> > > etc.  This will be more important to know when events are 
> > > being passed over the network to Linux boxes.
> > > 
> > > As to the plotting of such information, we think it may be 
> > > possible to put that on the same netlogger plot, but the 
> > > scale of event processing is much smaller that file caching, 
> > > so we may want to display that on a separate plot.
> > > 
> > > Comment?
> > > 
> > > Arie.
> > > 
> > > 
> 
> Arie Shoshani
> Mail Stop 50B/3238
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Rd.
> Berkeley, CA 94720
> 
> Tel: (510) 486-5171
> Fax: (510) 486-4004
> 
> Email: shoshani@lbl.gov
> URL: http://www.lbl.gov/~arie
> 
> 
>