runl* : client (MinimalQuery1.Linux) on linux rcas6025 and servers on rsun00
runs* : everything on rsun00 with SII
runu* : everything on rsun00 with MinimalQuery1

between runs, cache has been cleared so that fresh file transfer can happen


3/14/2000
runs0: 
	CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -c 'fzd;hist' -t 0.001
	found problem in FC (crashed): tape ID doesn't exist in some files
		:: put no-tape-id-was-fetched-from-the-hpss instead
runs00: 
	CM with -q 'dst=27536' -c 'dst' -t 0.001
	found QE has a problem selecting qualified events and file IDs
runs1: 
	fake CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -c 'fzd;hist' -t 0.001
	many files are not in HPSS, and gives wrong HPSS path, and
	those are not in the destination path either when it's permanent
runu0: 
	fake CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s  'fzd;hist' -t 0.001
	first test with the MinimalQuery on rsun00, worked well
	didin't touch any files since fake CM is used
runu00: 
	fake CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'fzd;hist' -t 0.001
runu1: 
	fake CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'fzd;hist' -t 0.001
	found QM has a race condition when the transfer rate is too high
		where in between stage request to CM and CM's response (stage
		done) to the stage request :: fixed

3/15/2000
runl2: 
	fake CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'dst;fzd;hist;runco;tags' -t 0.001
	newer QE: QE got fixed few points
	OIDs looks good, and ran well on linux machine
runs2: 
	fake CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -c 'dst;fzd;hist;runco;tags' -t 0.001
runu2: 
	fake CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'dst;fzd;hist;runco;tags' -t 0.001

runu3: 
	CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'fzd;hist' -t 0.001

runu31: 
	CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'fzd;hist' -t 0.001
	newer QE: newer index
	some files are in the cache already from the previous run,
	but the number of files and number events are different, so
	it jut saves some time to transfer files.
	newer QE: index rebuilt
	FC got changed and new ref is used, and now
		pemanent files are in /star/rcf/GC/tags/...
		non-permanent files go to /star/rcf/GC/cache/...
		before everything was in /star/rcf/GC/cache/...
	1 file was not on HPSS path that FC gave to CM, so
		1 less bundles delivered to the client
		cause: there is a possibility that CM has left over path
			information from the previuos file path
			cleared for the next run

runu32: 
	CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'fzd;hist' -t 0.001
	CM fixed from the previous run
	cache cleared from the previous run

runl3: 
	CM with -q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'fzd;hist' -t 0.001
	cache cleared from previous run

runu4: 
	CM with following queries 180 seconds apart and 300 seconds for the last
	-q 'dst==27511' -s 'dst' -t 0.001 
	-q '-15.658<qxc_2<-11.09 && 1<qyc_2<8' -s 'dst;geant' -t 0.001 
	-q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'dst;runco' -t 0.001 
	-q '24<NLa<100 && 24<NLb<100 && NXi>60' -s 'hist;runco;fzd;dst;geant' -t 0.001
	-q '0.0<mpta_1<1' -s 'hist;fzd' -t 0.001 
	-q '0.0<mpta_1<0.5' -s 'hist;fzd' -t 0.001 


	Multiple queries seemed running alright.
	The machine rsun00 was heavily used at the time, and rmine03, the 
	machine that our cache is mounted, has network access problem.
	So, the transfer rate from HPSS to local cache is very slow.
	The problem is not on HPSS side, but it's on the cache disk side.
	Even "ls" took several minutes and sometimes couldn't even CTRL-C.

Wed Mar 15 17:32:43 EST 2000

last pid:  1905;  load averages:  1.44,  1.93,  2.16                   17:31:03
195 processes: 190 sleeping, 3 zombie, 2 on cpu
CPU states: 37.2% idle, 42.8% user, 16.9% kernel,  3.2% iowait,  0.0% swap
Memory: 1280M real, 676M free, 520M swap in use, 6366M swap free

  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
20379 alexl      1   0    0 6592K 5712K cpu15 141.0H 45.54% john
 1905 asim       1  46    0 2184K 1936K cpu14   0:00  1.02% top
 1715 asim      11  58    0 4632K 3832K sleep   0:04  0.34% cmanager
26124 maguire    1  48    0 2240K 1624K sleep   3:39  0.34% top
 1882 asim       1  33    0 2968K 1232K sleep   0:34  0.08% pftp_client
  171 root       6  51    0 4944K 2272K sleep 189:39  0.07% automountd
14209 asim       1  58    0 4808K 3112K sleep   0:05  0.04% xterm
  337 root       1  58    0 4048K 2744K sleep  94:25  0.03% se.sparc.5.6
12050 root       1  58    0 2872K 1664K sleep   0:55  0.02% sshd1
28410 root       1  58    0 2752K 1008K sleep   0:16  0.02% sshd1
26783 vanyashi   1  38    0   12M 5144K sleep   0:43  0.02% xemacs-20.4
14433 mysql      7  11    0   12M 2576K sleep  20.7H  0.01% mysqld
 1719 asim       6  58    0   11M 7304K sleep   0:01  0.01% qe
  238 root       1  58    0  912K  400K sleep  20:19  0.01% utmpd
16074 root       1  58    0 2776K 1456K sleep   0:15  0.01% sshd1


	The rmine03 and rsun00 usage were very heavy, and somehow
	SSH connection to rsun00/rcf got crashed.
	and crashed all the running processes.
	will be returned to this run later.

	didn't have time to re-run this test, but it'll be re-run
	sometime soon.

runl4:
	CM with following queries 180 seconds apart and 300 seconds for the last
	-q 'dst==27511' -s 'dst' -t 0.001 
	-q '-15.658<qxc_2<-11.09 && 1<qyc_2<8' -s 'dst;geant' -t 0.001 
	-q '-17<=qxa_3<0.3 && 25>qxc_1>15' -s 'dst;runco' -t 0.001 
	-q '24<NLa<100 && 24<NLb<100 && NXi>60' -s 'hist;runco;fzd;dst;geant' -t 0.001
	-q '0.0<mpta_1<1' -s 'hist;fzd' -t 0.001 
	-q '0.0<mpta_1<0.5' -s 'hist;fzd' -t 0.001 

	Multiple queries seemed running alright.
	The rsun00 has some link problem with rmine03 where cache direcory is
	mounted/linked. "ls" takes too much time for CM.  CM got stuck
	in somewhere after the file transfer is done and before it calculates
	STATS and talks to QM for stage_done.

	New pftp error was found: when the cache disk access is not possible,
	the file transfer just hangs, and the pftp connection gets
	timed-out. It seemed that CM doesn't know how to handle the
	situation. (it never happened before, because the bottle neck
	so far was on HPSS side, not on cache disk cnnection).


Source (HPSS) file: rcf0100_07_1290evts.geant.root
Destination file: /star/rcf/GC/cache/auau200/mevsim/cascade/central/year_1h/hadronic_on/tfs_6/rcf0100_07_1290evts.geant.root
HPSS hostname: rmds01
Local hostname: rsun00
PWD: /star/rcf/GC/MDC3
Date: Wed Mar 15 18:39:21 EST 2000
************************ start pftp ******************************
Multinode is Disabled.
Connected to rmds01.rcf.bnl.gov.
220 rmds01 FTP server (HPSS 4.1 PFTPD V1.1.2 Wed Feb 2 08:16:24 EST 2000) ready.
Remote system type is UNIX.
Using binary mode to transfer files.
331 Password required for starreco.
230 User starreco logged in.
200 Type set to I.
250 CWD command successful.
200 Command Complete (1457364668, rcf0100_07_1290evts.geant.root, 0, 1, 1048576).
200 Command Complete.
150 Transfer starting.
226 Transfer Complete.(moved = 536870912).
200 Command Complete.
150 Transfer starting.
226 Transfer Complete.(moved = 536870912).
421 Timeout (900 seconds): closing control connection.
************************* end pftp *******************************
Date: Wed Mar 15 19:10:07 EST 2000


	instead of /star/rcf/GC/cache, /star/rcf/data06/GC/cache/ may be used
	in FC and therefore in CM.

	Problem found in EI, when no more bundles are returned from QM
	during retrieval, in case the rest of the bundles are in errors.
	Client part of EI keeps asking getFids.

	CM shows a bug also. HPSS pftp doesn't handle when the remote
	file path and local file path are too long. So, we had to
	break remote file path into two parts: one for remote file 
	directory and another for remote file name. The remote file directory
	produces some garbage characters sometimes, where it couldn't
	file the remote file name. It needs to be fixed soon.

	QM needs a fix regarding permanent files, depending on
	whether or not CM needs to transfer the permanent from HPSS to local
	permanent path in case the permanent file is not in the path. 
	Currently, QM does not ask CM for a permanent file to be staged, nor 
	be purged from local disk. It only delivers the permanent file ID
	along with cached non-permanent file IDs to the client, and keeps
	track of which file is permanent. i.e. QM assumes the permanent
	files are already in local disk.

	These fixes will be done in the near future, maybe tomorrow.


idxupdate:
	Index update test
	look at the directory for more details