Computing Sections of the Prop osal for A TLAS

Detector Dev elopmen t and Computing Preparatio ns

at the Univ ersit y of Mic higan

Myron Campb ell, Ja y Chapman, Homer Neal, Jianming Qian

Greg T arl  e, Rudi Th un, and Bing Zhou

with R ese ar ch sta

Rob ert Ball, Edw ard Diehl, Stev en Goldfarb, Suen Hou, Daniel Levin, and Sha wn McKee

R andal l L ab or atory of Physics

University of Michigan

A nn A rb or, MI 48109-120

Con ten ts

1 In tro duction 1

2 Mic higan’s Activities in the A TLAS Collab oration 2

2.1 Review of A TLAS Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.1.1 In tro duction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.2 Observ ations and ndings .. ... .. .. .. .. ... .. .. .. .. .. 3

2.1.3 Recomme ndations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.4 Arc hitectural and Distributed Data Handling Issues . . . . . . . . . . . . 3

3 Mic higan’s Sim ulation Activities 4

3.1 MDT Detector Sim ulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3.2 Studies of Data Flo w with V erilogHDL .. .. .. .. .. ... .. .. .. .. .. 5

4 Mic higan Plans for A TLAS Computation 7

4.1 Computing Infrastructure at Mic higan .. .. .. .. .. ... .. .. .. .. .. 8

4.1.1 The UoM A TLAS Computation Group . . . . . . . . . . . . . . . . . . . 9

4.2 Initial Areas of In terest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.2.1 Global T rigger and Ev en t Selection Database . . . . . . . . . . . . . . . . 9

4.2.2 Muon Geometry Database . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.2.3 Muon Detector Sim ulation and Ph ysics Studies with Com bined Detector

P erformance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.2.4 Collab oratory T ools.. .. .. ... .. .. .. .. ... .. .. .. .. .. 11

4.2.5 Remote Access Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.2.6 High P erformance Computing Resources and Metacomputing . . . . . . . 13

4.2.7 Upgrade of Mic higan’s Computing Resources . . . . . . . . . . . . . . . . 13

5 Summary 14

A TLAS

1 In tro duction

In the Spring of 1997 Univ ersit y of Mic higan Ph ysicists, Rob ert Ball, Myron Campb ell, J.

Chapman, Homer Neal, Jianming Qian, Greg T arl  e, and Andy T omasc h applied for Institutional

mem b ership in the A TLAS collab oration. F ollo wing the pro cedures of the organization, the

A TLAS collab oration accepted Mic higan at their rst opp ortunit y in the fall of 1997. Since

then additional individuals at Mic higan ha v e joined the team: Bing Zhou as a new facult y

mem b er, Ed Diehl as a new researc h scien tist, and Rudi Th un, Dan Levin, and Sha wn McKee

as mem b ers of the Departmen t recen tly attracted to the pro ject. A t presen t the A TLAS group

is comp osed of 7 facult y , 4 researc h scien tists, 3 mec hanical tec hnicians, and an electronics

engineer. Sev eral graduate studen ts are w orking on A TLAS pro jects but will lik ely b e directed

to CDF and D¹ data for thesis w ork at the T ev atron as Run I I b egins.

Discussions with the US A TLAS comm uni t y b egan coinciden t with our application for

mem b ership and Mic higan w as almost imm ediatel y included in the Muon Pro ject planning.

Since then w eha v emo v ed forw ard at a rapid place, signing a Memorandum of Understanding

with US A TLAS, reno v ating our high ba y assem bly area for m uon c ham b er construction, and

assuming a cen tral role in the testing and certication of fron t-end electronics for the m uon

MDT system. This latter pro ject has b een expanded to include data o w sim ulations and

the design of on-c ham b er data concen trators. These ma jor eorts ha v e b een coupled with

the con tin uation at Mic higan of lev el 2 m uon trigger w ork started b y Bing Zhou at Boston

Univ ersit y and b y R&D eorts on measuremen t resolution for straigh t tub es with wires under

gra vitational sag. An asso ciation with the TR T detector of A TLAS has also b egun with our

electronics engineer con tributing the Dela yLoc k Lo op design for the fron t-end time digitizer

of the TR T subsystem. It is no w established that Mic higan will:

 Build © 40,000 tub es (man y of the longest) and assem ble them in to the largest c ham b ers

of the m uon MDT endcap. P erform the needed R&D to establish a viable design for the

6 meter tub es of these c ham b ers.

 Manage and p erform the A TLAS m uon fron t-end design certication. This includes

pro duction of 10K c hannels of the design for c ham b er testing. An adjunct of this w ork is

the dev elopmen t of a windo ws based mini-data acquisition system to pro vide a mo dern

\what y ou see is what y ou get" or wysiwyg test and con trol system.

 Con tribute to the design of the TR T fron t-end digitizer and readout c hip, DTMR OC.

 Rene the lev el 2 trigger algorithms and ev aluate the success of these algorithms with

Mon te Carlo sim ulations.

 Pursue a computing and soft w are role that utilizes the considerable resources in infras-

tructure and p ersonnel at Mic higan. This will include incorp oration of collab oratory to ols

in to A TLAS to facilitate distributed researc h, early connection to the In ternet I I, the es-

tablishmen t of a program to assem ble and certify the sim ulation and trac king co de for

1

the m uon MDT, and p ossible dev elopmen t of general purp ose to ols for ob ject-orien ted

(or ob ject-relational) database handling of information for trigger and ev en t selection.

 Assist A TLAS in a review of its state of readiness for all asp ects of o-line computing.

 Con tin ue in v olv eme n t in the nation wide REU (researc h exp erience for undergraduate

program funded b y the NSF) to pro vide opp ortunities for U.S. studen ts to w ork on LHC

pro jects at CERN, as w ell as to engage in the recen tly added program whic h includes

U.S. high sc ho ol teac hers.

2 Mic higan’s Activities in the A TLAS Collab oration

Mic higan’s is con tributing to A TLAS in the areas where it has established strengths and also

in areas where mem b ers of the group see imp ortan t future requiremen ts that are not fully

represen ted in the collab oration as a whole. Those areas where w e can immedi ately con tribute

are represen ted b y the building MDT c ham b ers and the electronics instrumen tation of the MDT

and TR T detectors. With a strong bac kground in data acquisition and lev el 2 triggering, w e

exp ect to con tribute to these areas as w ell, fo cusing on the same MDT detector elemen ts. With

resp ect to soft w are and database dev elopmen t, w eha v ein terest in expanding our bac kground

in to the tec hnology of ob ject orien ted analysis co ding and database structure. It is in these

areas w ere w e are anxious to b ecome learners and then leaders b ecause w e b eliev e that only

with these mo dern to ols can individual ph ysicist hop e to comprehend and con trol the complex

and massiv e data from suc h a large exp erimen t. The list b elo win tro duces the topics that

follo w starting with the c ham b er construction and electronics w ork that is w ell underw a y and

ending with more general though ts on ho w Mic higan’s A TLAS team wishes to b e engaged in

computing.

 Cham b er Construction for the Muon MDT Subsystem

 Electronics Dev elopmen t & Data Acquisition in supp ort of the Muon MDT Detector T ests

 Muon Lev el 2 T rigger Algorithm Dev elopmen t

 F ron t-end Electronics Design and T esting for the TR T Subsystem

 A TLAS Computing Issues (Global, US, and Mic higan Plans)

2.1 Review of A TLAS Computing

Homer Neal c haired the A TLAS Review of Computing Committee that submitted its nal

rep ort in late F ebruary 1999. This commi ttee w as assem bled to assess whether A TLAS com-

puting is on course to meet the needs of the pro ject during the construction, commissioning

and running of the exp erimen t. The rep ort con tains man y relev an t observ ations and recom-

mendations. The full rep ort is a v ailable on the CERN w eb[1]. The list of quotes b elo w is tak en

directly from the executiv e summary of the rep ort. The sp ecic quotes are c hosen b ecause of

2

their relev ance to the con tributions w e prop ose to mak e as stated in section 4. W e exp ect to

expand up on the ne con tribution b y Homer Neal with eorts directed to w ard closing some of

the gaps iden tied b y the committee. The list of quotes from the rep ort has four subsections,

in tro duction, observ ations, recommendations, and arc hitectural and distributed data handling

issues. Muc h of the rep ort fo cuses on the detector p erformance ev aluation soft w are, database

soft w are, the need for engagemen t of the soft w are talen t within the collab oration, and on the

dev elopmen t of collab oratory to ols for eectiv e co ordination of all eorts.

2.1.1 In tro duction

 \The ob jectiv e of this review is to assess the status of the strategic planning and progress

of A TLAS computing and to recommend actions that will help ac hiev e the collab oration’s

goals in this imp ortan t area".

2.1.2 Observ ations and ndings

 \W e nd a p oten tially serious shortage of ph ysicists dev oting ma jor atten tion to A TLAS

soft w are issues".

 \W e nd a need for impro v ed emphasis on training to successfully tap the existing A TLAS

talen t to help prepare the nal A TLAS soft w are".

 \W e b eliev e that A TLAS could do m uc h more to facilitate in teractions b et w een the in-

stitutes and CERN b y taking a leadership role in relev an t collab oratory R/D".

2.1.3 Recommendations

 \Detector-sim ulation, reconstruction and com bined-p erformance groups should b e as-

signed, together with the exp erts w orking on OO soft w are to da y , the task of pro ducing the

new detector sim ulation and reconstruction soft w are for A TLAS. Clear and w ell-dened

milestones should b e set".

 \It is essen tial that a common detector geometry b e used for b oth sim ulation and re-

construction. If the new geometry is dela y ed, existing sim ulation geometry should b e

temp orarily transferred, so that reconstruction soft w are dev elopmen t can con tin ue".

2.1.4 Arc hitectural and Distributed Data Handling Issues

 \Skilled p eople m ust b e found in the Collab oration as so on as p ossible".

 \Guidance should b e pro vided on the desired structure of database systems for the collec-

tion and storage of data from sub detector comp onen t fabrication and testing op erations".

 \W e recomme nd that collab oratory R/D initiativ es relev an t to the needs of A TLAS con-

tin ue and b e giv en the supp ort of the A TLAS managemen t".

3

3 Mic higan’s Sim ulation Activities

Sim ulation w ork is an increasingly imp ortan t part of detector design for to da ys large exp er-

imen ts. W ork has b een done in t w o distinct areas for A TLAS. One area is that of trac king

sim ulation for tub es with wires that are allo w ed to gra vitationally sag without supp orts and

without complex tub e b ending to follo w the wire shap e. The question is whether the required

trac k-p osition accuracy can b e obtained in this simple conguration. The second sim ulation

attempts to dene the required transfer sp eed and buering for the MDT fron t-end readout to

insure that no data is lost in buer o v er o ws.

3.1 MDT Detector Sim ulations

An essen tial part of our long tub e R&D eort has b een to ev aluate the c ham b er p erformance for

realistic time-to-space functions. With this ob jectiv e in mind w eha v e congured a sequence of

programs to sim ulate the ph ysics pro cesses, to sim ulate the detector resp onse, and to reconstruct

the original ph ysical quan tities to examine the p erformance of the detector. The sequence is

describ ed b elo w.

 Mo del the MDT resp onse.

W e implem en ted the A TLAS MDT tub e conguration and op eration conditions in to the GARFIELD program pac k age. Using this program w eha v e mo deled the MDT electriceld distributions for dieren t amoun ts of wire sag and calculated the corresp ondingdrift v elo cities for dieren t gas mixtures. W e studied the dep endence of drift v elo cit yon op erating conditions: temp erature, pressure and high v oltage. Based on the elddistribution inside the tub e and the drift v elo cit y whic h is a function of the eld, w e

pro duced man y corresp onding space-time functions (the drift distance vs. the drift time)

for the MDT. The mo deled MDT resp onses w ere compared to measuremen ts to v erify

their accuracy . These resp onses w ere then parameterized and input to the next stage of

sim ulation studies.

 Muon sp ectrometer sim ulation.

W e set up a GEANT based program, LHCTOR to sim ulate the A TLAS m uon sp ectrom-

eter. This program accurately mo dels the A TLAS magnetic eld and the detector mass

proles for the inner trac k er and the calorimeters. It particularly describ es the m uon de-

tector materials and la y out in detail. LHCTOR also pro vides an in terface to the ph ysics

ev en t generator program PYTHIA .W e used this com bination of programs pro duce ev en ts

con taining m uons, trac k the m uons through the A TLAS detector, and to pro duce sim u-

lated \m uon hits" as the m uon tra jectory crosses the sensitiv e elemen ts of the detector.

The \m uon hits" pro vide input to the next stage of the sim ulation.

 Muon trac k reconstruction.

The trac k reconstruction w as carried out b y another program pac k age MUONBO X .W e

dev elop ed an in terface program whic h p ermits us to use the parameterized MDT space-

time functions in the digitization pro cess within the program. Th us, w e can ev aluate the

4

tub e op eration with wires that sag and determine the eect that wire sag has on m uon

momen tum measuremen ts.

 Analysis programs

Our nal analysis w as p erformed on n tuples that w e output from the MUONBO X co de.

This ga v e us maxim um exibilit y in the examination of in teresting ph ysics quan tities and

in the tuning of v ariables that pro duced the nal results.

With the computing programs describ ed ab o v e fully debugged and tested, w e are in a p osition to

pro duce fast and accurate results on detector p erformance for ph ysics studies. W e will con tin ue

these sim ulations and ph ysics analysis b y further dev elopmen t of the A TLAS in tegrated soft w are

pac k ages: the DICE [2] program whic h sim ulates en tire A TLAS detectors in detail including

EM calorimetry , tile calorimeter, and the m uon system; the A TRECON program [3 ] whic h

con tains the reconstruction co de for eac h sub-detector, including the MUONBO X pac k age for

the m uon system. Our new eorts will include a more realistic mo deling of MDT parameters

than w as done for wire sag studies. Additional eort will b e directed to the co de transitions

from F OR TRAN to C++, Ob ject Orien ted (OO) pac k age. W e also an ticipate an activ e role in

the con v ersion of the analysis pac k ages in to the C++ en vironmen t.

3.2 Studies of Data Flo w with V erilogHDL

T o examine the A TLAS m uon fron t-end electronics design, a full electronic sim ulation w as

constructed. This sim ulation is implem en ted in the Hardw are Description Language V erilog

(V erilogHDL) and pro vides a testb ed for studying p erformance of the design as w ell as b eing the

source to generate actual circuits in either FPGAs or ASICs. The TDC is already sp ecied in

V erilog, syn thesized, fabricated, and tested. The data o w sim ulation includes a sim ulation of

the TDC, the CSM, and the output sequencer to a b er driv er. Sim ulations for the b er driv er

em ulate a Cypress Hot-Link running at 25ns/b yte and a b er c hannel running at 50ns/w ord.

These c hoices are made as a pro of of feasibilit y and are not considered as A TLAS decisions.

The V erilog sim ulation uses a realistic sto c hastic selection of ev en t times and a P oisson dis-

tribution of n um b er of hits p er ev en t p er TDC. It buers the hits in the TDC part of the

sim ulation and serializes the data from this buer in the sp ecic proto col used b y the A TLAS

TDC design. The 18 serial streams are then con v erted bac k to parallel in the CSM sim ulation

and loaded in to short c hannel buers. The output from the 18 c hannel buers is p olled b y

m ultiplexe r and when data is found, it is transferred to a single output buer whose con ten ts

are systematically sen t to the b er driv er.

The c ham b er hit rates from the ph ysics Mon te Carlo for the highest rate c ham b ers when scaled

5x lead to c ham b ers running as high as 500kHz in an extreme example. In this extreme example

some hits are lost within the round tub es and within the TDC lev el 1 buering. A more t ypical

high c ham b er rate is less than 300kHz. The 300kHz to 500kHz wire rate when com bined with

the drift time windo w and the maxim um acceptable trigger rate (100kHz in A TLAS) leads to 5

to 9 hits/TDC ev ery 10  s. The question of whether the CSM design is appropriately sp ecied

is essen tially whether the FIF O (rst in rst out) buers in the CSM can spread out the arriv al

times of hits from the 18 TDCs, m ultiplex them in a single buer, and sync hronously sequence

5

TDC Output FIFO Occupancy

0

50000

100000

150000

200000250000

300000

350000

1

6

11

16

21

26

31

36

41

46

51

56

61

Number of Words

Frequency

6 Hits/TDC

5 Hits/TDC

7 Hits/TDC

9 Hits/TDC

CSM Output FIFO Occupancy (80 MByte/s)

0

50000

100000

150000

200000250000

300000

12

Number of Words

Frequency

5 Hits/TDC

7 Hits/TDC

9 Hits/TDC

Figure 1: TDC FIF O buer o c cup ancies for in-

put r ates of 5 to 9 hits/TDC channel. The physics

simulations suggest a maximum r ate of 5 when

the standar dA TLAS safety factor of 5 is applie d.

A maximum aver age r ate of 9 is al lowe d by the

TDC’s output serial link.

Figure 2: CSM output FIF Oo c cup ancy for input

r ates of 5 to 9 hits/TDC channel. A shift fr om a

majority of 1 to a majority of 2 data units in the

FIF Ois se en as the r ate incr e ases fr om5to9

hits/TDC.

them to the b er driv er without losing an y due to buer o v er o ws. Within the CSM there are

t w o dieren t FIF O sizes. The CSM has a small (4 depth) FIF O on eac h of the 18 TDC outputs.

This FIF O seldom con tains more than 1 digitization since it is loaded from a serial input that

requires 35 clo c k steps to assem ble a w ord and is emptied once ev ery 36 clo c k steps in the w orst

case and is often emptied in man y few er clo c k steps. The other FIF O could b e large since it

holds data from all c hannels. It is lled ev ery clo c k step if data is receiv ed from one of the 18

c hannel FIF Os and is emptied once ev ery 2 or 4 clo c k steps as the output b er driv er requests

another w ord to transmit. The 2 steps is for a b er c hannel that tak es one 32-bit w ord ev ery

50ns and the 4 steps is for the Hot-Link driv er whic h accepts a b yte eac h 25ns. It is this last

buer that requires sizing.

T o minimi ze the data v olume burden on the b er, the iden tifying TDC header and trailer

w ords (w ords that iden tify the ev en t and b eam crossing asso ciated with the trigger b eing

pro cessed) are stripp ed b y the CSM as ev en ts are assem bled for transmission to the b er. A

single CSM header and trailer is added to pro vide these same ev en t and b eam crossing IDs.

The sim ulations w ere run with 5, 7, and 9 hits/TDC as sho wn in Fig. 1. The sim ulations

(not sho wn) quic kly indicated that for rates ab o v e 6 hits/TDC a Hot-Link b er transmitter

accepting 8 data bits eac h 25ns could not handle the rate. F or rates ab o v e 6 the total data

o w exceeds the Hot-Link capacit y and th us no amoun t of buering can succeed in a v oiding

data loss. A b er c hannel link can accommo date the a v erage rate with up to 10 hits/TDC.

The distribution of data units in the output FIF Ow as studied for individual TDC rates from

5 to 9 hits/TDC. Fig. 2 sho ws the o ccupancy in the 2 cells of a CSM output FIF O for 5, 7,

and 9 hits/TDC. A shift from a ma jorit y of 1 hit to 2 hits with rate is sho wn. With these

sim ulations w e are able to select a design that has the appropriate capacit yto a v oid data loss

while minimi zi ng the circuit costs. The data rate limitation will remain with the c ham b ers

abilit y to digitize hits adjacen t in time and not imp osed b y the electronics.

6

The hardw are description in v erilog will b e expanded to include the TSC mo dule to complete

the sim ulation up to the R OB (ReadOut Buer) of the data acquisition system. Mic higan has

tak en the role of dev eloping and main taining the data o w sim ulations for the m uon MDT

system. As data b egins to b e tak en w e will b e able to use the sim ulations to ascertain the

readout p erformance of the system for future op eration at increasing luminosities.

4 Mic higan Plans for A TLAS Computation

The Univ ersit y of Mic higan A TLAS Group views participation in A TLAS Computing activities

to b e a v ery imp ortan t part of its o v erall researc h program. A strong computing base is essen tial

not only to our hardw are eorts, but to the ac hiev em en t of our ultimate goal of carrying out the

successful ph ysics studies made p ossible b y the exp erimen t. F urthermore, w e b eliev ew eha v e

all the comp onen ts necessary to mak e m a jor con tributions to the A TLAS computing eort.

First, w e are lo cated at a univ ersit y in the forefron t of computing researc h and that op erates

state of the art compute-in tensiv e and data-in tensiv e facilities and second, w eha v e a critical

mass of qualied individuals who can con tribute, at the v ery highest lev el. The computing

activities already underw a y for A TLAS are describ ed ab o v e and include:

 Lev el 2 Muon T rigger Sim ulations

 Muon T rac king Studies of Resolution with O-Cen ter Wires

 Muon F ron t-end Data Flo w Sim ulations with V erilogHDL

Because of the broad sp ectrum of m utually supp orting computational talen ts and in terests

presen t within the UM A TLAS eort, w eha v e decided to create, in our T ask A group, a cen ter

for computation and soft w are dev elopmen t. W e exp ect that this structure will help us prioritize

our v arious A TLAS computing eorts and to p ermit us to b enet from discussions of k ey issues

in a group setting. Moreo v er, suc h a c o ordinated eort will enhance our abilit yto w ork with

other soft w are cen ters within the collab oration, share the results of our w ork, and pro vide

an economical approac h to establishing and main taining a lo cal rep ository of curren tA TLAS

soft w are.

Key to our in terest in A TLAS soft w are is, of course, our desire to exploit the ph ysics from the

exp erimen t. A successful eort in doing this will require mem b ers of the group with detailed

kno wledge of A TLAS ph ysics sim ulation studies, with the OO sim ulation and reconstruction

pac k ages, and with the OO (or OR) databases. Moreo v er, our successful m uon hardw are

pro duction tasks, and the design of an ecien t trigger/D A Q OO (or OR) database, will require

man y of the same talen ts. F ortunately , these talen ts do exist within our group. In addition, a

program is en visioned where a gro wing n um b er of our group mem b ers will b ecome exp ert with

the "new" OO/C++ A TLAS soft w are in the coming y ear.

Bing Zhou has had a long in v olv eme n t with A TLAS sim ulation studies and that exp erience,

along with her leadership role in the m uon c ham b er construction pro ject, will pro vide an excel-

len t foundation for the prop osed computational studies, dra wing up on the supp ort pro vided b y

7

individuals in the group suc h as Dan Levin, Stev e Goldfarb and Sha wn McKee. In the area of

databases, w e exp ect the activ ein v olv em en t of Homer Neal, who dev elop ed the initial database

for the DZER O cen tral calorimeter, along with Levin, Goldfarb, and McKee, who ha v e b egun

to ev aluate OO database paradigms for the m uon pro duction pro ject, as w ell as Myron Camp-

b ell and Ja y Chapman, who ha v e b egun to ev aluate approac hes for the A TLAS trigger/D A Q

database. F urthermore, w e exp ect to deplo y our sim ulation skills to regularly ev aluate pro duc-

tion issues that b ear on the ultimate ph ysics p erformance of the m uon c ham b ers.

It it our goal to mak e substan tial additional con tributions to the exp erimen t within the domain

of soft w are and computing, capitalizing on the computing exp ertise at Mic higan b oth within

the A TLAS group and within the greater univ ersit y comm unit y . In the sections b elo ww e rst

indicate the computing resources at Mic higan. In the second section w e outline Mic higan’s

plans as w e see them at this time. As of this writing, the organization of computing for A TLAS

is in a state of considerable ux. A ma jor review of A TLAS Computing has just b een completed

b y a committee c haired b y Homer A. Neal whic h calls for an accelerated set of steps to prepare

the new soft w are required for A TLAS. In addition, a vigorous eort is underw a yno w in the U.S.

to dene the structure and resp onsibilities for a US A TLAS Computing en tit y . As this pro cess

tak es shap e w e exp ect Mic higan to assume a prominen t role. In the mean time w eha v e directed

our energies in to a few areas where w e b eliev e atten tion is needed and where w ean ticipate our

ultimate eorts will reside.

4.1 Computing Infrastructure at Mic higan

Mic higan is unique in sev eral asp ects of computing imp ortan tto A TLAS. 1) The Univ ersit y

is lo cated in Ann Arb or where the In ternet I I pro ject is based. In realit y the pro ject is a

spino of Univ ersit y activit y . Giv en the imp ortance of high bandwidth connectivit y to HEP ,

w eha v e already discussed the need for Ph ysics to b e the rst Univ ersit y Departmen t connected

with full bandwidth to In ternet I I. 2) The Sc ho ol of Information has engaged CERN in dialog

and has a program in place to examine the nature and needs of the A TLAS exp erimen t for

collab orativ e to ols for distributed researc h activit y . 3) The Univ ersit y through its Visualization

Lab oratory and Media Union has programs that oer ma jor con tributions to the handling and

viewing of data. 4) Bill Martin is the pro ject director on a ma jor gran t that pro vides facilities

and infrastructure (h uman resources and ph ysical facilities) for high p erformance computing,

including compute-in tensiv e and data-in tensiv e computing. This gran t is funded through the

NSF P artnerships for Adv anced Computational Infrastructure (P A CI) program, whic h has as its

express mission the dev elopmen t and deplo ymen t of tera ops computing facilities and p etab yte

data facilities for the US. The UM is b oth a compute-in tensiv e and data-in tensiv e institution in

this partnership (along with UCSD, Caltec h, Berk eley , and the UT-Austin) and can oer access

to these facilities and capabilities for the UM participan ts in this pro ject. 5) Barbara O’Keefe

of the Media Union leads a Kno wledge and Distributed In telligence group. Initial con tact has

b een made with eac h of the computing units represen ted ab o v e and areas of m utual in terest

ha v e b een iden tied.

8

4.1.1 The UoM A TLAS Computation Group

Mem b ers of the A TLAS group with plans to engage in computing include Bing Zhou, Daniel

Levin, Stev e Goldfarb, Sha wn McKee, Homer Neal, J. Chapman, and Rob ert Ball. W e exp ect

this group to b e augmen ted with studen ts and computer sp ecialist as pro jects b ecome b etter

dened.

4.2 Initial Areas of In terest

W e view the soft w are activities of our group as b eing fo cussed on:

 A Global T rigger and Ev en t Selection Database

 Muon Cham b er Pro duction, Calibration and Geometry Database

 Muon Detector Sim ulation, Reconstruction, and Com bined P erformance/Ph ysics Issues

 Collab orativ eT o ols (including net w orking and remote access)

 Upgrade of Lo cal Computing Resources

4.2.1 Global T rigger and Ev en t Selection Database

W e think that mo dern database tec hnology needs to b e globally applied to the problem of

dening and monitoring the hardw are and soft w are pro cess referred to as triggering. T o pre-

cisely dene the eciency of particular ph ysics c hannels requires one to kno w the detailed

path and selections made for all trigger routes through whic h a particular ev en t top ology has

passed. This includes the hardw are logic that accepts ev en ts at the earliest p oin t in the path

(m uc h of whic h is Field Programmable Gate Arra y , FPGA, co ded in to da ys detectors), the

programmable parameters do wnloaded in to the hardw are, the algorithms co ded in to the lev el 2

pro cessors, all parameters used in the ev en t acceptance, and similar algorithms and parameters

used in the oine pro cessing. The totalit y of these accept or reject steps m ust b e understo o d

for eac h of the detector subsystems. An additional complication deriv es from the time ev olution

of these criteria as the detector and soft w are mature. The magnitude of this problem suggest

a global approac h to storing and accessing this information. Ob ject orien ted (or ob ject rela-

tional, OR) databases most naturally meet these needs since the en tities that m ust b e sp ecied

to fully dene the ev en t path range from do wnloaded binary n um b ers to algorithms in co de.

As an adjunct activit y to the in v estigation of ob ject orien ted (or ob ject relational) database

tec hnology ,w e plan to examine the options for dening the full range of ob jects needed to

sp ecify a trigger path. If the initial in v estigation is promising, the next step will b e to b egin an

information collection eort, p olling all detector groups for the c haracteristics of their trigger

and selection co de. The goal will b e to pro vide the collab oration with a mec hanism to store and

retriev e eac h and ev ery ob ject that con trols the ev en t ltering pro cess from hardw are trigger

to oine selection cuts. When structured in to suc h a database the pro cess can, hop efully ,be

a transparen t tree whose ap ex is a ph ysics category and whose branc h-ends are parameters

9

do wnloaded in to hardw are or used in selection and pro cessing. This common database w ould

b e accessed for run time initialization, ev en t pro cessing, and Mon te Carlo calculations.

4.2.2 Muon Geometry Database

The w ork on geometry databases represen ts a logical activit y for a group that is in v olv ed in the

design and fabrication of a signican t detector segmen t. It includes trac king the pro duction

sequence, deciding on what elemen ts of the pro duction data are to b e arc hiv ed, exp orted, and

inserted in to the o v erall geometry database, dev eloping the to ols to query the database for

ph ysics analysis, and then doing the analysis.

During the onset of the detector construction, our group, in tends to implem en t a robust

database for storage of the pro duction parameters for the forw ard m uon sp ectrometer. The

op erating conditions and geometry for all c ham b ers and their sensitivities to c hanges in en viron-

men tal conditions will need to b e recorded. T ests will b egin imme diately on v arious relational

and OO database options. W e will w ork closely with CERN and US A TLAS m uon colleagues

in the selection of the nal pro duct. Discussions are already underw a y with the author of the

CRIST AL database at CERN ab out the suitabilit y of that soft w are for our needs. W e exp ect

to launc hanev aluation so on.

4.2.3 Muon Detector Sim ulation and Ph ysics Studies with Com bined Detector

P erformance

As describ ed in this prop osal, mem b ers of the Mic higan group w ere hea vily in v olv ed in sim ula-

tion w ork for the A TLAS Muon T ec hnical Design Rep ort. The w ork included extensiv e studies

of the m uon detector design, p erformance, and construction. W e feel that con tin ued eort on

m uon detector sim ulations, reconstruction and p erformance analysis will b e needed to b etter

understand the detector resp onse and that this additional w ork should b e undertak en while

the detector is b eing built. Exp erience gained from the A TLAS ph ysics studies, has con vinced

us that w em ust dev ote m uc h more eort to w ard m uon reconstruction when com bined with

information from other detector subsystems, i.e. , reconstruction of the m uon spatial tra jectory

and momen tum join tly with measuremen ts from the inner trac k er. This collectiv et m ust

include corrections for the energy loss in the calorimeters. A clear understanding of the full

reconstruction capabilit yof A TLAS is essen tial if w e are to maximize disco v ery sensitivit y and

p oten tial of the detector.

F or detector sim ulation and reconstruction w e will rst fo cus on mo deling the detector sp eci-

cations and resp onse parameters. The details of these sp ecications and parameters are crucial

in sim ulating the detector p erformance. Examples include:

 The wire sag due to gra vit y and electrostatic force m ust b e included in the soft w are co de,

since eac h wire serv es as a trac king co ordinate.

 Corrections for the drift tub e space-time function asymmetry (due to wire sag) should,

in principle, b e implem e n ted in the reconstruction program.

10

 A temp erature, pressure, and HV database con taining the parameters for the space-time

function m ust b e impleme n ted.

 A T-zero calibration database and co de to mak e appropriate use of it should b e imple-

men ted.

The ab o v e men tioned reconstruction eort, is complem en tary to the database dev elopmen t

program an ticipated at Mic higan, since success of the reconstruction eort will dep end on

access to the related database. W e also w an t to emphasize that a determination of sim ulation

parameters for mo deling basic p erformance data represen ts a unique con tribution to the A TLAS

m uon soft w are pac k age. So far, no signican t eort has b een directed to this imp ortan t need.

F or com bined p erformance ph ysics studies, w e will use the most promising Higgs disco v ery

c hannels with m uon nal states to dev elop the programs for nal m uon momen tum reconstruc-

tion. It is v ery clear to us that a realistic full detector p erformance com bination is imp ortan tto

truly understand the ph ysics reac h at the LHC with the A TLAS exp erimen t. W e also note that

great eort will b e required to pro duce a fully debugged and tested soft w are pac k age in time

for the LHC’s turn-on. Mic higan ph ysicists are exp ected to b e vital con tributors to the eort.

Indeed, w eha v e already started testing and dev eloping the com bined p erformance soft w are in

Ann Arb or in collab oration with Boston Univ ersit yph ysicists.

In our dev elopmen t pro cess w eha v e set for ourselv es an imp ortan t additional goal, the con-

v ersion of our sim ulation pac k ages to C++. This will require an in tense eort b oth in the

training of individuals in the tec hnology and tec hniques of ob ject orien ted design and in the

restructuring of the programs to capture the imp ortan t features of the new metho dology .

4.2.4 Collab oratory T o ols

Giv en the fact that the fron tier of hadron collider ph ysics in the next decade will lik ely b e

at CERN, the issue of ho w to facilitate the w ork of U.S. ph ysicists there has b ecome v ery

imp ortan t. Within a group suc h as ours, the need for US/CERN in teractions in a giv en w eek

include participation in the w eekly soft w are meetings, video links with colleagues regarding

c ham b er pro duction topics, and almost daily in teractions of group mem b ers (including studen ts)

stationed at CERN.

This scenario is not a passing one. In just the instance of the Univ ersit y of Mic higan, it

represen ts the needs of a group of a few dozen p eople o v er a p erio d of time measured in

decades. T ra v eling from Ann Arb or to Genev a to attend to pressing issues is not alw a ys an

option, giv en the cost of tra v el, and the cost in absence from one’s primary w ork site. Th us, w e

are en tering an era that is somewhat dieren t from the past for US high energy ph ysicists - one

where the norm for man y decades will ha v e us sp end more and more of our time o v erseas. This

has implications for the future of high energy ph ysics in US researc h univ ersities, and certainly

will ha v e an impact on our graduate and undergraduate instructional programs.

W eha v e long recognized at the Univ ersit y of Mic higan that this dilemma w as approac hing - for

higher education and for American businesses who ha v e an increasing fraction of their activities

o v erseas. Our Sc ho ol of Information has set as one of its priorities R&D on w a ys to impro v e the

11

abilit y of scien tists in so-called "collab oratories" to in teract in a distributed en vironmen t, suc h

as in high energy ph ysics. W eha v e b een w orking with facult y in that Sc ho ol to insure that w ein

high energy ph ysics will ha v e access to the b est a v ailable tec hnology for video conferencing and

for shared applications. Homer Neal is a co-PI on a NSF KDI prop osal designed to sp ecically

lo ok in to these issues, using the CERN/CalT ec h/A TLAS/CMS link ages as a testb ed.

One migh t reasonably question what fron tiers exist for impro v em en ts in this area - after all a

video conference is a video conference. Suc h a view, ho w ev er, is one of the reasons wh y there

has b een so little adv ance in this area o v er the decades. W e b eliev ew e are on the threshold of

ma jor adv ances in this area - taking adv an tage of the massiv e impro v em e n ts in the transatlan tic

net w ork bandwidths, the dev elopmen t o f the virtual video ro oms b y our CalT ec h collab orators,

and the immi nen t promise of Qualit y of Service proto cols. W een vision a da y so on when eac h

of our w orkstations will ha v e a small, inexp ensiv e camera that will p ermit high qualit y video

link ages b et w een colleagues at CERN and Ann Arb or, with a ready assortmen t of shared to ols

on eac h end that will allo w join t image manipulation, le sharing, troublesho oting, and other

activities that are not no w within our reac h. Mo dest in v estme n ts in this area could result in

tremendous pa y os, in terms of p ermitting our studen ts and facult y to optimize the time they

ha v etow ork on the really imp ortan t asp ects of the exp erimen t.

W e exp ect to con tin ue to co op erate closely with these collab orativ e to o l R&D eorts whic h,

though they will b e based in areas outside the Ph ysics Departmen t, hold great hop e for making

our w ork m uc h more ecien t. Of course, dev elopmen ts in this area w ould b e shared broadly

within the comm unit y . Key Univ ersit y of Mic higan researc hers in collab oratory to ols R&D

ha v e already made presen tations to the net w orking comm unit y and to LHC exp erimen t repre-

sen tativ es at CERN.

4.2.5 Remote Access Issues

It has b een long recognized that fast access to curren t data pro duced b y the A TLAS exp erimen t

could b e problematic for U.S. univ ersities and institutes. T ransmission of data at the lev el of

1PByte p er y ear across the A tlan tic, along with soft w are up dates and calibration data, will

place an unpreceden ted demand on net w ork capacit y .

Kno wing of this constrain t, as w ell as b eing a w are of the emerging creation in Ann Arb or of

In ternet I I (UCAID), Homer Neal approac hed the UCAID CEO, Doug v an Hou w eling, ab out

the p ossibilit yof ha ving CERN b ecome a part of the In ternet I I program. Discussions on this

p ossibilit y pro ceeded forth with, follo wing a visit b yv an Hou w eling to CERN in the summer of

1998. It is our understanding that an agreemen t with CERN to join In ternet I I is immi nen t.

This is an imp ortan t rst step not only in pro viding the kind of data access w e will certainly

need, but also in addressing our eorts in collab oratory to ol R&D, an area where UCAID is

also in tensely in terested. Indeed, w e are lo oking forw ard to w orking with UCAID in trac king

adv ances in collab oratory to ols along with its expansion of the transatlan tic bandwidths and

its institution of Qualit y of Service proto cols.

12

4.2.6 High P erformance Computing Resources and Metacomputing

As a result of its participation in the NP A CI partnership, the UoM has in v ested substan tial

resources in b oth the compute-in tensiv e and data-in tensiv e facilities. These facilities are part of

the UoM Cen ter for P arallel Computing, whic h pro vides access to these facilities as w ell as sta

supp ort and consultation for users (at the UoM and nationally for the NP A CI comm uni t y) who

w an t to use these facilities in their researc h. Curren tly , the UoM op erates (1) t w o IBM SP2

computers: a 48 no de system (160 MHZ) with 1 gigab yte of memory p er no de, and an older

48 no de system (66 MHZ) with 256-512 megab ytes p er no de and (2) a mass storage system

with 576 gigab ytes of disk storage and 42 terab ytes of tap e storage (tap e rob ot), using the

ADSM system for storage and retriev al of data and use in metacomputing applications. This

hierarc hical storage system is con trolled b y ADSM and is a v ailable to the UoM and NP A CI

comm unit y .

One of the ma jor goals of the NP A CI alliance is to enable and supp ort large scale metacom-

puting pro jects, using distributed compute and data resources. The real-time computing needs

of A TLAS to pro vide the sim ulation base necessary to supp ort the sp ecied data collection

rates and the sheer size of the databases that are needed to supp ort the A TLAS sim ulations

and data collections, argue for in tegrated high p erformance computing and high p erformance

data resources, capable of w orking together on these complex and c hallenging sim ulations. In

other w ords, a metacomputing en vironmen t, and the UoM is in a unique p osition to dev elop

and test this metacomputing en vironmen t due to its exp ert sta and state of the art facilities

for high p erformance scien tic computation and large-scale data-in tensiv e computing.

T o mak e full use of the UM capabilit y , the net w ork describ ed in 11.5 will b e congured to

connect to the CPC compute/data facilities as necessary to supp ort the sim ulation and data

collection needs to A TLAS. In those cases where signican t computing resources b ey ond those

describ ed in 1.5 are needed to supp ort the computational task at hand, the SP2 system (or its

replacemen t) could pro vide the computational capabilit y . In extreme cases, the resources of

the national NP A CI system (tera ops computing system with p etab yte data collection system

using HPSS) migh t b e deplo y ed, as a natural part of the metacomputing en vironmen t that w e

are dev eloping at Mic higan.

4.2.7 Upgrade of Mic higan’s Computing Resources

Under the T ask H p ortion of this prop osal w e are requesting funds to mo dernize the computing

resources at Mic higan as needed b y the collab orations: CDF, D0 and A TLAS. It is felt w e can

b est meet our computing needs b y joining together in a common eort, thereb y reducing costs

and consolidating managemen t. All three exp erimen ts ha v ec hosen similar computing platforms

for future w ork. This prop osal is based up on the ideas that there should b e no single p oin tof

failure, large n um b ers of inexp ensiv epo w erful mac hines are most cost eectiv e, high-bandwidth

in ter-connections are essen tial, and cen tralized storage is most cost ecien t and manageable.

Our prop osal is to pro vide desktop w orkstations running either LINUX or Windo ws NT for eac h

scien tist. A cen tral NT Serv er will pro vide access via X-windo ws to commonly used soft w are,

suc hasW ord, P o w erP oin t, Pro ject, etc. A cen tral 3 TB disk farm will b e set up with redundan t

13

high-bandwidth access from eac hw orkstation (Fib erc hannel for example). The disk space can

b e logically divided b y exp erimen t, but is ph ysically one system. A cen tralized tap e library

will pro vide bac kup for the disk farm. In addition, eac hw orkstation will optionally ha v e a tap e

device to mo v e data to and from the disk farm area. Gigabit ethernet will pro vide the common

net w ork connections within the ph ysics departmen t and to the In ternet I I. Computing will b e

on a com bination of the desktop w orkstations and a compute farm. The compute farm is to b e

comp osed of screenless LINUX or Windo ws NT computers, with gigabit ethernet connections to

the desktop w orkstations and high-bandwidth data connections to the disk farm. The compute

farm will additionally include a separate serv er for managemen t and bac kup con trol. The last

part of the prop osal is one high-end Unix system (m ultiple CPUs, 1 GB+ of RAM) for use as

a sim ulations platform.

5 Summary

Since joining the A TLAS exp erimen t, the Univ ersit y of Mic higan A TLAS group has gro wn

in b oth n um b er and resp onsibilit y . Curren tly sev en facult y , four researc h scien tists and three

senior mec hanical tec hnicians ha v e b egun to mak e signican t con tributions to a n um ber of

k ey activities for whic hw eha v e sp ecial exp ertise. These include construction of the largest

c ham b ers for the m uon MDT endcap and w ork on the A TLAS m uon and TR T fron t end

electronics. W e are also pursuing a computing and soft w are role that will tak e adv an tage of the

substan tial exp erience of our scien tists and the infrastructure that exists here while assisting

A TLAS in a review of computing and soft w are readiness.

In the next v ey ears Mic higan will b e resp onsible for constructing 104 large m uon end-cap

c ham b ers and installing them in to the A TLAS detector. In the last y ear w eha v e p erformed

R&D on long drift tub es and ha v e made considerable progress on c ham b er pro duction facilities.

W eha v e determined that it is not necessary to use a cen tral wire lo cator for the long tub es,

nor is it necessary to b end these c ham b ers to conform to the wire sag resulting in an enormous

simplication in fabrication. A tub e assem bly and testing facilit y has b een built to construct

the nearly 40,000 tub es that will mak e up the c ham b ers w e are to pro duce. Construction of a

c ham b er assem bly area with a 60,000 lb granite surface table and an en vironmen tall y con trolled

ro om has b egun follo wing an in tensiv e design p erio d. Devices for tub e and c ham b er qualit y

assurance tests ha v e b een constructed and tested in an ticipation of pro duction of mo dule 0 b y

the end of 1999.

Mic higan has tak en on the resp onsibilit y for the pro duction of 10,000 c hannels of protot yp e

m uon MDT fron t-end electronics to b e used for c ham b er testing at the v arious sites. This

has included a review of the existing o v erall design to insure that the lev el 2 trigger is capa-

ble of handling the exp ected data rates. These studies ha v e resulted in mo dications to the

to w er grouping in the forw ard region lev el 2 trigger. As part of the test electronics pro duc-

tion, Mic higan is in the pro cess of completing a mini-D A Q system complete with test xtures,

do cumen tation and training for all the sites.

Mic higan is also assisting in the design of the fron t end electronics for the A TLAS TR T.

W eha v e designed the Dela yLo c k ed Lo op for the DTMR OC TDC ASIC used to read out the

14

TR T.

In the area of computing Mic higan has b egun to fo cus on issues relating to a dev eloping

a database for triggering and ev en t selection as w ell as a database for the m uon c ham b ers

including pro duction, calibration and geometry data. W eha v e also b egun w ork on the m uon

detector sim ulation, reconstruction and p erformance as it relates to the v arious ph ysics issues.

W e are dev eloping plans to upgrade our lo cal computing resources as w ell as explore the use

of collab orativ e to ols suc h as net w orking and remote access including the use of the In ternet I I

pro ject and the Visualization Lab oratory based at the Univ ersit y of Mic higan. Homer Neal is

c hairing the Review of A TLAS Computing committee .

In eac h of these areas, the Mic higan A TLAS group has b egun to mak e imp ortan t con tributions.

W e exp ect that our in uence and impact on A TLAS will con tin ue to gro w in the coming y ears.

Our group remains committed to a strong presence in A TLAS throughout all phases of the

pro ject.

15

References

[1] Rep ort of the A TLAS Computing Review Committee , Homer Neal, c hair,

h ttp://edms.cern.c h/TWDM/cgi/t wdmdo c?pro ject=A TO&action=start

[2] Artamono v, A. et al , \DICE-95 " A TL-SOFT-95-014 , 1995.

[3] \DRAFT A TLAS A TRECON man ual (V ersion 0.015)" A TL-SOFT-94-015 , 1994.

16

Back to top