University of Michigan ATLAS Computing

U NIVERSITY OF M ICHIGAN ATLAS C OMPUT ATION AND S OFTWARE D EVELOPMENT

The University of Michigan ATLAS Group views participation in ATLAS computing activities to be a very important part of its overall research program. A strong computing base is essential to our hardware effort and to attaining our ultimate physics objectives made possible by the experiment. We currently have the critical components necessary to make major contributions to the ATLAS computing effort. First, we are located at a university in the forefront of computing research and that operates state of the art compute-intensive and data intensive facilities. Second, we have several qualified individuals who are prepared for long-term, high level contributions. The computing activities already completed or underway for ATLAS include:

Because of the broad spectrum of mutually supporting computational talents and interests present within UM ATLAS, we intend to create, in our Task A group, a center for computation and software development. We expect that this structure will help us prioritize our various ATLAS computing projects and to permit us to benefit from discussions of key issues in a group setting. A coordinated effort will enhance our ability to work with other software centers within the collaboration, share the results of our work, and provide an economical approach to establishing and maintaining a local repository of current software.

Key to our interest in ATLAS software is, of course, our desire to exploit the physics from the experiment. A successful effort in doing this will require members of the group with detailed knowledge of ATLAS physics simulation studies, with the OO simulation and reconstruction packages, and with the OO (or OR) databases. Moreover, our successful muon hardware production tasks, and the design of an efficient trigger/DAQ OO (or OR) database, will require many of the same talents. Fortunately, these talents do exist within our group. In addition, a program is envisioned where a growing number of our group members will become expert with the "new" OO/C++ ATLAS software in the coming year.

Bing Zhou has had a long involvement with ATLAS simulation studies and that experience, along with her leadership role in the muon chamber construction project, will provide an excellent foundation for the proposed computational studies. She will draw upon the support provided by individuals in the group such as Daniel Levin, Steve Goldfarb and Shawn McKee. In the area of databases, we expect the active involvement of Homer Neal, who developed the initial database for the DZERO central calorimeter. He, along with Levin, Goldfarb, and McKee, are evaluating OO database schema for the muon production project. In addition, Myron Campbell and Jay Chapman, Neal and other members of our group who have begun to evaluate approaches for the ATLAS trigger/DAQ database. Furthermore, we expect to deploy our simulation skills to regularly evaluate production issues that affect the physics performance of the muon chambers.

It is our goal to make substantial additional contributions to the experiment within the domain of software and computing, capitalizing on the computing expertise at Michigan both within the ATLAS group and within the greater university community. In the section below we first survey the computing resources at Michigan, in terms of infrastructure, hardware and personnel. In the second section we outline Michigan's future plans. Finally we summarize our proposed budget and the emphasis of this proposal.

The University is located in Ann Arbor where the Internet II project is based. The project is a spin-off of University activity. Given the importance of high bandwidth connectivity to HEP, we have already discussed the need for Physics to be the first university department connected with full bandwidth to Internet II.
The School of Information has engaged CERN in dialog and has a program in place to examine the nature and needs of the ATLAS experiment for collaborative tools for distributed research activity.
The University through its Visualization Laboratory and Media Union has programs that offer major contributions to the handling and viewing of data.
Bill Martin (a faculty member in the UM department of Nuclear Engineering who has committed to join our ATLAS effort) is the project director on a major grant that provides facilities and infrastructure (human resources and physical facilities) for high performance computing, including compute-intensive and data-intensive computing. This grant is funded through the NSF Partnerships for Advanced Computational Infrastructure (PACI) program, which has as its express mission the development and deployment of teraflops computing facilities and petabyte data facilities for the US. The UM is both a compute-intensive and data-intensive institution in this partnership (along with UCSD, Caltech, Berkeley, and the UT-Austin) and can offer access to these facilities and capabilities for the UM participants in this project.

Members of the ATLAS group with plans to engage in computing include: Bing Zhou, Daniel Levin, Steve Goldfarb, Shawn McKee, Homer Neal, J. Chapman, Robert Ball and Bill Martin. We expect this group to be augmented with students and computer specialists as projects become better defined. This proposal is a first step in developing the necessary computing environment for ATLAS.

Robert Ball has been working most recently on data acquisition software and hardware for testing the MDT chambers. He has a long-standing expertise in data acquisition software dating back to some of the earliest days of FNAL. He was a part of the SDC collaboration team at the SSC that produced the UNIDAQ suite of portable, UNIX-based DAQ software. For the L3 experiment he assembled and operated a Fermilab ACP/R3000 parallel processing farm which was the largest, non-CERN computing resource for the L3 experiment over a period of several years. More recently he took the lead in ensuring that a modern network backbone was installed in the physics building complex at the University of Michigan and is still called upon to help trouble shoot difficulties when they arise. As a staff member of the HEP electronics shop at Michigan he administers or oversees the administration of the HEP UNIX clusters. With his background he could play a natural role in the Michigan efforts in ATLAS data acquisition, in the establishment and operation of the ATLAS/CDF/DO computing cluster, and in collaboratory work with the UM School of Information over the period of this grant.

J. Chapman has traditionally worked in trigger hardware and software development but has plans to work in the area of database definition specifically with respect to trigger and data selection. In large hadron collider experiments the hardware and software trigger criteria are central to the understanding of the data samples and to the physics that can be extracted from those samples. It is time that modern database technology be applied to this very central problem of defining, relating, and tracking the selection of events.

Steven Goldfarb has been contributing over the past several months to modeling of the muon spectrometer geometry for the ATLAS Detector description database. This involves producing a complete description of the various components of the spectrometer with generic C++ objects. The description will be placed into Objectivity/DB to be used by simulation and reconstruction software both for general purposes and for test beams. This work, being carried out in close coordination with the ATLAS Database Group, represents a key contribution to the general ATLAS database architecture and is on the critical path to nearly all ATLAS muon software development efforts. In addition to placing the University of Michigan in a key position to participate in high-level software architectural decisions, this early experience will be invaluable for application to future muon analysis software, as well as detector production and calibration database efforts. Steve has very recently been appointed as the database coordinator for the entire ATLAS muon subsyste m.

Daniel Levin has participated in the MACRO collaboration and more recently in short baseline neutrino oscillation efforts (COSMOS and TOSCA). He has had significant experience in hardware implementation and detector simulation studies. In MACRO he participated in multiple muon analysis and conducted a pioneering effort to detect in coincidence surface air Cerenkov light and deep underground muons. In COSMOS he conducted beam tests and analysis of calorimeters and performed a comprehensive simulation of COSMOS triggering. More recently he has joined ATLAS where he has began a major review of muon chamber performance and the impact on momentum resolution for drift tubes whose sense wires are displaced from their axes. This outcome of this work indicates that the difficult and tedious tube bending procedures are not necessary to maintain good momentum resolution. This work has included extensive modeling of electron drift properties using GARFIELD, detailed muon system GEANT based simulation and track reconstruction. It is anticipated that this work will evolve into other software and pattern recognition efforts such as Level II trigger algorithm development and trigger database development. Dan will also assume responsibility for final quality assurance and commissioning of monitored drift chambers.

Shawn McKee has been setting up the Michigan ATLAS online software for production control and measurement. As a future extension of this system, he is evaluating the CRISTAL software² (created for CMS) as a possible tool for ATLAS. This software manages workflow and production data during the detector construction and tracks critical parameters of detector components. In parallel, he is working to use Microsoft ACCESS as a local production database. Shawn intends to concentrate on ATLAS muon detector simulation studies using GEANT4 and has an extensive background doing such work. On the GEM experiment at the SSC, he chaired the central tracking simulation committee and co-authored a GEM technical note on Z' detection³ . More recently he has worked on simulating ultra-high energy (PeV) cosmic rays in thin hadronic calorimeters ⁴ and on data analysis from many balloon borne cosmic-ray experiments ⁵ . In addition he has been active in both the COSMOS and TOSCA, short-baseline neutrino experiments. These experiences should prove valuable for ATLAS-wide combined physics analysis and simulation work.

Homer Neal developed the first production database for the DZERO central calorimeter using a DEC RdB relational database in a distributed environment. Shape profiles for each uranium plate, readout plate and other media were tracked by this database which was then used to determine the initial sampling fractions for the entire Liquid Argon Central Calorimeter. He has also developed detector performance and run quality database packages using third-generation query applications. With the completion of the ATLAS-wide computing review which he chaired, it is his intent to concentrate on UM ATLAS muon production database issues, and to work on overall ATLAS OO database architecture topics. In addition, Homer Neal has been active in bringing the needs of US universities for high-speed links to CERN to the attention of both CERN Management and to the leadership of the Ann Arbor based INTERNET II. CERN is now poised to become the first international member of INTERNET II, and it has already committed to an arrangement where it will have a direct connection into VBNS in Chicago. These steps will represent an enormous increase in bandwidth available to s upport LHC research by US universities.

Homer intends to continue to nurture the CERN/INTERNET II partnership, and to use the new technologies that derive from this partnership to enhance collaboratory tools used for high-energy physics research. At one level, such tools can promote the joint analysis of data by colleagues at CERN and Ann Arbor. One example already being examined is the WIRED tool developed at CERN, and now licensed for use at Michigan, that allows the 3-D rotation of events being simultaneously viewed by scientists at CERN and at Ann Arbor, with shared joystick control, in a videolink setting. At another level, R&D efforts are planned to continually improve the quality of videoconferencing as network bandwidths and quality of service protocols become available. He and Stephen Goldfarb will continue to do this work in collaboration with the University of Michigan School of Information, and with Harvey Newman (CMS/Cal Tech). Some aspects of this work are of considerable importance to high energy physics as we go about the tasks of coordinating detector construction issues within a dispersed environment, of reducing travel expenditures, and of keeping important links between faculty and students as they are increasingly separated by the demands of on-campus presence and off-campus research.

Bing Zhou has carried out many important simulation studies related to detector design and optimization (in particular, for the central tracker and muon system design studies for GEM at the SSC and for the ATLAS muon spectrometer drift tube layer configuration at the LHC). In 1997 Bing coordinated the simulation effort of the US ATLAS muon team in studies of spectrometer performance for the ATLAS Muon Technical Design Report (TDR). The studies were carried out using a variety of physics benchmarking processes, including single-muon, di-muon, and four-muon final states, as well as the unavoidable background processes. Using the performance features obtained from the full detector simulations of the muon spectrometer, Bing performed searches for SM Higgs bosons and MSSM Higgs bosons, as well as searches for new gauge bosons (Z' and W') with muon final states. The results illustrate the great discovery potential of the ATLAS muon system⁶.

In providing a response to the LHCC review committee's questions regarding the Muon TDR, Bing performed very intensive studies of muon track reconstruction for measurements in the non-gaussian tails. This study quantitatively addressed the issues relating to Higgs detection sensitivity through the four-muon final state, and the probability of charge mis-identification for high mass new gauge bosons (Z' and W')⁷ . In 1998, Bing led another important study on the ATLAS MDT neutron response sensitivity. The study was carried out by measuring the MDT neutron sensitivity and by developing a sophisticated simulation model to emulate the measured data. This model provided crucial input to the muon Level I trigger design⁸ . In collaboration with the BMC physicists, Bing contributed significantly to the muon Level 2 trigger algorithm development and code implementation ⁹ .

We present the details of our computing plans below, even those outside the scope of this proposal, to give a context for the muon subsystem software development work we are proposing for in the concluding budget description section. We would like to emphasize that the work described below will utilize the ATLAS-wide OO/GEANT4 computing environment.

1) Muon Detector Simulation and Physics Studies with Combined Detector Performance

As described in this proposal, members of the Michigan group were heavily involved in simulation work for the ATLAS Muon Technical Design Report. The work included extensive studies of the muon detector design, performance, and construction. We feel that continued effort on muon detector simulations, reconstruction and performance analysis will be needed to better understand the detector response and that this additional work should be undertaken while the detector is being built. Experience gained from the ATLAS physics studies, has convinced us that we must devote much more effort toward muon reconstruction when combined with information from other detector subsystems, i.e. , reconstruction of the muon spatial trajectory and momentum jointly with measurements from the inner tracker. This collective fit must include corrections for the energy loss in the calorimeters. A clear understanding of the full reconstruction capability of ATLAS is essential if we are to maximize discovery sensitivity and potential of the detector.

For detector simulation and reconstruction we will first focus on modeling the detector specifications and response parameters. The details of these specifications and parameters are crucial in simulating the detector performance. Examples include:

The wire sag due to gravity and electrostatic force must be included in the software code, since each wire serves as a tracking coordinate.
Corrections for the drift tube space-time function asymmetry (due to wire sag) should, in principle, be implemented in the reconstruction program.
A temperature, pressure, and HV database containing the parameters for the space-time function must be implemented.
A T-zero calibration) database and code (autocalibration) to make appropriate use of it should be implemented.

The above mentioned reconstruction effort is complementary to the database development program anticipated at Michigan, since success of the reconstruction effort will depend on access to the related database. We also want to emphasize that a determination of simulation parameters for modeling basic performance data represents a unique contribution to the ATLAS muon software package. So far, no significant effort has been directed to this important need.

For combined performance physics studies, we will use the most promising Higgs discovery channels with muon final states to develop the programs for final muon momentum reconstruction. It is very clear to us that a realistic full detector performance combination is important to truly understand the physics reach at the LHC with the ATLAS experiment. We also note that great effort will be required to produce a fully debugged and tested software package in time for the LHC's turn-on. Michigan physicists are expected to be vital contributors to the effort. Indeed, we have already started testing and developing the combined performance software in Ann Arbor in collaboration with Boston University physicists.

In our development process we have set for ourselves an important additional goal, the conversion of our simulation packages to C++. This will require an intense effort both in the training of individuals in the technology and techniques of object oriented design and in the restructuring of the programs to capture the important features of the new methodology.

The work on geometry databases represents a logical activity for a group that is involved in the design and fabrication of a significant detector segment. It includes tracking the production sequence, deciding on what elements of the production data are to be archived, exported, and inserted into the overall geometry database, developing the tools to query the database for physics analysis, and then doing the analysis.

During the onset of the detector construction, our group, intends to implement a robust database for storage of the production parameters for the forward muon spectrometer. The operating conditions and geometry for all chambers and their sensitivities to changes in environmental conditions will need to be recorded. Tests will begin immediately on various relational and OO database options. We will work closely with CERN and US ATLAS muon colleagues in the selection of the final product. Discussions are already underway with the author of the CRISTAL database about the suitability of that software for our needs. We have already launched an evaluation of CRISTAL’s suitability for our needs. In the meantime we have prepared the standard ACCESS database required to insure the smooth integration of our present work with the of the other chamber construction sites.

We think that modern database technology needs to be globally applied to the problem of defining and monitoring the hardware and software process referred to as triggering. To precisely define the efficiency of particular physics channels requires one to know the detailed path and selections made for all trigger routes through which a particular event topology has passed. This includes the hardware logic that accepts events at the earliest point in the path (much of which is Field Programmable Gate Array, FPGA, coded in today’ ;s detectors), the programmable parameters downloaded into the hardware, the algorithms coded into the level 2 processors, all parameters used in the event acceptance, and similar algorithms and parameters used in the offline processing. The totalities of these accept or reject steps must be understood for each of the detector subsystems. An additional complication derives from the time evolution of these criteria as the detector and software mature. The magnitude of this problem suggests a global approach to storing and accessing this information. Object oriented (or object relational, OR) databases most naturally meet these needs since the entities that must be specified to fully define the event path range from downloaded binary numbers to algorithms in code. As an adjunct activity to the investigation of object oriented (or object relational) database technology, we plan to examine the options for defining the full range of objects needed to specify a trigger path. If the initial investigation is promising, the next step will be to begin an information collection effort, polling all detector groups for the characteristics of their trigger and selection code. The goal will be to provide the collaboration with a mechanism to store and retrieve each and every object that controls the event filtering process from hardware trigger to offline selection cuts. When structured into such a database the process can, hopefully, be a transparent tree whose apex is a physics category and whose branch-ends are parameters downloaded into hardware or used in selection and processing. This common database would be accessed for runtime initialization, event processing, and Monte Carlo calculations.

Given the fact that the frontier of hadron collider physics in the next decade will likely be at CERN, the issue of how to facilitate the work of U.S. physicists there has become very important. Within a group such as ours, the need for US/CERN interactions in a given week include participation in the weekly software meetings, video links with colleagues regarding chamber production topics, and almost daily interactions of group members (including students) stationed at CERN.

We have long recognized at the University of Michigan that this dilemma was approaching - for higher education and for American businesses who have an increasing fraction of their activities overseas. Our School of Information has set as one of its priorities R&D on ways to improve the ability of scientists in so-called "collaboratories" to interact in a distributed environment, such as in high energy physics. We have been working with faculty in that School to insure that we in high-energy physics will have access to the best available technology for videoconferencing and for shared applications. Homer Neal is a co-PI on a NSF KDI proposal designed to specifically look into these issues, using the CERN/CalTech/ATLAS/CMS linkages as a testbed.

We expect to continue to cooperate closely with these collaborative tool R& D efforts which, though they will be based in areas outside the Physics Department, hold great hope for making our work much more efficient. Of course, developments in this area would be shared broadly within the community. Key University of Michigan researchers in collaboratory tools R&D have already made presentations to the networking community and to LHC experiment representatives at CERN.

A pilot project currently underway by our group at CERN is the web based archiving of the CERN summer lecture series, using a proprietery package developed by a University of Michigan staff member. With this system the video and audio from a lecturer is presented on the web along with the synchronized POWERPOINT slides used by the lecturer. These talks can be viewed at any time by anyone in the world who possesses a web browser. CERN training managers are watching the project with great interest for its possible relevance to LHC software training programs.

It has been long recognized that fast access to current data produced by the ATLAS experiment could be problematic for U.S. universities and institutes. Transmission of data at the level of 1PByte per year across the Atlantic, along with software updates and calibration data, will place an unprecedented demand on network capacity.

Knowing of this constraint, as well as being aware of the emerging creation in Ann Arbor of Internet II (UCAID), Homer Neal approached the UCAID CEO, Doug van Houweling, about the possibility of having CERN become a part of the Internet II program. Discussions on this possibility proceeded forthwith, following a visit by van Houweling to CERN in the summer of 1998. It is our understanding that an agreement with CERN to join Internet II is imminent. This is an important first step not only in providing the kind of data access we will certainly need, but also in addressing our efforts in collaboratory tool R& D, an area where UCAID is also intensely interested. Indeed, we are looking forward to working with UCAID in tracking advances in collaboratory tools along with its expansion of the transatlantic bandwidths and its institution of Quality of Service protocols.

The scope of the proposed computing work we have described is large and we have a correspondingly large and experienced group of physicists with the interest and talent to carry out this work. We are not requesting any funds to support physicists in this proposal. Rather, we only request US-ATLAS computing funds to support our muon subsystem software development activities. The level of support we are asking for is to establish our initial ATLAS computing environment to successfully carry out the muon subsystem tasks described above.

Our groups’ experience with running GEANT simulations, coupled with estimates of the required number of simulated events for reasonable statistics, lead us to conclude we will require approximately 10 “fast” CPUs (500 Mhz PIII or equivalent) dedicated to simulation production. We intend to assemble a LINUX CPU farm, by purchasing 10 inexpensive computers, each with at least 20 gigabytes (GB) of disk space and 512 megabytes (MB) of fast ram. Since these will be compute servers, we will not normally require monitors, keyboards or mice for each machine, but instead will use KVM (keyboard, video and mouse) switches to connect to each machines console when necessary. GEANT simulations require a good graphics environment, so we will purchase three 21”, multi-synch monitors, thereby creating three “simulation” stations for software development. These ten compute servers will be tied into our local network via a 12 port, 10/100 Mbps manageable switch. The last critical piece of hardware is a backup device capable of quickly storing all the code and data we generate as well as backing up our development environment. Current DDS-3 4mm tape loaders can backup 192 GB of data on one 8 tape cartridge overnight, which is sufficient for our anticipated data load.

It is absolutely crucial to have a computer specialist within the group full time to set up, maintain and upgrade our computers, as well as install and upgrade the global ATLAS computing software. We request support for such a person whose primary responsibilities will include:

The last part of the budget requests funds to purchase any required software and supplies. While our chosen operating system (LINUX) is free, there are many components we will need to purchase: compilers, graphics software, backup software, databases, etc. For example, the current LHC++ software environment requires some commercial licenses. In addition, many of the possible database engines for our production databases would need to be purchased. The main supplies we would need are backup media to store ~1 Terabyte of data. Currently this would correspond to 40 DDS3 4mm tapes, assuming ~2:1 compression.