\documentclass[10pt]{article} \usepackage{epsfig, multicol} %\include{latex} \begin{document} % This is mondo annoying: you have to put the space in or it will % butt it up against the next word in a sentence, but if it is at % the end of a sentence then you don't want the space. %\newcommand{\dz}{D\O\ } \newcommand{\bs}{$\backslash$} % I think these sorts of commands can be used to mess with the pic's % Right now I can only get one pic-per-page %\renewcommand\floatpagefraction{1} %\renewcommand\textfraction{0} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % FEATURES AND COMPONENTS % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{flushright} D\O\ Note 4052 \\ version 1.1 \\ 21 November 2002 \end{flushright} \bigskip \bigskip \bigskip \begin{center} {\LARGE \bf \sc The Calibration Manager} \\ \medskip {\large \sc A Web Interface for Monitoring the D\O\ Pre-Shower Calibration Database} \end{center} \bigskip \begin{center} Jeremy Herr (herrj@umich.edu), Homer Neal (haneal@umich.edu), Eric Myers (myers@vassar.edu), University of Michigan Physics Dept. \end{center} \bigskip \bigskip %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % INTRODUCTION % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \setcounter{section}{0} \section{Introduction} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % PREMISE % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Premise} In the D\O\ experiment at Fermilab, each subdetector provides a means for collecting calibration information. It is useful for physicists concerned with calibration issues to know the value of pedestals, mips and gains of each channel and each run. For the Central Pre-shower subdetector, a database has been designed to store calibration data, which will be described in detail in a separate document. The Calibration Manager is an application designed to provide web-based access to the Central Pre-shower calibration database, and because of its flexibility, may also be used for other subdetectors' calibration databases. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % DESIGN GOALS % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Design Goals} The purpose of the Calibration Manager is to provide a convenient web-based interface for Pre-Shower Calibration Users to access the Calibration Databases at Fermilab D\O. Armed with only an internet connection and a web browser, any interested party may read calibration data and view relevant graphs, and authorized physicists may also use the same interface to manage the database, making modifications and updating it. Some desired features that motivated the design of the Calibration Manager are for users to be able to: \begin{itemize} \item Examine calibration database tables \item Plot distributions of any numeric calibration field \begin{itemize} \item as a function of run id \item as a function of time {\em t} (when {\em t} is available) \item as a frequency distribution of occurrence of field values within given binning \item as a scatterplot of two fields \item classed by a single channel or by specified groups of channels \end{itemize} \item Conduct comparative analysis of current run with previous specified runs, with a template, or with a running average distribution \item Provide Access to a SQL prompt via a web-interface (with and without restrictions, depending on user's database permissions) \end{itemize} \begin{figure}[ht] \centering \epsfig{figure=VDoverview.eps,width=10cm} \caption{``General Design Overview''} \label{VDoverview} \end{figure} In addition to providing graphs of calibration data, the Calibration Manager also serves as a web-based interface to other applications. A separate calibration tool called ``TreeBuilder'' is an application that uses sophisticated logic to find which calibration runs are associated with which data runs. The Calibration Manager can be used to verify that the TreeBuilder logic is functioning properly, run other applications that perform advanced summary functions, and even make alterations to the database. The Calibration Manager logs all modifications to the database, and this logbook may be viewed by authorized users. Thus by calling up the Calibration Manager's web pages in his or her browser, the user can conveniently access all functionality related to the calibration database. {\bf Figure \ref{VDoverview}} shows an overview of the design. Some advantages over existing database client software include: \begin{itemize} \item Availability to anyone with a browser and internet connection \item No need for users to install client software (database client resides on the web server) \item Graphical User Interface is intuitive and self-explanatory \item Central location for all database access needs \item Access to other tools, including ROOT and Python scripts, for performing advanced functions \end{itemize} The Calibration Manager lives on a web server ({\bf Figure \ref{servers}}), which acts as an intermediate step between the end users and the data server at Fermilab. Users request information by filling out web forms and clicking on buttons, and the web server in turn communicates with and fetches data from the database, returning it to the end user in a user-friendly format. The web server can serve multiple users at once, but it acts as a single client when accessing the database. This setup circumvents the issue of licensing for multiple Oracle Clients, since only one Oracle Client is needed, which lives on the web server. \begin{figure}[ht] \centering \epsfig{figure=servers.eps,width=10cm} \caption{``web server / data server relationship''} \label{servers} \end{figure} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % OVERVIEW OF WEB PAGES % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \bigskip \hrule \bigskip \section{Overview of Web Pages} The Calibration Manager is accessed through a collection of dynamically created web pages. In this section we describe the functionality and use of these web pages, and include screen shots of the actual web pages for clarity. \subsection{The Welcome Page} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Screen Shot of Welcome page % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=SSwelcome.eps,width=8cm} \caption{``Welcome Page screen shot''} \label{SSwelcome} \end{figure} The starting point is the welcome page (currently located at \\ \verb+http://circe.physics.lsa.umich.edu+), whose purpose is to present the user with all available functions. These functions have been broken down into three categories: Selection, Browsing and Managing. All users may freely access the first two categories, but an authorized login is required for management functions (these are functions that modify data in the database). {\bf Figure \ref{SSwelcome}} shows a screen shot of the welcome page. Anonymous users (those who have not logged in using their D\O\ database password) will see the words ``anonymous login'' in the upper-left corner, which will be replaced by the user name upon authorized login (click on {\tt SECURE DATABASE LOGIN} to authenticate). The user must have a database account to use any of the manager functions. \begin{itemize} \item {\bf \em DATABASE SELECTION} \\ The user should first select which database (offline development, integration or production) he or she would like to look at, as well as which subdetector (CPS, FPS or CFT), then click on the {\tt SET} button. The page will reload, and the updated choices will be displayed in the upper right corner of the welcome page. Once selected, the subdetector and database settings are passed along from page to page as hidden variables (there is no need to log in or accept any cookies). Whenever the Calibration Manager is used to access a D\O\ database, the user's current selections are assumed. They can be changed at any time by returning to the welcome page and choosing again. \item {\bf \em BROWSING THE DATABASE} \\ The second group of items in the welcome page contains database browsing functions, which are available to all users. These functions are restricted to reading data, so there is no danger of careless or malicious users damaging the contents of the database. The following list of options is expounded upon in greater detail in the next sections of this document. \begin{itemize} \item {\bf FastGraph} \\ View complex ROOT graphs and analyses of pre-defined data sets. \item {\bf Plotting Portal} \\ View several simple graphs at once, side by side. \item {\bf WebSQL} \\ A handy command-line interface for the user who knows SQL. \item {\bf Listing} \\ Get a summary of calibration runs (generated by a Python script). \item {\bf Entity-Relationship (E-R) Diagram} \\ Click here to pop-up a window with an E-R diagram of the current database, which shows a summary of the tables and how they are related to each other. \end{itemize} \item {\bf \em MANAGING THE DATABASE} \\ All of the functions under this heading require authorized login, which currently means that the user must have a D\O\ Oracle Database account to use them. Attempts to use them anonymously will result in failure. See the section on Security below for an explanation of how authentication and authorization are handled. A list of management functions follows, and is given in greater detail in later sections. \begin{itemize} \item {\bf Secure Database Login} \\ Submit your Oracle database password via an encrypted (SSL) link. \item {\bf TreeBuilder} \\ Runs the ``TreeBuilder'' routine from the package {\sc cps\_calibration} to collect together separate calibration runs for pedestals, MIPS, and gains for different parts of the detector to create a complete set of calibration data for all channels. \item {\bf Delete Runs} \\ Runs a Python script for deleting specified calibration runs from the database. \item {\bf Delete Calibrations} \\ Deletes a specified calibration from the database, along with all the ``subsets'' that tie it to a calibration collection. \item {\bf On- to Offline Transfer} \\ Runs the transfer tool script {\tt calDbTransfer.py} from the package {\sc onl\_calDbTransfer} to transfer calibration data from the on-line calibration database to the off-line calibration database. \item {\bf LogBook} \\ View modifications made to the calibration database, sort and search based on various criteria. \item {\bf Submit Graph Query Object (GQO)} \\ Create a new Graph Query Object using a simple HTML form, enabling one-click SQL queries and graphs. \end{itemize} \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % WALKTHROUGH OF EACH COMPONENT % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % FASTGRAPH % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % This page provides a list of pre-defined database queries for commonly % used functions. For example, the user need only enter a desired % channel and click a button to see, the pedestal values sorted by run % number for that particular channel. Using the Graph Query Object % submission page, authorized users can easily create new queries and % graphs to be displayed on the FastGraph page. MISWEB is another % package that can be used to make web-based pre-defined database % queries, but currently without any graphing capability. % %I'm not sure where this section was supposed to go. I am thinking %that I meant to get rid of it but didn't. \subsection{The FastGraph Page} The FastGraph page presents the user with a list of links corresponding to database queries. The user might input qualifying information, such as channel number or a choice between pedestals/mips/gains, then click on the button and get a ROOT graph. The purpose of FastGraph is to make commonly desired database queries and graphs very easy to access, requiring no knowledge of SQL or ROOT. The links that the user sees are not hard-coded into an HTML document, but reconstructed from {\em Graph Query Objects} stored in an XML data file and dynamically output to the screen (this XML file is created using the {\tt dump} function from the {\sc Data-DumpXML} module and is simply a dump of a complicated perl data structure into a string). These Graph Query Objects may be created and edited by authorized users. With this system, there is no need for a developer to create FastGraph queries. When the need arises for a commonly desired graph, any authorized user can create a Graph Query Object using simple HTML forms. This process is described in more detail in the {\bf submit GQO} section. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % ROOT GRAPHS % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{ROOT Graph Page} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Screen Shot of ROOT Graph % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=SSrootgraph.eps,width=10cm} \caption{``ROOT Graph screen shot''} \label{SSrootgraph} \end{figure} The FastGraph page calls the ROOT Graph page, which displays a single large ROOT Graph as a PNG image (shown in {\bf Figure \ref{SSrootgraph}}), accompanied by a PostScript link (for downloading the image in higher quality PostScript format) a Data link (for downloading the raw data from which the graph was generated), a ROOT file (for loading the graph directly into ROOT for further analysis) and the SQL {\tt SELECT} statement used to obtain the data. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Screen Shot of Welcome page % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=VDrootflow.eps,width=12cm} \caption{``Diagram of ROOT Graph control path''} \label{VDrootflow} \end{figure} A flow-chart demonstrating the process of generating a ROOT graph is given in {\bf Figure \ref{VDrootflow}}. ROOT is a free standing application, which requires several pre-defined environment variables to run. Therefore it is necessary to run a batch file in a command shell that sets these environment variables, then calls ROOT. Since command line arguments are passed, this requires that the arguments be fed into the batch file, which in turn passes them to ROOT. Each step alters the command line variables in some way, and this must be taken into account. A preferrable method would be to pre-define ROOT's environment variables so that the batch file were not necessary, and then fork a process and pipe data directly into ROOT. Neither of these are possible on a Windows server, so the elegant (and faster) solution will have to wait for the unix version. Currently ROOT loads one of three C++ scripts that contain the necessary code to generate a run series, frequency distribution or scatter plot. Although not yet implemented, the ROOT field of a Graph Query Object will contain user-defined ROOT functions for creating arbitrary graphs. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % PLOTTING PORTAL % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Plotting Portal} At the Plotting Portal page, the user may choose any number of data sources, to be plotted as run series, frequency distributions or scatter plots. The default number of graphs per page is three, but the user may change this under {\tt NUMBER OF GRAPHS}. To select a data source, the user chooses a table name, field name and can optionally add a SQL constraint. Clicking on {\tt GRAPH IT} creates multiple GD Graphs for side by side comparisons. The inability to perform table joins is this page's only weakness. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Screen Shot of GD Graphs % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=SSgdgraphs.eps,width=12cm} \caption{``Multiple GD-Graph screen shot''} \label{SSgdgraphs} \end{figure} The Plotting Portal Results page displays three (or more) GD Graphs tiled across the screen ({\bf Figure \ref{SSgdgraphs}}). A user could use this to compare, for example, frequency distributions for several different channels. The SQL command used to generate the data is displayed below each graph, and a DATA link is provided for the raw data. There is also a ROOT button to access the same graph with ROOT, although this option has been turned off. The Plotting Portal will eventually be discontinued, since FastGraph is much more powerful and will also have multiple-graph ability in the future. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % WEB SQL % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Web SQL} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Screen Shot of Web SQL % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=SSwebsql.eps,width=12cm} \caption{``Web SQL screen shot''} \label{SSwebsql} \end{figure} Web SQL is a command line interface where the knowledgeable user can submit raw SQL queries to the database ({\bf Figure \ref{SSwebsql}}). Commands are entered into an HTML form textbox, and an {\tt ENTER} button submits the command. The SQL is then sent to the database, and the command and result are displayed, along with a new command box. Previous commands and results are stored on the server, so that the user can go back and edit and re-submit commands. This is more convenient than Oracle's SQL*Plus, which provides a command line environment, but with no ability to edit previous commands. However, SQL*Plus does have an expanded SQL command set, whereas the Calibration Manager is restricted to regular SQL. Web SQL does not check the input for validity or security. It simply passes the SQL command, along with the user's login and password, to the database. If the user is not authorized to perform a certain command it will be rejected by the database and an error message will be displayed. Web SQL simply provides a user-friendly interface to the database. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % LISTING % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Listing} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Screen Shot of CPS Lister % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=SSlisting.eps,width=12cm} \caption{``CPS Lister screen shot''} \label{SSlisting} \end{figure} The {\tt Listing} button on the welcome page calls the {\bf Script Executor} ({\bf Figure \ref{SSlisting}}) an all purpose page that runs external scripts, such as the Python scripts {\sc CpsLister}, {\sc CpsDeleteRun}, {\sc CpsDeleteCalib}, {\sc CpsLoadDefault}, {\sc CpsLoadDummy} and {\sc CpsTreeBuilder}. {\sc CpsLister} requires no special authorization as it simply performs {\tt SELECT} (read-only) commands to summarize Calibration data. The {\sc CpsLister} script is run in a command shell, and the output is simply passed back to the web browser. The Python scripts mentioned above will be described in more detail in a separate document, and are to be found in the {\bf cps\_calibration} package %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % E-R Diagram % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{E-R Diagram} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Image: E-R diagram % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=ERdiagram.eps,width=12cm} \caption{``E-R diagram''} \label{ERdiagram} \end{figure} For the user's convenience, a link to an Entity-Relationship diagram is provided ({\bf Figure \ref{ERdiagram}}). This is a diagram that illustrates the structure of the database, displaying a summary of each table, as well as showing how the tables are related to each other by connecting lines. This will aid users in developing SQL queries and creating Graph Query Objects. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % SECURE LOGIN % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Clicking here causes the server to initiate a SSL (Secure Sockets % Layer) encrypted connection with the user's browser. The effect is % that the user will be presented with a {\em certificate}, which he or % she must accept before continuing. After the certificate is accepted, % a login page is loaded, which is encrypted and not visible to other % internet users. \subsection{Secure Login} Whereas the Database Browsing functions listed above will work for all users (the Calibration Manager accesses the databases using the {\tt d0read} account by default), the Database Management functions require the user to have an authorized database account. The user must provide the web server with his or her password once, which is stored locally on the server until the end of the session. To submit the login information, he or she clicks on the {\tt SECURE DATABASE LOGIN} button. This starts a SSL (Secure Sockets Layer) connection and loads the login page ({\bf login.asp}) where the user enters his or her password and the database he or she would like to use. This (encrypted) form information is then sent to the verification page ({\bf verify.asp}), which sends a simple test query to the database using the given password. If the query succeeds, the user's password is stored locally as a server variable associated with the user's session cookie. Now all database queries this user requests will be submitted to the database with the stored password, until the session expires and all user information is deleted. It should be noted that for a SSL connection to be established, the web server must present the browser with a certificate establishing its identity. Most web businesses pay a fee to a Certificate Authority such as VeriSign, which is trusted by most browsers, so that the user experiences an immediate secure connection, unaware of this acceptance of trust. We have not paid VeriSign for such trust, but have generated our own certificate. This means that upon ``logging in'' for the first time, the user must decide to trust our web server, which is as simple as clicking a button on the pop-up window. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % AUTHORIZATION % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Authorization} Once the user has submitted his or her database password, he or she is ``authenticated'' as that user as long as the session last. This simply means that the Calibration Manager uses his password to query the database. Authorization to perform SQL commands is handled by the Oracle database itself, by means of ``roles''. Each user has ``roles'' determining what they may or may not do. For example, the {\tt d0read} account may read anything, and alter nothing. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Screen Shot of Insufficient Authorization % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=SSinsufficient.eps,width=12cm} \caption{``Authorization screen shot''} \label{SSinsufficient} \end{figure} The screen capture in {\bf Figure \ref{SSinsufficient}} shows what happens when the user {\tt herrj} attempts to delete some calibration runs from the database. He or she is not authorized to do this per the Oracle database's {\sc roles} (which is another topic altogether). The database returns an error message, which the Calibration Manager passes on to the user. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % LOGGING % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Logging} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Visio Diagram of Database Query Logging % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=VDlogging.eps,width=12cm} \caption{``Logging of Database Queries''} \label{VDlogging} \end{figure} Anonymous users are only allowed to view the database, and no record is needed of their many {\tt SELECT} commands. But authorized users can use the Calibration Manager to modify the database, whether this means running the {\bf CpsDeleteRun} script or inserting and deleting records using raw SQL commands in the {\bf Web SQL} environment. Any such modifications should be logged, so that changes may be reconstructed later by interested parties. Web server logs keep track of page hits, but not input information for form variables. For these reasons the Calibration Manager has its own logging capability, recording any activity that might modify the database. Any query (other than {\tt SELECT} queries) that passes through the \verb_DBquery()_ function will be logged, and any time a script such as {\bf CpsDeleteRun} is called, the name of the script and any error messages generated by it will be logged as well. The log files are stored in the directory {\tt D:\bs logs\bs CalibMgr\bs }, and each log file is an XML file that contains one month's records. Other authorized users can view these logs using the Log Viewer ({\tt logbook.asp}), and may search by date, user, keyword and other factors to reconstruct modifications. See {\bf Figure \ref{SSlogview}} for a screen shot of this page. Currently {\sc Perl regular expressions} are used to do searches. For example, the default value for most fields is \verb_.*_, which is a regular expression meaning ``match any and all possibilities''. Not everyone is familiar with regular expressions, so eventually a more user friendly system should be implemented. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Screen Shot of Log Viewer % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=SSlogview.eps,width=12cm} \caption{``Log Viewer screen shot''} \label{SSlogview} \end{figure} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % GRAPH QUERY OBJECTS % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \bigskip \hrule \bigskip \section{Graph Query Objects} As discussed earlier in the FastGraph section, certain types of queries and graphs are desired quite frequently by the users. For example, a popular graph is a simple run series of pedestals/mips/gains (values or errors) for a particular channel in either the FPS or CPS detector. \subsection{Components of a FastGraph Query} The FastGraph page was developed to make access to graphs simple and fast for users. The notion of Graph Query Objects was conceived to enable users to create their own FastGraph queries without intervention of a developer, and with a modicum of effort. {\em Note that in order to submit Graph Query Objects, the user must have a valid Fermilab database account and use it to log in to the Calibration Manager. } The components necessary to construct a meaningful graph and deliver it to the user are {\bf data} via a {\bf SQL query}, a {\bf HTML form} for the user to choose settings, and a {\bf ROOT C++ macro} to produce a graph. These three components (as well as some additional settings) define a Graph Query Object, and are stored in an XML file {\tt GQObjects.xml}. They can be submitted, edited or deleted by the user by means of a HTML form as shown in {\bf Figure \ref{SSsubmitGQO}}. This screen shot shows the three main components mentioned, and farther down the screen the user will find additional settings. It should be noted that user-defined ROOT macros have not yet been implemented. \begin{itemize} \item {\bf SQL query template} \\ To get data, a SQL query is needed. First, the user can utilize {\bf WebSQL} to edit a particular query, and once perfected, this can be associated with a FastGraph link. For example, to get a 2-column list containing run numbers and pedestal values for channel number 1139210 (the database uses an internal ID in which the crate, slot, HDI, SVX and channel are encoded), sorted by run number, he or she would enter: \bigskip % \begin{minipage}{10cm} \verb+SELECT CPS_CALIB_RUNS.RUN_NUMBER, CPS_pedestals.value+ \\ \verb+FROM CPS_pedestals, CPS_CALIB_RUNS+ \\ \verb+WHERE CPS_pedestals.CCR_RUN_ID = CPS_CALIB_RUNS.RUN_ID+ \\ \verb+AND CPS_pedestals.CA_ID = 1139210+ \\ \verb+ORDER BY CPS_CALIB_RUNS.RUN_NUMBER+ % \end{minipage} \bigskip Note that this query would also work if {\tt FPS} replaced {\tt CPS}, and if {\tt pedestals} were instead changed to {\tt mips} or {\tt gains} (and SQL is not case-sensitive). For a FastGraph link, it would be nice to be able to choose between FPS/CPS, pedestals/mips/gains, value/err and also the channel ID. Therefore, instead of simply storing a static SQL command in our Graph Query Object, we store a {\bf SQL template}, which consists of a SQL command with embedded ``variables''. Building on our example, when submitting his Graph Query Object, the user would replace {\tt CPS}, {\tt pedestals}, {\tt value} and {\tt 1139210} with the variables {\em subdet}, {\em PMG}, {\em val} and {\em channel}, so that the values for these variables may be chosen at the time of graphing. The SQL template he or she enters looks like this: \bigskip % \begin{minipage}{10cm} \verb+SELECT {subdet}_CALIB_RUNS.RUN_NUMBER, {subdet}_{PMG}.{val}+ \\ \verb+FROM {subdet}_{PMG}, {subdet}_CALIB_RUNS+ \\ \verb+WHERE {subdet}_{PMG}.CCR_RUN_ID = {subdet}_CALIB_RUNS.RUN_ID+ \\ \verb+AND {subdet}_{PMG}.CA_ID = {channel}+ \\ \verb+AND {constraint}+ \\ \verb+ORDER BY {subdet}_CALIB_RUNS.RUN_NUMBER+ % \end{minipage} \bigskip Note that variable names must be surrounded by curly braces. \item {\bf HTML form} \\ The user is also able to specify the layout of the HTML form that will be used to select between {\tt CPS} and {\tt FPS}, etc. Instead of requiring the user to be familiar with the (simple but always annoying) syntax of HTML forms, it is only necessary for the user to indicate with a couple HTML tags how they want the form layed out. For example, an entry corresponding to our example would be: \bigskip \verb+{val} of {pmg}
+ \\ \verb+on channel {channel}
+ \\ \verb+constraint: {constraint}+ \bigskip Now, on the FastGraph page, the user will see either pulldown menus or textboxes in which they will choose e.g. {\tt value}, {\tt pedestals}, the channel number, and any additional constraints, such as an additional {\tt WHERE} clause. Note, however, that the variable {\em subdet} is missing from the HTML form! This particular variable name is a special reserved variable. On the welcome page, the user chooses which subdetector he or she is interested in, and this setting is automatically inserted into the {\em subdet} part of the SQL template. It should also be noted that {\tt channel} and {\tt multichannel} are special variable names. {\tt channel} may be entered by the user as an internal database ID (e.g. {\tt 1139210}) or in a more user friendly format (e.g. {\tt 0x51/10/6/2/10}). {\tt multichannel} is similar but allows the use of wildcards. This is explained in more detail on the Calibration Manager Help page. \item {\bf ROOT macro} \\ Once the choices have been made and the SQL query has resulted in actual data, this data must be passed to ROOT, which will use a ROOT C++ macro to create a graph. In future versions, the user will be able to enter in a ROOT macro in the same web form (there is already a space for it), but this feature has not yet been implemented. Instead, there are four ROOT macros written by the original developer and currently available for use, all contained in the directory \\ {\tt D:\bs www\bs cgi-bin\bs ROOT\bs }. These macros are described here: \begin{itemize} \item {\bf Run Series } \\ This macro is contained in the file {\tt runseries.C}. It takes as input two columns of data and produces a run series plot where the first column is plotted along the x-axis and the second along the y-axis (the SQL command that gets this data should sort it by the first column with the {\tt ORDER BY} command). \item {\bf Scatter Plot} \\ This macro is contained in the file {\tt scatter.C}. It takes as input two columns of data, just like the Run Series macro, but the data need not be sorted, as a scatter plot consists of points with no connecting lines. \item {\bf Histogram} \\ This macro is contained in the file {\tt freqdist.C}. It takes as input one column of data and plots a frequency distribution histogram of this data. \item {\bf Profile Plot} \\ This macro is contained in the file {\tt profile.C}. It takes as input two columns of data, where there may be many repetitions in the first column. It then uses the {\sc TProfile} object to make a profile plot. For more information on what a profile plot is, consult the ROOT documentation available for download at \verb+http://root.cern.ch/+. \end{itemize} \end{itemize} \begin{figure}[ht] \centering \epsfig{figure=SSsubmitGQO.eps,width=12cm} \caption{``Submit Graph Query Object screen shot''} \label{SSsubmitGQO} \end{figure} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % SERVER % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \bigskip \hrule \bigskip \section{The Server} The Calibration Manager is currently installed on a Dell Optiplex GX240 with 1.5GHz clock speed and 256 MB RAM, and one 20 GB hard disk. So far, the hard disk has two NTFS partitions, a 4 GB partition ({\tt C:}) for the {\sc Windows 2000 Advanced Server} operating system and software, and a 2 GB partition ({\tt D:}) for the Calibration Manager (this is for security reasons, to prevent crackers from ascending the directory tree to access system files). This leaves 14 GB, part of which has been partitioned as Linux {\tt ext2} file systems. It is a dual-boot machine, in anticipation of a future transition to Linux Apache. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % SOFTWARE % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \bigskip \hrule \bigskip \section{The Software} The Calibration Manager consists of a set of {\sc Active Server Pages} (ASP) running with an {\sc Internet Information Service (IIS) 5.0} Server on a {\sc Windows 2000} machine. ASP pages fulfill the same role as CGI scripts, in that they enable the server to generate dynamic web content. ASP allows a developer to embed script code inside an HTML document, as with PHP, with the added convenience that many different scripting languages may be used (whereas PHP requires one to learn a new language). For the Calibration Manager, {\sc PerlScript} has been used as the scripting language, which is identical to the Perl language in all but one or two minor details (for unknown reasons, the {\tt system} command is not available in {\sc PerlScript}, but this can be circumvented by using an ASP extension called {\sc ASPexec}. We have purposely avoided using the PHP-like features of ASP (embedding multiple snippets of code inside an HTML document), which means that all these pages consist entirely of {\sc Perl} code and may be easily converted into {\sc Perl} CGI's (or {\sc mod\_perl}) and run on an Apache server (either on a Windows or Linux machine), if that is desired. Currently the Calibration Manager uses two very different graphing packages, GDGraph and {\sc ROOT}. {\sc GDGraph} is a {\sc Perl} module using the {\sc GD} graphing functions to create basic graphs, and has the advantages of ease of use (simple graphs can be easily designed within the {\sc Perl} environment) and speed (Perl is already loaded and running). {\sc ROOT} is an extremely powerful and flexible set of C++ libraries that can perform many graphing functions as well as data analysis. However, producing even the simplest graph involves writing a nontrivial C++ program, which runs in its own shell. The Calibration Manager accesses {\sc ROOT} by spawning an external command shell with the {\sc ASPexec} ASP extension, and passing data to it with a temporary file. Graphs are created by pre-defined C++ ``scripts'' loaded into ROOT. There are currently 4 C++ scripts written by the original developer for {\bf Run} {\bf Series}, {\bf Scatter}, {\bf Histogram} and {\bf Profile} plots, and a planned feature is for users to create and submit their own C++ scripts for data analysis. ROOT outputs graphs in {\sc PostScript} format, which is unsuitable for easy viewing in a web browser. The {\sc ASPexec} extension is used again to call {\sc GhostScript} and {\sc ImageMagick} to convert the {\sc PostScript} file into the widely supported {\sc PNG} image format, which is displayed in the user's browser. The server on which the Calibration Manager runs has {\sc Oracle Client} software installed, and the ASP pages send SQL commands to the Fermilab database using the ASP interface to Microsoft's {\sc ActiveX Data Objects} (ADO), which communicates with the {\sc Oracle client} and then with the data server at Fermilab. Generally, all data retrieved from the database is stored in a temporary file and kept for a few hours so that the user can keep looking at it. {\bf Figure \ref{VDdatalayers}} demonstrates the layers through which the data travels. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Visio Diagram illustrating Data Layers goes here % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=VDdatalayers.eps,width=12cm} \caption{``Data Layer Diagram''} \label{VDdatalayers} \end{figure} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % SECURITY % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \hrule \bigskip \section{Security} \subsection{Use of Cookies} HTTP servers do not maintain ``state'', that is, the server receives a request for a web page and sends a file back to the user without being aware of the fact that the user is sitting down to a session and requesting pages in a logical sequence. Netscape invented the {\em cookie} to surmount this shortcoming, which is a simple tag that the server and client send back and forth to identify the user and keep track of 'state' information. In its current implementation, the Calibration Manager runs on a Microsoft {\sc IIS} 5.0 web server, which is configured by default to present the user with a ``{\sc session}'' cookie. This cookie has only one {\tt name=value} pair that contains a unique encrypted session number generated for that user. Thus, when a user visits any page on the site, he will receive the session cookie, and after 20 minutes of inactivity, the cookie expires. Microsoft built this feature in for the webmaster's convenience, so he or she would not have to create his own cookie. For anonymous users, the Calibration Manager makes no use of this built-in cookie whatsoever. It ``remembers'' which subdetector and database are selected by passing this information from one page to the next as hidden variables, which are embedded in the HTML document. Hidden variables are fully visible to anyone, insecure and can be tampered with by the user, but we do not pass any sensitive data in this way, so there is no problem. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Visio Diagram illustrating Security Setup % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[ht] \centering \epsfig{figure=VDsecurity.eps,width=12cm} \caption{``Security Overview''} \label{VDsecurity} \end{figure} For authorized users, the Calibration Manager would like to keep track of their database passwords while the session continues, to avoid encrypting all communication with SSL (this slows performance). Transmitting them as hidden variables would be highly insecure, so it now uses the session cookie to identify the user, and stores the password locally on the server. It gets the password in the first place by setting up an encrypted SSL session for the {\bf login.asp} page, then ends the session and uses the cookie to match the user with his password. {\bf Figure \ref{VDsecurity}} demonstrates the link that is secured via SSL for acquiring the user's password. The link between the Calibration Manager's server and the Fermilab database is probably unencrypted, but since the planned physical location for this server is on the Fermilab network, sniffers on the internet will not be able to see any of the data transferred. If the Calibration Manager is ported to unix, it will be able to run locally on a Fermilab server and issues of data security between the Calibration Manager and the database server will be nonexistent. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % MAINTENANCE % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \bigskip \hrule \bigskip \section{Maintenance} There are currently two Perl scripts for performing maintenance on the Calibration Manager: {\bf cleanup.pl} and {\bf backup.pl}, both stored in the {\bf maintenance\bs } directory. \subsection{Cleanup} As is the case with CGI's, ASP applications run over the web and therefore do not naturally preserve state. But the Calibration Manager enables users to make queries and get graphs, keeping track of all their requested data for a short duration. Graphs and data are all stored in temporary files on the server, which without any maintenance would build up quickly. So a separate Perl script {\tt cleanup.pl} has been written to delete any temporary files more than 6 hours old. It simply stats each file in temporary directories to get the time of last modification. This script is run every hour of the day by the server, as a Windows {\em Scheduled Task} (on a linux server the same script would be run as a cron job), and is run from the directory in which it resides. \subsection{Backup} While the code is being developed, it is a good idea to make regular backups. Currently all the Perl and Python scripts fit on one floppy disk, and the Perl script {\bf backup.pl} puts them there. Although {\bf backup.pl} is stored on the server, it is meant to be run from the root directory of a floppy disk. It copies all files (recursively) from the following directories onto the floppy: \\ \begin{itemize} \item {\tt D:\bs www\bs cgi-bin\bs } \item {\tt D:\bs www\bs doc\bs } \item {\tt D:\bs www\bs maintenance\bs } \item {\tt D:\bs www\bs script\bs } \item {\tt D:\bs www\bs secure\bs } \item {\tt D:\bs www\bs style\bs } \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % APPENDIX % % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % SOFTWARE COMPONENTS - INSTALLATION CHECKLIST % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{APPENDIX} \subsection{Software Components to be Installed} The {\tt C:} partition contains the {\sc Windows 2000 Advanced Server} Operating System as well as various applications. Here is a list of these applications, which can also be found in the file {\tt INSTALL-W2K}. This list may be used as an installation checklist, as it contains all components necessary for running the Calibration Manager. Note that for each entry, the second line contains the directory name where the software lives on the Calibration Manager server, the next line (optional) contains a link for downloading the software, and the next line (optional) contains additional notes. \begin{itemize} \item {\bf Windows 2000 Advanced Server, Service Pack 2, and all critical updates} \\ {\tt c:\bs WINNT\bs } \\ remember to check Critical Updates daily \item {\bf ActiveState Perl 5.6.1 Build 631} {\sc free} \\ {\tt c:\bs Perl\bs } \\ {\tt http://www.activestate.com} \item {\bf Perl modules} {\sc free} \\ {\sc GDGraph, GDGraph3D, Win32-ASP, Data-DumpXML, Array-RefElem} \\ (obtain these with {\sc PPM} that comes with ActivePerl, using commands {\tt SEARCH, INSTALL}) \item {\bf Python 2.2} {\sc free} \\ {\tt c:\bs Python22\bs } \\ {\tt http://www.python.org/2.2/} \\ {\sc DCOracle2} has been tested successfully with this release of Python. This is NOT ActiveState Python, which also has a version 2.2, and which does not work with {\sc DCOracle2}. \item {\bf Python package} {\sc free} \\ {\sc DCOracle2} \\ {\tt c:\bs Python22\bs DCOracle2\bs } \\ The Python package for connecting to an Oracle database \item {\bf WinZip} {\sc free} \\ {\tt c:\bs Program Files\bs WinZip\bs } \\ {\tt http://www.winzip.com/} \\ General Windows compression utility \item {\bf AFPL GhostScript 7.04} {\sc free} \\ {\tt c:\bs Program Files\bs gs\bs } \\ \verb+http://www.cs.wisc.edu/~ghost/doc/AFPL/+ \\ Used for converting {\sc PS} $\rightarrow$ {\sc PNG} \item {\bf ImageMagick 5.4.2-3} {\sc free} \\ {\tt c:\bs Program Files\bs ImageMagick\bs } \\ {\tt http://brie.bchem.washington.edu/man/ImageMagick/ImageMagick.html} \\ used for rotating {\sc PNG} images 90 degrees \item {\bf ROOT Production version 3.02/07 with Install Shield} {\sc free} \\ {\tt c:\bs Program Files\bs root\bs } \\ {\tt http://root.cern.ch/} \\ I have never gotten the Install Shield to work for uninstalling {\sc ROOT}. \item {\bf Oracle Client} \\ {\tt c:\bs ORANT\bs } \\ This is an {\sc Oracle} Client written for {\sc NT}. You must do the custom installation and select all files, and install to the default directory {\tt c:\bs ORANT}. Installing to other directories will likely result in failure. There is no un-install option. I tried removing it once, and had to manually edit the registry (and restart several times) to get rid of it. Also, don't forget the crucial file {\tt tnsnames.ora} that goes in {\tt c:\bs ORANT\bs NET80\bs ADMIN\bs }, or you will curse the day of your birth. \item {\bf ASP objects} {\sc free} \\ {\sc ASPexec} \\ {\tt c:\bs Program Files\bs ASPexec\bs } \\ {\tt http://www.serverobjects.com/products.htm\#free} \\ To get this working, you must register the DLL in the directory \\ {\tt c:\bs WINNT\bs system32\bs } with the command {\tt regsvr32 aspexec.dll} \item {\bf Norton Corporate Antivirus} \\ {\tt http://www.physics.lsa.umich.edu/phys-manage/} \\ A virus detection utility obtainable from the Physics Department \item {\bf SFTP} {\sc free} \\ {\tt c:\bs Program Files\bs SSH Secure Shell\bs } \\ {\tt http://www.ssh.com/} \\ a very user-friendly secure FTP client for {\sc Windows}, that supports drag-drop and persistent connections. \item {\bf putty} {\sc free} \\ {\tt c:\bs WINNT\bs system32\bs } \\ {\verb+http://www.chiark.greenend.org.uk/~sgtatham/putty/+} \\ A SSH client for {\sc Windows} \\ (I put it in {\tt c:\bs WINNT\bs system32} so I can go to \\ {\tt Start $\rightarrow$ Run $\rightarrow$ 'putty'}). \item {\bf emacs} {\sc free} \\ {\tt c:\bs Program Files\bs emacs-21.2\bs } \\ The best editor in the world. Especially for {\sc Perl} \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % FILE LIST % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \hrule \bigskip \subsection{File List} The {\tt D:} partition (2 GigaByte NTFS partition) contains the web virtual directory. The base of the web virtual directory, where files get served, is {\tt D:\bs www\bs }. The Calibration Manager consists of {\sc ASP} pages living in this directory, and a list of all the important files can be found below. Directories are listed before files, and otherwise it is in alphabetical order (This is the exact order they are listed in the Windows 2000 File Explorer, when you click on the {\tt Name} tab). \begin{itemize} \item {\tt D:\bs logs\bs CalibMgr\bs } \\ XML logs created by the Calibration Manager to indicate modifications made to the database, as well as timing logs for performance monitoring. \item {\tt D:\bs temp\bs } \\ temp directory used for storing intermediate image files \item {\tt D:\bs www\bs cgi-bin\bs } \\ The bulk of the Calibration Manager lives here. \item {\tt D:\bs www\bs cgi-bin\bs charts\bs } \\ a directory for storing temporary files \\ (cleaned out periodically by {\bf cleanup.pl}). \item {\tt D:\bs www\bs cgi-bin\bs commands\bs } \\ a directory for storing temporary files \\ (cleaned out periodically by {\bf cleanup.pl}). \item {\tt D:\bs www\bs cgi-bin\bs data\bs } \\ These are temporary directories for charts/graphs, Web SQL command histories, and SQL request data \\ (cleaned out periodically by {\bf cleanup.pl}). \item {\tt D:\bs www\bs cgi-bin\bs GQO\bs GQObjects.xml} \\ Directory for storage of {\em Graph Query Objects}. \item {\tt D:\bs www\bs cgi-bin\bs GQO\bs GQObjects.xml} \\ The user-defined {\em Graph Query Objects} are stored in this file. \item {\tt D:\bs www\bs cgi-bin\bs graphs\bs } \\ a directory for storing temporary files \\ (cleaned out periodically by {\bf cleanup.pl}). \item {\tt D:\bs www\bs cgi-bin\bs ROOT\bs freqdist.C},\\ {\tt D:\bs www\bs cgi-bin\bs ROOT\bs profile.C},\\ {\tt D:\bs www\bs cgi-bin\bs ROOT\bs runseries.C},\\ {\tt D:\bs www\bs cgi-bin\bs ROOT\bs scatter.C}\\ C++ ROOT scripts for generating graphs. \item {\tt D:\bs www\bs cgi-bin\bs ROOT\bs runroot.bat} \\ A DOS batch file that sets the necessary environment variables and then runs root with the appropriate C++ script. \item {\tt D:\bs www\bs cgi-bin\bs constants.inc} \\ Perl include file containing many constants, used by all the ASP pages. \item {\tt D:\bs www\bs cgi-bin\bs editGQO.asp} \\ Edit Graph Query Objects (edit, copy, delete). \item {\tt D:\bs www\bs cgi-bin\bs fastgraph.asp} \\ The FastGraph page - ROOT graphs are one click away \item {\tt D:\bs www\bs cgi-bin\bs functions.inc} \\ Perl include file containing commonly used functions for HTML form output, Database queries, and other useful stuff needed by all the ASP pages. \item {\tt D:\bs www\bs cgi-bin\bs graphs.asp} \\ The Plotting Portal Results page (multiple GD Graphs) \item {\tt D:\bs www\bs cgi-bin\bs portal.asp} \\ The Plotting Portal page (select your data sources here) \item {\tt D:\bs www\bs cgi-bin\bs rootscript.asp} \\ The Root Script page that displays ROOT graphs \item {\tt D:\bs www\bs cgi-bin\bs script.asp} \\ Script Executor page. Runs (currently Python) scripts of any kind. \item {\tt D:\bs www\bs cgi-bin\bs specialvars.asp} \\ Help page explaining special Graph Query Object variables. In the future it will be part of a more general help page. \item {\tt D:\bs www\bs cgi-bin\bs submitGQO.asp} \\ Form for submitting user-defined Graph Query Objects \item {\tt D:\bs www\bs cgi-bin\bs websql.asp} \\ Command-line SQL environment (with command history). \item {\tt D:\bs www\bs cgi-bin\bs welcome.asp} \\ The main welcome page \item {\tt D:\bs www\bs D0CM\bs welcome.asp} \\ This is a pointer to the real {\tt welcome.asp} (just in case anyone still has the old URL we handed out at the very first video-conference presentation). \item {\tt D:\bs www\bs doc\bs CalibMgr.pdf} \\ This document. \item {\tt D:\bs www\bs doc\bs CalibMgr.proj} \\ A list of files to be analyzed by {\tt pcad}. \item {\tt D:\bs www\bs doc\bs CalibMgr.project} \\ A subroutine map of the entire Calibration Manager project generated with the utility {\tt pcad} (written by Jeremy Herr). Useful for developers. \item {\tt D:\bs www\bs doc\bs CalibMgr.ps} \\ A PostScript version of {\tt CalibMgr.pdf} \item {\tt D:\bs www\bs doc\bs GQO.txt} \\ a map of the Graph Query Object's Perl data structure (for developers). \item {\tt D:\bs www\bs doc\bs INSTALL-W2K} \\ Installation instructions for Windows 2000 IIS/ASP. \item {\tt D:\bs www\bs doc\bs passed\_data.txt} \\ an abortive attempt at cataloging which hidden variables go where (please don't refer to this; it is unfinished). \item {\tt D:\bs www\bs doc\bs README} \item {\tt D:\bs www\bs images\bs } \\ Fancy picto-buttons and such. \item {\tt D:\bs www\bs maintenance\bs backup.pl} \\ Back up the important files to floppy. \item {\tt D:\bs www\bs maintenance\bs cleanup.pl} \\ Delete old temporary files generated by WebSQL and FastGraph etc. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % From cps_calibration/script README file % These might appear in the calibration manager someday? % % CpsExcludeChannels.py - adds channels to the list of excluded channels % ('parameters') for a given collection % % Cft2Cps.py - copy from CFT on-line calibration database % to CPS off-line calibration database. This is slower % than using calDbTransfer.py but more direct. % % CpsChannelMap.py - fill in the channel address map from a flat file. % (NB: you probably want something else, this was just for testing) % % CftOnlLister.py - list summary information about the CFT on-line % calibration database \item {\tt D:\bs www\bs script\bs CpsDeleteCalib.py} \\ A Python script that deletes calibrations from the database (not yet implemented). \item {\tt D:\bs www\bs script\bs CpsDeleteRun.py} \\ A Python script that deletes calibration runs from the database. \item {\tt D:\bs www\bs script\bs CpsLister.py} \\ A Python script that generates a summary of calibration data in the database. \item {\tt D:\bs www\bs script\bs CpsLoadDefault.py} \\ A Python script that loads default values (zeros for pedestals and mips, ones for gains) into the database (not yet implemented). \item {\tt D:\bs www\bs script\bs CpsLoadDummy.py} \\ A Python script that loads dummy data into the database (not yet implemented). \item {\tt D:\bs www\bs script\bs CpsTreeBuilder.py} \\ A Python script to collect calibration runs (separately for pedestals, mips, or gains) added to the 'bottom' of the database and 'index' them to create 'Calibrations' (not yet implemented). \item {\tt D:\bs www\bs secure\bs } \\ This directory is designated as an SSL directory, which means that all ASP pages in this directory are sent over an encrypted SSL link. \item {\tt D:\bs www\bs secure\bs logbook.asp} \\ Page for viewing the Calibration Manager's database modification logbook. \item {\tt D:\bs www\bs secure\bs login.asp} \\ Page for retrieving the user's database login and password. \item {\tt D:\bs www\bs secure\bs logout.asp} \\ This page destroys session information about the user stored on the server. \item {\tt D:\bs www\bs secure\bs verify.asp} \\ This page sends the user's login and password to the database and logs them in if the query is successful. \item {\tt D:\bs www\bs style\bs D0.css} (that's a zero) \\ The main Cascading Style Sheet file used for the Calibration Manager \item {\tt D:\bs www\bs style\bs indented.css} \\ Some of the pages look better with 5\% left and right margins \item {\tt D:\bs www\bs ADOPS.inc} \\ Include file containing ADO database access constants used by {\tt D:\bs www\bs cgi-bin\bs functions.inc} (sub {\tt DBquery}). You include it using the {\sc SSI} command: \\ \verb++ \\ On a {\sc Windows} server, {\sc SSI} commands are automatically enabled for {\sc ASP} pages. \item {\tt D:\bs www\bs index.html} \\ Redirects the user to the welcome page. \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % CODE SNIPPETS % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \hrule \bigskip \subsection{Code Samples} % \begin{figure}[ht] % \centering % \epsfig{figure=code_sample_rootscript.eps,width=12cm} % \caption{``Code Sample: Calling ROOT''} % \label{code_sample_rootscript} % \end{figure} Several sample excerpts from the code of the Calibration Manager are included below, for the user's perusal. These excerpts are meant to demonstrate some of the Calibration Manager's key functionality, including database queries, the {\sc ROOT} interface, and the general form of an ASP page. \begin{itemize} \item[{\tt DBquery()}] Here is the perl code of the function {\tt DBquery()}, which is defined in the include file {\tt functions.inc}. {\sc WebSQL}, {\sc FastGraph} and {\sc Plotting Portal} all call this function whenever data from the database is needed. {\tt DBquery()} simply opens a database connection using ASP and ADO objects and methods, grabs all the data, and returns it along with any error messages. {\scriptsize \begin{verbatim} sub DBquery { my ($dsn, $username, $password, $hostname, $comment, $mySQLrequest) = @_; my $mySQLresult = ""; my ($rowsptr, $titlesptr); my ($conn, $rst); $conn = $Server->CreateObject("ADODB.Connection"); $rst = $Server->CreateObject("ADODB.Recordset"); $rst->{CacheSize} = $DBCACHE; # force ADO to get many records at once (faster) $OLEDBstring = "Provider=MSDASQL.1;Data Source=$dsn;User Id=$username;Password=$password;"; $conn->Open($OLEDBstring); # open DB connection $rst->Open($mySQLrequest, # get recordset $conn) ; my $numfields = $rst->Fields->{count}; # NOTE: if you do a bad SQL query, so that $numfields is undefined (""), # this loop will cause the inetinfo.exe bug that requires restarting the machine # which is why I check to make sure $numfields is ok if(not $conn->Errors->{Count} and $numfields ne "") { $mySQLresult = ""; &debug(5, "DBquery: no errors detected for SQL request"); # Get the field names into the array @$titlesptr for($fnum=0; $fnum<$numfields; $fnum++) { push(@$titlesptr, $rst->Fields($fnum)->{name}); } # slurp the entire recordset into a 2D array (fast) # NB: idiotically enough, GetRows indexes the array transposed from # what a reasonable person would do, so printing the array out is # a pain. But it's fast and efficient $rowsptr = $rst->GetRows(); } else { $mySQLresult = &getDBerr($conn); } if ($mySQLrequest !~ m!^select!ims) { &make_log_entry($username, $hostname, $mySQLrequest, $mySQLresult, $dsn, $comment); } return ($mySQLresult, $titlesptr, $rowsptr); } \end{verbatim} } \item[{\tt rootscript.asp}] The next code sample is taken from the file {\tt rootscript.asp}, which is responsible for creating {\sc ROOT} graphs. This sample demonstrates the necessary parts of an ASP document. Note that the applications {\sc GhostScript} and {\sc convert} (from ImageMagick) are called in addition to {\sc ROOT} to manipulate the image's format (if {\sc ROOT} could output {\sc GIF} images in batch mode as well as in user mode, these extra time-consuming manipulations would be unnecessary). {\scriptsize \begin{verbatim} # Set ASP language to (Active State) PerlScript <% @LANGUAGE = PerlScript %> # Delimiter to mark beginning of ASP <% # needed for some special data structures use Win32::OLE::Variant; # load in huge list of constants and functions require ("d:\\www\\cgi-bin\\constants.inc"); require ("d:\\www\\cgi-bin\\functions.inc"); # check for hidden variables &die_gracefully() unless &status_ok(); # Get filename from constants.inc $THIS_DOC = $rootgraphs_page; $THIS_TITLE = "ROOT Graph"; # Get session variables from server ($AUTHLEVEL, $LOGGED_IN, $CURR_HOST, $USER, $PASSWORD) = &get_user_info(); # Get current database and subdetector $DB = &get_form_var("DB"); $SUBDET = &get_form_var("SUBDET"); $DSN = $DB; # make the HTML header &initialize_buffer(); &pr("\n"); &pr("\n"); &pr(&set_style("D0.css")); &pr(&page_title($THIS_TITLE)); &pr("\n"); # make top of page &pr("\n"); &pr(&back2home($DB, $SUBDET)); &pr(&display_title($THIS_TITLE)); &pr(&display_user_DB($USER, $DB, $SUBDET, $LOGGED_IN)); &flush_buffer(); # load the Graph Query Objects from the XML file, # and get the ID for the one we want. $GraphObjects = &reconstruct_GraphObjects(); $templateID = &get_form_var("templateID"); # Load the SQL command ($error, $uniqID, $datafile, $SQL, $settings) = &construct_fastgraph_data($GraphObjects-> {$templateID}, $templateID); # Get graph settings $plottype = $GraphObjects->{$templateID}-> {'graphtype'}; $graphtitle = &escape_spaces($GraphObjects-> {$templateID}->{'graphtitle'}); $xtitle = &escape_spaces($GraphObjects->{$templateID}-> {'xtitle'}); $ytitle = &escape_spaces($GraphObjects->{$templateID}-> {'ytitle'}); $xmin = $settings->{'xmin'}; $xmax = $settings->{'xmax'}; $ymin = $settings->{'ymin'}; $ymax = $settings->{'ymax'}; $bintype = $settings->{'bintype'}; $numbins = $settings->{'numbins'}; $binwidth = $settings->{'binwidth'}; ########################################################## # # make a ROOT graph # ########################################################## # Copy data into temp working directory &syscom("copy " . $wwwhome_dir . $CM_dir . $graphing_dir . $data_dir . $datafile . " " . $temp_dir); # Call runroot.bat batch file with graph settings # as arguments. This in turn opens root and passes it # the setttings as well as the name of the ROOT script. # The script tells ROOT to output a PostScript graph, # as well as a .root file for ROOT users. &syscom($wwwhome_dir . $CM_dir . $graphing_dir . $ROOT_dir . "runroot.bat $macrofile \\\"$uniqID\\\" \\\"$graphtitle\\\" \\\"$xtitle\\\" \\\"$ytitle\\\" \\\"$xmin+$xmax+$ymin+$ymax\\\" \\\"$bintype\\\" \\\"$numbins\\\" \\\"$binwidth\\\""); # Copy the new PostScript graph back into the web # serving area. &syscom('copy ' . $temp_dir . $uniqID . '.ps' . ' ' . $wwwhome_dir.$CM_dir.$graphing_dir.$chart_dir); # Use GhostScript to convert the PostScript graph # into a PNG image. &syscom("$GS_app -dBATCH -dNOPAUSE -sDEVICE=png16m -sOutputFile=" . $wwwhome_dir . $CM_dir . $graphing_dir . $chart_dir . "$uniqID.png " . $temp_dir . $uniqID . '.ps'); # Use ImageMagick to rotate the PNG image 90 degrees &syscom($IM_dir."convert -rotate 90 " . $wwwhome_dir . $CM_dir . $graphing_dir . $chart_dir . $uniqID . ".png" . $wwwhome_dir . $CM_dir . $graphing_dir . $chart_dir . "$uniqID.png"); # Copy the .root file from temp working space to # web serving area. &syscom('copy ' . $temp_dir . $uniqID . '.root' . ' ' . $wwwhome_dir . $CM_dir . $graphing_dir . $chart_dir); ######################################################## # HTML FOR DISPLAYING GRAPH/LINKS GOES HERE ######################################################## my $rootfile = "$uniqID.root"; &pr("
"); &pr("

$GraphObjects->{$templateID}-> {'name'}

\n"); &pr(""); &pr("Right-click to download: "); # b2f converts backslashes to forward slashes &pr("[ PostScript ]\n"); &pr("  "); &pr("[ Raw Data File ]\n"); &pr("  "); &pr("[ ROOT file ]\n"); &pr("\n"); &pr("
\n"); # The argument cachebuster is passed to the PNG image link, # which does not use it, but the point is to force the # server not to keep using cached copies of the graph. $nocache = rand; &pr(""); &pr("
The following SQL query was used to generate the ROOT graph:
\n"); &pr("
$SQL
\n"); # Make a button to link back to the welcome page &pr("
\n"); &pr(&back2home_footer($DB, $SUBDET)); &pr("\n"); &pr(""); &flush_buffer(); # Delimiter to end the ASP part of the document. # (The whole file is enclosed in these delimiters) %> \end{verbatim} } \item[{\tt runroot.bat}] The following sample shows the windows batch file needed to open {\sc ROOT}. {\tt rootscript.asp} calls this batch file, instead of calling {\sc ROOT} directly, because several environment variables must be defined before {\sc ROOT} can run. Setting the environment variables in {\sc Windows} is not sufficient. One advantage to running the Calibration Manager on a unix-based system is that environment variables should not be a problem, and also forking is supported, which means that all these ungainly graph settings could be piped directly into {\sc ROOT} instead of these command-line escape-character hi-jinks. {\scriptsize \begin{verbatim} REM These are environment variables needed for root set TEMP=c:\temp set HOME=c:\ set TMPDIR=c:\temp set ROOTSYS=c:\Program Files\root set PATH=%PATH%;%ROOTSYS%\bin REM this batch file takes several arguments, the macro (function) name, REM and an argument (list?). It opens ROOT with the given macro REM and argument(s). REM REM -b means batch mode, i.e. run a macro REM -q means quit after done REM macroname(arguments). In bash, you have to put slashes in front of the REM parentheses, but it doesn't seem to be an issue in Windows REM %1 macro name REM %2 seconds since midnight (uniqID part 1) REM %3 random long int (uniqID part 2) REM %4 Title REM %5 x-axis label REM %6 y-axis label REM IMPORTANT: NO SPACES BETWEEN ARGS HERE! REM (ROOT will truncate everything after the first space) REM Another note: %10 will get evaluated as %1 and then a 0, REM so the number of args needs to be less than 10 root -b -q %1(%2,%3,%4,%5,%6,%7,%8,%9) \end{verbatim} } \item[{\tt freqdist.C}] The following {\sc ROOT} {\sc C++} script source code demonstrates how convoluted drawing a very simple graph can be with {\sc ROOT}. Fortunately, the original developer only had to write this function once, and busy users can obtain this graph from the Calibration Manager with the click of a button. The name of this script is passed to {\sc ROOT} along with other settings, and is loaded on the fly. {\scriptsize \begin{verbatim} #include "TH1.h" #include "TCanvas.h" #include "iostream.h" #include "TGraph.h" int freqdist(const char *uniqID, const char *graphtitle, const char *xtitle, const char *ytitle, const char *window, const char *bintype, const char *numbins, const char *binwidth) { char xmin[40], xmax[40], ymin[40], ymax[40]; // Turn the plusses back into spaces // static char bad_chars[] = "/ ;[]<>&\t"; static char bad_chars[] = "+"; for (char *cp = graphtitle; *(cp += strcspn(cp, bad_chars)); ) *cp = ' '; for (char *cp = xtitle; *(cp += strcspn(cp, bad_chars)); ) *cp = ' '; for (char *cp = ytitle; *(cp += strcspn(cp, bad_chars)); ) *cp = ' '; for (char *cp = window; *(cp += strcspn(cp, bad_chars)); ) *cp = ' '; // Parse the window settings string printf("window = [%s]\n", window); sscanf(window, "%s %s %s %s", xmin, xmax, ymin, ymax); // debugging info printf("graphtitle is %s.\n", graphtitle); printf("xtitle is %s.\n", xtitle); printf("ytitle is %s.\n", ytitle); printf("xmin = %s\n", xmin); printf("xmax = %s\n", xmax); printf("ymin = %s\n", ymin); printf("ymax = %s\n", ymax); // initialize/declare variables ifstream in, out; Double_t value; Int_t numpoints; Double_t xlo, xhi, ylo, yhi; Bool_t first_time = 1; // (first time we go through the loop) // reset ROOT's stack (not the heap, although it is irrelevant here) gROOT->Reset(); // open data file for input. // use the key to name the input file needed char infile[100]; sprintf(infile, "d\:\\temp\\%s\.dat", uniqID); in.open(infile, ios::in); // get number of data points from datafile, // create an array to hold the data in >> numpoints; // In order for TGraph to work, we need to pass it arrays, but // I don't know the size of the array until I read it in from the // file, so I force the computer to do it by pretending n is constant const Int_t n = numpoints+1; Double_t data[n]; printf("numpoints = %d, n = %d\n", numpoints, n); value = 1; // read values from file, putting them into an array for (int i=0; in.good(); i++) { in >> value; // a single column of numbers if(first_time) { first_time = 0; xlo = value; xhi = value; ylo = 0; yhi = 0; } else { if (value < xlo) xlo = value; else if (value > xhi) xhi = value; } printf("i=%d: value=%lf, xlo=%lf, xhi=%lf\n", i, value, xlo, xhi); data[i] = value; } // If "auto" is given, use the min/max of the data for the window if (strcmp(xmin, "auto") != 0 && strcmp(xmin, "") != 0) { xlo = atof(xmin); } if (strcmp(xmax, "auto") != 0 && strcmp(xmax, "") != 0) { xhi = atof(xmax); } // If numbins/binwidth are given, convert string to number Int_t numb = 20; Double_t binw; if (strcmp(numbins, "") != 0) numb = atoi(numbins); else numb = 20; if (strcmp(binwidth, "") != 0) binw = atof(binwidth); else binw = 5; // determine numbins or binwidth if (strcmp(bintype, "numbins") == 0) { numb = numb; } else if (strcmp(bintype, "binwidth") == 0) { numb = (xhi - xlo) / binw; } // create a blank canvas to draw on TCanvas *c1 = new TCanvas("c1", "Calibration Manager -> ROOT Histogram", 0, 0, 100, 70); // create a histogram TH1D *histo = new TH1D("histo", graphtitle, numb, xlo, xhi); // put data points into histogram for(i=0; iFill(data[i]); // no weight needed - just a simple counter } histo->GetXaxis()->SetTitle(xtitle); histo->GetYaxis()->SetTitle(ytitle); histo->GetXaxis()->CenterTitle(); histo->GetYaxis()->CenterTitle(); histo->DrawCopy(); // Why PostScript? Well, ROOT can't do anything else! It sucks. // But we'll fix this with GhostScript and ImageMagick. char outfile[100]; sprintf(outfile, "d\:\\temp\\%s\.ps", uniqID); // gStyle->SetPaperSize(kUSLetter). c1->Print(outfile, "Landscape"); // save to a ROOT file sprintf(outfile, "d\:\\temp\\%s\.root", uniqID); printf("attempting to save histogram in [%s]\n", outfile); TFile *f = new TFile(outfile, "new"); histo->Write(); // To load this histogram into a root session, type the following: // TFile f("filename.root"); // TH1D *h = (TH1D*)f.Get("histo"); // hprof->Draw(); in.close(); } \end{verbatim} } \end{itemize} \end{document}