UMHE-02-10
UMHE-http://atlas.physics.lsa.umich.edu/docushare

UMHE-02-10

UMHE-http://atlas.physics.lsa.umich.edu/ docushare

CSM-1 & CSM

Design Manual

Design of the Chamber Service Modu le & Prototype 1 (Rev D)

University of Michigan

October 9, 2003

J. Chapman, P. Binchi, R. Ball , T. Dai, & J. Gregory

CSM-1 Design

Development at UoM

O VERVIEW OF P ROGRAM

The CSM-0 (a prototype CSM) is docume nted in specifications

on the University of

Michigan ATLAS web server Docushare. It was developed as a first step toward the

design of a production muon front-end r eadout multiplexer, initialization controller, tim-

ing distributor, and calibration contro ller. Documents on the DocuShare server are

linked via ATLAS Electronics > MDT Electronics and are called “CSM-0 Development

and Users Manual” and “CSM-0 Design Internals Document”. The general characteris-

tic of the design has not changed and is specified in numerous ATLAS muon publica-

tions as the near chamber part of the NIMROD

. The work to design the CSM began

with simulations of the datafl ow and were reported in the LEB99 meeting in Snow-

mass

and are available on the Docushare serv er. The simulations were performed in

VerilogHDL. A similar VerilogHDL descrip tion of the AMT-3

will when a similar

description of the ROD

is complete, provide a tool for continu ously evaluating the

design as it matures. The prototype CSM-0 synthesized from the original VerilogHDL

code will serve as an active t est fixture for the design. This prototype has been built as a

6U VME module containing the CSM unit, a JTAG interface, an LHC clock emulation,

and a VME readable output FIFO. Since the timing and control functions of the LHC,

the TTCrx

chip and its associated driver componen ts, were not available in the test

1. http://atlas.physics.lsa.umich.edu/do cushare/default.htm

2. http://u maxp1.physics.lsa.umich.edu/~chapman/atlas/nimrod.ps

3. Proceedings of the LEB99 conference a t Snowmass (1999)

4. http://a tlas.kek.jp/%7earaiy/amt1/index.html

5. http://www.nikhef.nl/pub/department s/et/atlas)mdt/index.html

6. http://www.cern.ch/Atlas/GROUPS/FRONTEND/Ttc1 .htm

Overview of Program

CSM-1 Design Development at UoM 1

environments, these functions are designed into a separate Xilinx called the TTCem cir-

cuit. This additional Xilinx is requ ired to accept the external trigger, to provide the

EVID and BCID, and to synchronize the tr igger with the free running clock that gener-

ates the simulated LHC crossing interv als.

The differences between the CSM-0 and the final CSM unit mounted on chambers are

significant but the two units have a on e to one correspondence. The final on-chamber

version will be simpler in that it will not need to perform event building. Since the ROD

must form events from 6 CSM output str eams, there is no need to do this job twice. The

on-chamber CSM will therefore be ex pected to do simple time-division multiplexing of

data from the 18 TDCs. With this scheme the TDC from which any particular unit of

data originates can be determined from the word position in the time sequence. If no

data is available from a specific TDC, an idle position-holder will need to be sent. The

full event building version of the CS M-0 can be thought of as a CSM/ROD pair that

handles only one chamber in contrast to the final system which will incorporate six sim-

ple time multiplexing CSMs feeding a si x input ROD module (called an MROD) where

the event building will take place. Thus , the on-chamber CSM will not have a trigger ID

FIFO, a deep input FIFO, an output data FIFO, or a word count FIFO. It will have an

optical output encoder and driver to sen d 32-bit words to the MROD module which

accepts its output and the output from 5 other CSM modules.

The ov erall flow of data from the chamber wires up to the modules in the ROD crate i s

represented in Figure 1 on page 1. The simulation performed for the CSM/ROD con-

tained the 18 TDC inputs each with their own serial data and clock. It described the out-

put flow of data along the channel des tined for the ROD. Data units were collected from

the TDCs by the CSM and transmitted in turn to the ROD modules. Data is transmitted

from the CSM as 32-bit units seriali zed and sent on a single fiber. The 32-bit data units

are either TDC headers, trailers, time digitizations, or various control words. The pri-

mary task of the prototype CSM, cal led the CSM-1, is time-division multiplexing. Data

from each TDC is sent as requested by the level 1 trigger when the front-end link from

the individual TDC becomes free, i.e. , when data from all previous tri ggers are sent.

Individual ev ents are separated by header and trailer words. These header and trailer

words are selectively en abled for transmission in the TDC. At least trailer words are

required to indicate the end of event. T he simulation assumes that both headers and

trailers are sent for redundancy.

FIGURE 1. The Position of the CMS in the muon data flow.

The time -division CSM collects data from each of 18 individual TDCs into individual

FIFOs, polls for data availa ble in these 18 FIFOs, and if present simply forwards the

TDC data to the ROD module in the time s lot specified for that TDC. The time position

CSM

ROD

Mezzanine Card Chamber Service Module Re adOut Driver

Containing ROD Car ds

18/CSM 6/ROD

ASD/TDC

In Faraday cage 1 per chamber 1 ROD Card/TgrTower

Design Specifications

2 CSM-1 Design Development at UoM

in the output stream defines the source of the data. As a means to guarantee that the time

positions are synchronized between the s ending CSM and the receiving ROD module, a

spacer word is sent as the 19th data uni t each cycle. To insure synchronization of the

sending and receiving clock, 2 optical link idle words (not to be confused with the idle

word used to flag “no data” from an in dividual TDC) are sent following the spacer

word. The basic flow of information is illustrated in Figure 2 on page 2.

FIGURE 2. Serial data flow and event transmission by the CSM

D ESIGN S PECIFICATIONS

Module Subsections

The CSM will have six subsections. Figur e 3 on page 4 illustrates these blocks, the

JTAG initialization, the trigger timing and control, the optical transmitter, the serial to

parallel receivers, the multiplexer, an d the environment monitor. The list below further

defines the content of these blocks.

1. The JTAG initialization block communicates to an external controller. This control-

ler can be any unit which adheres to the JTAG standard. Opto-isolation circuits pro-

vide for local ground separation from th e ground of the external JTAG controller.

Thus the external controller must supply its own power and ground to the CSM unit.

2. A second subsection contains th e TTCrx chip and its optical receiver. It contains the

reset functions, the EVID and BCID regis ters, the phase adjustments for the 40MHz

clock, the decoding of the LVL1 trigger, and the calibration functions. The CSM

makes standard use of the TTC system and requires no special interconnections.

TDC

10 9

910

CSM

HeaderDataTrailer

Spacer Word

Spacer Word

18 TDCs maximum

13 14 15 16117 23456

Time Multiplexed TDC Data Words

Idle Words

Design Specifications

CSM-1 Design Development at UoM 3

Since the CSM is configured as a simple time division multiplexer, it does not use

the BCID and EVID registers.

3. The optical transmitter is based on the CERN-designed radiation-hard GOL chip and

an Infineon optical transceiver. This un it can accept 32-bit words at up to 40Mhz for

transmission along an optical fiber. Sin ce we need only send 32-bit words at 25MHz

to remain ahead of the maximum ra te at which data can be received from all TDCs

(along with the spacer/idle words) , we will use the transmitter at 25Mhz. At the

MROD the 32-bit words are collected, the spacer word checked, the TDC empty

words removed, and the actual TDC data s tored. The optical transmission idle words

do not appear at the receiver output. Th ey serve only as synchronization characters.

4. Input from the TDCs arrives as a serial bit stream that is assembled into 32-bit words

under the control of a sequence that s eeks a start bit, assembles 32 data bits, tests a

parity bit, and outputs the 32-bit data word with an accompanying data available bit.

Since the entering data from the TDC arr ives with arbitrary phase with respect to the

local copy of the 40MHz clock, a phase s ampling circuit must be activated during

the initialization to select the appropr iate phase for sampling the incoming data.

After this sample period the bit clock f rom the TDCs can be disabled to reduce the

EM noise and mezzanine card power associ ated with generation of this clock.

5. An input multiplexer subsection p olls for data from the Serial to Parallel circuit at

40MHz, transmits found data to 18 indivi dual FIFOs, one for each TDC. Once the

data is stored the data available bit is reset.

6. A second polling multiplexer s cans the individual TDC FIFOs for data and if found

sends the 32-bit data unit to the optica l transmitter. If no data is present in the FIFO

for the polled TDC, an empty TDC flag is transmitted for the TDC.

7. At the conclusion of the 18 s tep TDC poll, a spacer word is inserted to insure time

step synchronization between the CSM an d the MROD. This word contains a special

code, D0000000, in hex. Two optical link idle code words follow the spacer word.

These insure link synchronization and do not appear as words to the MROD.

8. The TDC data words and their pa rity bits are modified so as to include parity infor-

mation both for the received parity fro m the TDC and to define independently the

outgoing parity. Since the TDC identifi er field (4-bits) of the TDC word is redun-

dant, given the data source to time divi sion position relationship, part of this field is

overwritten with two parity related bits . The lower two bits remain unchanged. Bit

27 is set to contain a parity error flag if the incoming word from the TDC fails the

parity test done within the CSM. Bit 26 of the outgoing TDC word is set so that the

parity of the outgoing word is odd (inc luding the Bit 27 error flag).

9.The final subsection pr ovides for voltage and temperature monitoring. The cable

from each mezzanine card provides a conn ection to its analog voltage regulator out-

put, its digital voltage regulator outpu t, and to an on-board temperature sensor.

These 18 x 3 lines plus 3 lines from the CSM regulators and temperature sensor, are

routed to a 64 channel analog multiplexe r and ADC. The 64-channel ADC is a direct

copy of the ELMB version fabricated on the back side of the CSM. The CSM-1 does

not contain the analog multiplexer. It will be added in the final design step.

Design Specifications

4 CSM-1 Design Development at UoM

The Bl ock Diagram

FIGURE 3. The Block Diagram of the CSM. The CSM-1 does not have the ELMB and

ELMB Mux. The opto-is olated JTAG connects through an adapter to various JTAG

sources.

Serial to Parallel

The TDC resets to an idle state from wh ich it sends no data. The reset can be derived

from the TTCrx and also received by th e CSM via JTAG. This reset erases any previous

activity on the input lines from the TDC. After a reset, the phase of the incoming data

from the TDCs must be sensed. This is done using the 4 phases of the 40MHz clock pro-

vided by the Xilinx Digital Clock Mana ger, DCM. This unit includes a DLL for clock

locking and also provides 0, 90 180, and 270 degree versions of the clock. To perform

the needed sampling of the phase of each TDCs arriving data bits, an automatic selec-

tion of the best choice between the 4 phases is made. The clock phase is chosen so that

the data changes between 6 and 12 nanos econds after the chosen clock phase. When the

sampling circuit stores its choice (2-3 cycles of the clock), normal data sensing can

begin. At this point the individual cloc ks from each of the TDCs can be disabled saving

power and potential clock noise pickup at the mezzanine card.

In norm al operation the start bit is sensed, 32 data bits are assembled into a shif t register,

parity is tested, and a stop bit is demanded for each TDC data word. Parity is tested and

an error bit is saved for reporting to the MROD. At this time the data ready flag is set

indicating that the 32-bit output regist er plus a parity flag contains data. This register

cannot change more than once ev ery 36 bit times since the data stream contains a start

bit, 32 data bits, parity bit, and 2 sto p bits.

Input Accumulation

The 32-bit data words arri ve from each TDC based on the data present in that particular

TDC and the availability of its output s equencer. Therefore, although all TDCs send

data for a given EVID before processing the next, data from a given TDC does not have

a well defined timing relationship to data for the same EVID from other TDCs. The

CSM is designed to be a simple time divi sion multiplexer and is not responsible for

event building. Thus, it can simply tran smit the assembled 32-bit words to the MROD in

the next available time slot for th e particular TDC. Since the data arrives at a maximum

Tubes

Hedgehog

Mezzanine

TTCrx

MROD

100 Mbyte/s

Gigabit

TTC

TTC Fibre

Clk, L1A, Calib

Mux

JTAG

Monitor

ELMB

Mux

18 Mezzanine

Cards Maximum

64 ADCs

ELMB

CAN

Bus

ROB

TTCvi

Central

Control

CSM

Opto-isolated copper JTAG

Serial to Parallel

Xtmr

Ethernet

Not in CSM-1

Design Specifications

CSM-1 Design Development at UoM 5

rate of 1.143 MBits/s from e ach TDC and 21 words are transmitted for each polling

cycle, any rate of serial transmission on the fiber to the MROD greater than 97MBytes/s

will not require any buffering within the CSM. This data rate can easily be handled by

the GOL sending 32-bit words at 25MHz or greater.

Multiplexer

It is expected that both h eader and trailer words will be enabled in the TDC. These

header and trailer words each contain th e EVID. Since the CSM does no test beyond

parity checking, these words are process ed the same as any other data words from the

TDC. Data words can be sent to the MROD whenever the link is up and the currently

polled TDC has data available. The link will be unavailable from a reset until resynchro-

nization is established. The CSM will send data only when the link is active and must

therefore, be resynchronized when ever the link goes down. The amount of data lost will

generally be significant since resynchr onization requires many clock cycles. The loss of

synchronization is sensed at th e MROD, were a reset will be initiated. This reset will

probably need to set the CSM and TDC to idle over JTAG, disable triggers being sent

from the CSM to the TDCs, resynchronize the link, and reestablish triggers. Lost data

for the interval required to do this re synchronization needs to be flagged at the MROD

where the BCID and EVID for triggers is available. Information defining the last event

fully transmitted to the MROD is av ailable in the EVID/BCID values within the data.

Fibre Protocol

The output of the CSM is 32-bit data uni ts, either TDC data words, empty TDC codes

for TDCs that have no data for the cu rrent polling cycle, spacer words, or link idle

codes. These words are to be sent to a fiber encoder/driver. The low power CERN GOL

chip can accept 32-bit words at rates up to 40MHz. This is well above the 25MHz

required to remain ahead of the data ar riving from the TDCs. The higher the rate chosen

for the CSM polling and hence for the GO L transmission, the more empty TDC words

that will be sent. A choice of the 25M Hz transmission clock, is a natural since this rate

can be generated with commercial os cillator components. Data will flow uninhibited

unless the link goes down and a full re set is initiated. The code in the CSM-1 is cur-

rently arranged to send no data to th e GOL until the first trigger is seen and to not send

data for time division cycles (19 word groups) when no data is available from any TDC.

Data Word Formats

The data words from the TDC are describe d in the AMT-1 document

and summarized

in Table 1 on page 6. Note that the head er and trailer words from the TDC provide the

EVID and the BCID words. The location of these bits permits a check of the data arrival

consistency as described above. The TDC data words also contain an error bit that indi-

cates when data has been missed. The dat a miss bit is to be interpreted as a flag that data

was lost between the time of the last da ta unit without a flag set for a given TDC and the

7. http://atlas.kek.jp/%7earaiy/amt1/in dex.html

Design Specifications

6 CSM-1 Design Development at UoM

data units with flags for th at TDC. The CSM defines two additional IDs, one for the

TDC empty word and one for the spacer wo rd. The IDs for these words are tentatively

defined in Table 2 on page 6. In the tim e division multiplexing scheme each TDC word

is sent in a time slot defined in terms of the spacer word position. Following the spacer

word, TDC unit 0 data is sent or if no d ata is available, an TDC empty word is sent. The

next time slot contains TDC unit 1 data if any is available. This process is described in a

note by Thei Wijnen

on the NIKHEF web page. The TDC field i n Table 1 on page 6 is

repl aced by the 4-bit field [Bit 27 - 24] composed of [TDC parity error, odd pari ty, 2 bits

unchanged]. Where TDC parity error = 1 if the parity of the received TDC word was not

consistent with the parity bit received and odd parity was not set to make the overall 32-

bit word odd parity.

A special set of formats exist for the CSM-1 when operating in event building mode. In

this mode the CSM-1 generates output com parable to that of the CSM-0. These special

outputs are defined in Table 3 on page 7. There are CSM headers and trailers defined,

wire encoding for TDC 16 and 17, and a s et of error conditions that flag incomplete

events, events with one or more parity errors in transmission from one or more TDC,

TABLE 1. TDC Data Word IDs and contents

ID Word Contents Depend s on ID Type

31-28

9876543210

1010

TDC EVID BCID

1100

TDC EVID Word Count

0010

TDC Mask Flags

0011

TDC Channel T E Coarse Time Fine Time

0100

TDC Channel Width Coarse Time Fine Time

0110

TDC Unused Errors

0111

TDC 0000 BCID

0111

TDC 0001 R L1 Occupancy

TABLE 2. CSM IDs for TDC empty (0000 ) and Spacer word (1101)

ID Word Contents Depends on ID Type

31-28

9876543210

0000

0100 0

1101

0000 0

8. http://...lookup Thei Wijnen note on the data stream for TDM

Design Specifications

CSM-1 Design Development at UoM 7

and the conditions that cause d the error. The wire encoding used in the CSM-0 is also

used here. See the CSM-0 Users manual

for details of this encoding.

For the CSM-1 event builder, the erro r words beginning with ID and subID 0x0101 and

0x0 the meanings of the remaining bits are given in Table 4 on page 7 below.

TABLE 3. Special CSM-1 Output Word Form ats in Event Builder Mode

ID Word Contents Depends on ID Type

31-28

9876543210

CSM-1 event builder output formats defin ed to date

0101 1001

EVID BCID

0101 1011

EVID Word Count

0101 1101

a. This subID flags an abnormal event te rmination which is preceeded by the error word

with subID zero.

EVID Word Count

0101 0000

Error Code Abnormal TDC

0101 0010

b. This CSM sub-type is a pad word, a nd appears only when event buffers are padded to a

multiple of 64(256) words. These follow the CSM trailer (sub-type 1101 or 1011), and

the Word Count does not include them in the total.

EVID Word Count

1010

TDC EVID BCID

1100

TDC EVID Word Count

0011

Wire Number T E Coarse Time Fine Time

0100

Wire Number Width Coarse Time Fine Time

9. http://atlas.physics.lsa.um ich.edu/docushare/dscgi/ds.py/View/Collection-207

TABLE 4. Flag Bits for CSM-1 Event Buil der Errors

Error or Flag Bit Posn

Abort was due to missing header/trailer (hdr_trl_abort)

Abort was due to missing header (miss_hd r_abort)

Abort was due to dpram over threshold

Abort was due to a timeout waiting for data

Flag -- BCID matching is disabled

Flag -- EVID matching is disable

Flags for AMTs with both good and comple te data

17-0

The CSM Module Specifications

8 CSM-1 Design Development at UoM

T HE CSM M ODULE S PECIFICATIONS

Component Details

The TTCrx

The TTCrx has the following functions:

• The TTCrx receives fibre dat a from the central timing and control logic, regenerates

the 40MHz LHC clock, receives the comm ands, and presents these commands

including the LVL1 trigger to extern al logic. It maintains counters for EVID and

BCID that are presented to external pins along with the LVL1 accept and other com-

mands.

• It provides for independent phase adju stments of two copies of the 40MHz clock.

• It provides for calibration pulses co ordinated with LVL1 triggers appropriately

delayed from the calibration timing.

The timing and calibration ro le of the TTC system is illustrated in Figure 4 on page 9. In

this figure, the role of the TTCrx for timing and calibration pulsing is shown. In addi-

tion, the role of the JTAG in establishi ng the parameters for the test calibration within

the CSM and mezzanine cards is illust rated. The TTCrx provides two clock outputs

each with its own phase adjust ment down to the 104ps least count. One of these phases

will be adjusted for gating of command s from the TTCrx including the LVL1 accepts

and resets. The second phase will be us ed to initiate the calibration pulse, permitting

adjustable timing to the TDCs in 104ps steps.

The CSM-1 acts in respo nse to a subset of both long and short format TTC commands.

The only short format command to whic h it responds, besides the EVID and BCID

resets, is that which initiates a calib ration trigger sequence for the attached mezzanine

boards. This command has user bits 3 an d 6 set on, and all other bits set off or ignored

(010010xx). The two least significant bi ts are ignored as they correspond to the EVID

and BCID clear strobes. A document sp ecifically concerned with calibration

is avail-

able.

10.CSM-1 Calibration Triggers, http://at las.physics.lsa.umich.edu/docushare/dscgi/ds.py/View/

Collection-214

The CSM Module Specifications

CSM-1 Design Development at UoM 9

Three Individually Addressed C ommands (IAC), or long format commands, initiate

CSM-1 activity. These are detailed in Ta ble 5 on page 9 below, and are discussed at

length in Reference on page 8.

The three commands detailed in Table 5 on page 9 are “external” TTC commands, i.e.,

they are relayed to the circ uitry attached to the external busses and lines of the TTCrx.

Each TTCrx also responds to seve ral “internal” commands, sometimes with detectable

external actions. In particular, the CS M-1 monitors for the response to the internal

ERDUMP and CRDUMP commands, which result in the external presentation by the

TTCrx of the contents of 10 of its 20 8- bit registers. This content is latched by the CSM-

1 and included in the JTAG bit-stream sh ifted out of the chip. See the section on JTAG

programming in this manual for more deta ils.

FIGURE 4. The Role of the TTCrx and JT AG Systems for Calibration. Calibration

parameters are established via JTAG fr om the DCS system followed by TTCvi

commands as needed to generate the sign al injection and to trigger a level 1 accept for

acquisition of the data.

The Serial to Parallel Units

In the CSM-0 the serial to parallel conversion of data from the TDCs was done in three

small Xilinx FPGAs, each processi ng 6 channels of TDC data. This same arrangement

could be implemented in the final CSM. However, the availability of more logic and

higher pin counts in newer FPGAs makes it possible to place all channels in a large chip

along with logic for the multiplexer. Th e presence of DLL clock circuits with multiple

phase outputs further aids this process by enabling incoming data sensing at the phase

appropriate channel by channel. The se rial to parallel circuits implement the logic to:

• Autosense the individual TDC data st reams and select the appropriate 1of 4 possible

clock phase to sample the arriving da ta. Disable the TDC clock after sampling

through the JTAG programming of the AMT- 3.

TABLE 5. Individually Addressed Comma nds recognized by the CSM-1

SubAddress Data Bits Action

xxx Initiate TTC reset

Strobe Duration Set the duration of the calibration strobe to the

given number of 25ns clock tics

Strobe Duration Set the duration of the calibration strobe, as

a bove, and initiate the strobe itself.

TTCvi

Set

orbit

pulse

time

N ticks

TTCrx

CSM

ASD

Setup data

for pulse on

JTAG lines

delay

L1A

synchronized pulse

pulse

JTAGsetup

Mk II

DCS

Crossing

L1A

to CSM

The CSM Module Specifications

10 CSM-1 Design Development at UoM

• Reset to a quiescent st ate expecting a logic true start bit as the first data element

from each TDC.

• After a start bit is received, begi n assembly of the next 32-bits into a register.

• Compute and test the parity of the 32- bit string, setting an error flag if a fault is seen.

• Transfer the 32-bits to a holding regi ster upon receipt of the stop bit, set a “data

ready” flag indicating data present, and return to the quiescent state.

• Reset the “data ready” flag upon receipt of an “acknowledge” signal from the poll-

ing multiplexer.

The JTAG Interface

The JTAG signals connect to the CS M at a programmable PROM, and then to the

FPGA, passing next to the GOL chip, and finally to the TTCrx. From there it returns to

the FPGA on user handled pins. The JT AG chain is distributed to the mezzanine cards

as defined by the enable bits for indivi dual TDCs. Connection of the JTAG bus to the

mezzanine cards is totally controlled in the FPGA code and can be modified as required.

The Virtex-II series of Xilinx chips us ed for the CSM implementation support numerous

I/O levels and standards which include L VDS and differential PECL. The chips pow-

erup and initialize to an active TAP (JT AG) control for downloading configuration data.

To provide external control of this bo otstrap procedure, external circuitry must be able

to force this “first state” for conf iguration to occur. The external JTAG can be provided

by any commercial controller or by the DCS system. One means of forcing the “first

state” will be to cycle the power. The TTCrx system could also generate a global reset.

Careful evaluation of the startup op eration must be done to insure that no frozen states

exists other than through a true hardwar e device failure. This JTAG link will support the

following functions:

• Initialization of the FPGA code with in the Xilinx chips.

• In itialization of the parameters of the CSM and ASD/TDC cards

• Initialization of the parameters of th e TTCrx.

• Controlling the run/ reset/resynchronization of the CSM, TDC, and ASD

The Polling Multiplexer

The polling multiplexer is actually two polling multiplexers, one that scans for data

ready flags from the 18 serial to parall el units and enters data found into 18 distinct

FIFOs and a second that scans for FIFO data and if found places it into the time division

sequence of the output optical transmitt er. The polling rate of the first multiplexer is

based on the 40MHz LHC clock and the rat e of the second is 25MHz as set to guarantee

to empty the FIFO faster than it can fi ll. The first multiplexer sequences through the

steps:

• Reset to polling address to TDC 0 an d to idle mode (not active).

• If in active mode, examine the “data ready” flag for the currently addressed TDC.

• If data is available, s end it to the FIFO for the addressed channel and reset the flag.

• Increment the TDC addressed and loop.

The CSM Module Specifications

CSM-1 Design Development at UoM 11

The second multiplexer has 21 steps. Eighteen of these steps are associated with the 18

FIFOs that hold data from the indi vidual TDC. One, 18, defines the state when the

spacer word is to be transmitted. Th e remaining two correspond to optical fiber idle

words. The sequence through the steps:

• Reset the polling to address 18 and to idle mode (not active).

• If the polling multiplexer address is 18 output a TDM spacer word to the GOL chip.

• If the multiplexer address is 19 or 20 send a Gigabit Ethernet idle code.

• If the address is 0-17, test the FIFO for data in the addressed channel.

• If no data is present in the FIFO, sen d a TDC empty word.

• If da ta is available for FIFO (TDC) addressed, send an “acknowledge” to the FIFO

to declare that the word has been taken and send the word to the GOL chip.

• Increment the polling address modulo 2 1 to advance to the next step.

The Fibre Sequencer

The GOL chip accepts 32-bit data words from the polling multiplexer and a 2-bit write

control that:

• Sends a Gigabit Ethernet idle, if th e 2-bit control is 00. The 32-bit data is ignored.

• Sends the 32-bit data presented to th e GOL, if the control is 01.

• Sends an Extend code, if the control is 10.

• Sends a transmit error code, if the co ntrol is 11.

The DCS An alog Monitor

The 64-cha nnel analog monitor is a direct copy of the 64-channel ELMB multiplexer

temperatures of the mezzanine cards an d of the CSM itself. It is powered by the CSM

3.3 volts via a switched capacitor regul ator that outputs 5 volts for the unit. Connections

to the ELMB processor are optical is olated using the same components as used for the

ELMB multiplexer that monitors chamber conditions. The location of the monitor on

the CSM is chosen to minimize the interc hange of signals between the CSM and the

DCS circuitry. Only 5 opto-isolated sign als are needed to provide JTAG to the CSM. In

addition, a second set of 8 I/O lines fr om the ELMB processor will be extended to the

CSM and opto-isolated. The extension pro vides for monitoring 64 differential signals or

128 connections. Of these 57 differenti al signals are anticipated, 3 from each mezzanine

card and 3 from the CSM. Figure 5 on pag e 12 illustrates the relationship of the DCS to

the CSM mounted portions of the DCS.

The CSM Module Specifications

12 CSM-1 Design Development at UoM

FIGURE 5. Relationship of the DCS elem ents on the CSM to the DCS System

CSM Interconnects

Figure 6 on page 12 shows the layout of the CSM. An arrangement with all passive

components on a motherboard whose role is to provide attachment to the 18 mezzanine

cards via their 40-pin connectors.

FIGURE 6. The Layout and Interconnectio n of the CSM

These signals must be routed to various subsections according to the list below.

• Mezzanine to CSM Motherboard x 18 max

8 JTAG (LVDS), 8 Data/CLK (L VDS) = 16 lines

8 Analog PW R, 8 Digital PWR, 6 Sense, 2 Calibration = lines total

CAN

signals

CAN Power

ELMB Power

T sensors

DCS Motherboard

Credit Card

B field meter

regulator

CAN

driver

CAN interface

+ micro

controllers

ELMB

SPI

64 Channel

ADC

64 Channel

Mux/ADC

CSM SPI

Digital I/O

CSM

Power

57 Analog

Channels

SPI

CSM

JTAG

Cross Plate

18 connectors (40 pin) Blue at near end , Red at far

TTC Receiver CSM Multiplexer

Gigibit Ethernet

ELMB Mux

140 Pin

Interconnect

ELMB JTAG & SPI

Note: For the small

chambers these headers

are not needed and board

can be shorter.

Note: ELMB Mux is copy of 64 channel DCS mux

to MROD

24 40 ?

The CSM-1 Prototype Devel opment

CSM-1 Design Development at UoM 13

• CS M to CSM Motherboard - 3 x 140 pins = 420 pins

8 JTAG (LVDS), 10 Data/CLK (LVDS) x 18 mezzanine cards = 324 lines

6 Sense, 8 Digital PWR, 8 ELMB PWR = 22 lines

• ELMB Mux to CSM Motherboard

6 Sense x 18 Mezzanine cards, 6 Sense x 1 CSM = 114 lines

5 JTAG I/O, 5 ELMB SPI, & 3 Spares = 13 lines

The Layout, as seen from above, is shown in Figure 7 on page 13. Note that for cham-

bers not requiring more than 12 TDCs, th at the motherboard can be reduced in size to

the area required by the active componen ts. This will entail the design of 2 different

motherboards but affords the option of a short unit for small chambers.

FIGURE 7. The CSM-1 and Motherboard.

T HE CSM-1 P ROTOTYPE D EVELOPMENT

Design Elements

The CSM-1 has as it goal the rapid desig n of a module more closely aligned with the

final CSM than is the CSM-0. It presumes an early version of the MROD and the use of

fiber connections to the trigger, timing , and control and to the data acquisition flow. The

fiber interconnect more realistically represents the clock noise, data noise, and ground-

ing environment of the final configur ation. The only justification for the intermediate

design of a second prototype is expedi ency. The goal is to have a fiber connected design

that can reside on chamber using the sa me interconnect motherboard, TTC system, and

TTC optical input

GOL optical output

DCS connection for JTAG

& 8-bit I/O communication

Alternate JTAG

via Jumpers

5Volt power

18 Mezzanine I/Os

on 40-pin ribbons

2 rows (near seen)

9 on each side in

Mounting plate

Standoff

The CSM-1 Prototype Development

14 CSM-1 Design Development at UoM

the opto-isolated JTAG co nnection anticipated in the final CSM. The items to be

included in the CSM but not present in the CSM-1, are the analog multiplexer for volt-

age and temperature sensing, an d the final TTC system with calibration programming.

The connections to do calibration will b e in place in the CSM resident FPGA, but per-

haps not from the TTCvi programming. The TTC system is described on the CERN web

pages at CERN

where the TTCvi Mark I is described. The specifics of the CSM-1 are:

• The main multiplexing task of the CSM- 1 has been compressed from 4 FPGAs into

one using a scheme of clock phase sampli ng to remove the need for many indepen-

dent clock networks, using a large fine pitch ball grid array chip, and using a LVDS

compatible FPGA from Xilinx, the XC2V100 0FPBG456. Prototype testing of the

optical fiber output to the MROD will be done using the CERN GOL chip and the

Infineon V23818-K305-L57. The final desi gn will likely use a simple laser diode in

place of the transceiver since the in put channel is unused in the CSM to MROD link.

• The CSM-1 requires clocking and event triggering. For the CSM-1 design the Trig-

ger, Timing, and Control will be based o n the CERN developed TTC system. Our

transition from the internal simulation of the trigger and timing to the externally gen-

erated functions implies and integration of the CERN chip into the module and the

software integration of the TTCvi modu le into our environment. We plan to use the

Mark I version of the TTCvi initially an d switch to the Mark II version in 2003.

• The initialization of the ASD/TDC m ezzanine cards is done via JTAG protocol

which in the case of the CSM-0 is bu ilt into the VME functions of the CSM-0 mod-

ule. In the CSM-1 no VME connections is available. The CSM-0 will use an opto-

isolated JTAG function as will the fi nal CSM. However, the final CSM will have a

DCS connection and a CAN bus interf ace to provide the programming for the JTAG.

To make progress toward this final goa l, the opto-isolated JTAG will be driven from

a commercial module.

• The CSM will interconnect to the AS D/TDC modules via shielded ribbon cables that

leave the Faraday cages and converge at the spacer plate of the MDT. At this loca-

tion they will attach to a passive inter connect on which the active CSM will reside.

To make use of the current CSM-0 with th e new interconnect board, a MiniAdapter

board has been constructed with the sa me circuit design as the adapter currently in

use with the CSM-0, but designed to at tach to the passive interconnect. Following

use with the CSM-0, a simple replacemen t of this new adapter with the CSM-1 will

convert a chamber from CSM-0 readout to the MROD readout via the CSM-1.

• The CSM-1 will not contain the analog multiplexer nor will it interact with the DCS

system initially.

• The CSM-1 will not contain any Single Bit Upset detection or correction code. Since

this code is firmware, it is anticipated that work on this code will follow the certifi-

cation of the design in the first half of 2003.

11.TTC system web page s, http://ttc.web.cern.ch/TTC/intro.html

JTAG Programming

CSM-1 Design Development at UoM 15

JTAG P ROGRAMMING

CSM-1 JTAG

The CSM-1 has a JTAG string which is a composite of several bit strings for the TTCrx

the TTC RO status, and the CSM module RO status. These strings are defined in the

CSM VerilogHDL code and can be modi fied rather easily. The current sizes of the

JTAG elements are represented in Table 6 on page 15 along with the overall order of

these strings in the JTAG sequence.

The TTCrx string is defined in the TTCrx manual

. In the CSM, this string is loaded

via JTAG and presented to th e TTCrx chip following a reset signal. This reset initiates

the timing sequence for loading the TTCr x from PROM with the FPGA taking the role

of the PROM. The CSM parameter string of 72 bits provides RW control for the CSM,

the GOL, and TTCrx as shown in Table 7 on page 15. The CSM status bits are RO and

represent the version number of the CS M code, the state of various DLL lock lines, and

a few error flags.

TABLE 6. JTAG String Lengths & Position

String Name # bits Bi t posn

CSM_parm_len

72 206

TTCrx_init_len

80 278

CSM_status_len

28 0

AMT_parity_err_len

18 28

TTC_readback_len

160 46

12.Timing receiver ASIC (TTCrx) Referenc e Manual, http://ttc.web.cern.ch/TTC/intro.html

TABLE 7. CSM JTAG Control Bit Definiti ons

Field Bit posn

TDC enable low bit

TDC enable length

Include mezzanine cards in JTAG flag

CSM enable trigger bit

Mezzanine command delay pipe length lo w bit

Mezzanine command delay pipe length, le ngth

Spare bits low bit

Spare bits length

Suppress idle cycle bit (TDM only)

Include sync status bit (TDM only)

GOL differential input selection flag

JTAG Programming

16 CSM-1 Design Development at UoM

The TTCrx readb ack bits hold the contents of the 20 I

C addressed 8-bit registers of the

TTCrx. Two fpga build versions are possible. In the default version, one I

C register is

read every 400ms, covering all 20 regist ers. In the second version, only the first ten reg-

GOL ld0 bit

GOL ld1 bit

GOL pll bit

GOL laser bit

GOL negative edge selection bit

GOL mode selection bit

GOL tdi bit

GOL address low bit

GOL address length

TTC tdi bit

TTC use prom flag bit

CSM next state low bit

CSM next state length

CSM take command action

CSM BCID no match bit (event builder on ly)

CSM EVID no match bit (event builder on ly)

CSM send all AMT types bit (event builde r only)

CSM no header or trailers bit (event bu ilder only)

CSM disable TTC and GOL JTAG bit

Spare bit 1

Spare bit 2

Spare bit 3

Spare bit 4

Spare bit 5

Spare bit 6

Spare bit 7

Spare bit 8

Spare bit 9

Spare bit 10

TABLE 8. TTCrx Readback Register

Field bit posn

TTCrx registers 9 to 0

TTCrx registers 19 to 10 ( I

C access only)

TABLE 7. CSM JTAG Control Bit Definiti ons

Field Bit posn

JTAG Programming

CSM-1 Design Development at UoM 17

isters are available. Thes e 10 registers are accessed passively via the ERDump and

CRDump commands of the TTCrx. (See Table 8 on page 16) The CSM status register

bits reflect this difference, with the f irst column describing the ERDump/CRDump situ-

ation, and the second describing the I

C situation. (See Table 10 on page 17 ) The

ERDump/CRDump version will eventually be phased out.

The AMT parity error bits are cleared wh en data acquisition is enabled and accumulated

from the parity error calculation fo r each data word separately for each of the AMT

chips (each mezzanine card). A stop and restart of the data acquisition begins another

set of 18 accumulated OR bits.

The CSM status register bit definition s are indicated in Table 10 on page 17 using the

same convention as provided in the param eters file. In this case the bits are RO with the

data sent via the JTAG output stream be ing ignored.

TABLE 9. AMT Parity Error Flags

Field bit posn

TDC 0 Flag

TDC 1 Flag

TDC 2 Flag

TDC 3 Flag

TDC 4 Flag

TDC 5 Flag

TDC 6 Flag

TDC 7 Flag

TDC 8 Flag

TDC 9 Flag

TDC 10 Flag

TDC 11 Flag

TDC 12 Flag

TDC 13 Flag

TDC 14 Flag

TDC 15 Flag

TDC 16 Flag

TDC 17 Flag

TABLE 10. CSM Status Register Bit Defi nitions

Field bit posn

CSM version number low bit CSM version number low bit

GOL ready bit GOL ready bit

TTC ready bit TTC ready bit

LHC clock locked bit LHC clock locked bi t

JTAG Programming

18 CSM-1 Design Development at UoM

If the "Suppress Idle Cy cle" bit is turned on in the CSM JTAG Control Bit sequence,

then any 21 word GOL transmission cycle which contains only TDC empty words

(Table 2 on page 6) will be suppressed i n favor of optical idle transmissions for the

duration of the cycle. If this bit is n ot set, then all cycles will be transmitted over the

fiber once acquisition is enab led and the first trigger happens. This bit can be changed

at any time and will take effect at the start of the next transmission cycle.

If the "Include Sync Status" bit is turn ed on in the CSM JTAG Control Bit sequence,

then 26 bits of additional information will be output in each transmitted TDC Spacer

Word. The nominal value of this word as shown in Table 2 on page 6 is 0xd0000000.

When the control bit is turned on the co ntent of this word is modified as shown in Table

11 on page 18. With the bit set, when a trigger is received by the CSM-1 the number of

clock tics (32 bits) since the previous trigger is latched. This value is transmitted in the

next two Spacer Words as shown in the ta ble. The spacer word then reverts to transmit-

ting a "fill content type zero" word un til the next trigger arrives.

XMT internal clock lock bit (FPGA) XMT internal clock lock bit (FPGA)

XMT external clock lock bit (GOL) XMT ex ternal clock lock bit (GOL)

Unused OR (for fooling synthe sizer) Unused OR (for fooling synthesizer)

TTC prom load error flag TTC prom load e rror flag

CSM state low bit CSM state low bit

Sample phase error bit Sample phase erro r bit

C Operation Failure I

C Operation Failure

Spare = 0 TTC I

C compare error

TTC dump compare error bit Spare = 0

CSM error bit CSM error bit

TABLE 11. Modified TDC Spacer Word cont ent

Bit Range Value Mean ing

31-28 0xd ID Code (fixe d)

27 0x0 None (fixed)

26 0,1 Fiber transmission parity (odd )

25 Status bit 7 Always zero

24 Status bit 6 LHC clock lock ed

23 Status bit 5 XMT internal clock lock

22 Status bit 4 XMT external clock lock

21 Status bit 3 Sample phase error

20 St atus bit 2 I

C Operation Failure

19 Status bit 1 TTC I

C compare error

TABLE 10. CSM Status Register Bit Defi nitions

Field bit posn

JTAG Programming

CSM-1 Design Development at UoM 19

Care should be taken in se tting this bit, as it is likely that the MROD does not under-

stand any value for the Spacer Word ot her than the default, and will not work if that

value is not transmitted. A similar cau tion exists for older versions (such as that used at

H8) of the Michigan CSMtest code. Fill word content definitions appear in Table 12 on

page 19

CSM-1 States

The CSM-1 operates in states defined cur rently according to Table 13 on page 19. Most

of the states have to do with initi alization. Once running the CSM-1 requires no inter-

vention as long as synchronization is maintained and errors are rare. Since the CSM-1

continues to be under development, one s hould expect the number of states, parameters,

and status bits to be fluid. Check regul arly for updates of this manual by examining the

Docushare pages for new versions.

18 Status bit 0 CSM error bit

17-16 0x0,1,2 Fill content type

15-0 Fill content Type defin itions, see next

TABLE 12. Fill content Type Definiti ons

Type Meaning

00 Zero

01 Low 16 bit count of clk25 since last trigger (32 bit value)

10 High 16 bit count of clk25 since last trigger (32 bit value)

TABLE 11. Modified TDC Spacer Word cont ent

Bit Range Value Mean ing

TABLE 13. CSM-1 States

State Code

IDLE_STATE 0

FPGA_RESET_STATE 1

RESET_TTC 2

RESET_GOL 3

WAIT_LOCK 4

DCM_RESET_CLK25 5

SAMPLE_AMT_PHASE 6

JTAG_RESET 7

TTC_LOAD 8

JTAG Programming

20 CSM-1 Design Development at UoM

The CSM-1

Operational State

Machine

The states detailed in Table 13 on page 19 can be individually entered, but a few are

really only useful when automatically en tered as a result of selecting some other opera-

tion.

This section attempts to document the actions taken as a result of selecting the various

states. Following each state name is the decimal value of the enumerated type to which

it corresponds, a'la Table 13 on page 19 .

IDLE_STATE (0): This is the n ormal state in which the FPGA operates. All other states

eventually flow back to this one follo wing the completion of any actions taken in those

other states. Issued JTAG Operational St ate changes will only be made if this is the cur-

rent state when the JTAG command is received.

RESET_TTC (2): Entr y to this state toggles the Reset_b pin of the TTCrx chip for

approximately 400ns. Exit from this stat e at the end of the reset period depends upon the

setting TTC_use_prom_bit of the CSM para meters JTAG register. If the TTC should

use the FPGA as its load prom, then that is the next action taken (state WAIT_PROM).

Otherwise, reset of the DCM core contro lling the re-generation of the 25ns LHC clock

within the FPGA is undertaken (state DC M_RESET_CLK25).

WAIT_PROM (10) : Wait while the TTCrx chip uses the CSM-1 FPGA as its load prom.

Exit from this state is to the DCM_RESET _CLK25 state.

DCM_RESET_CLK2 5 (5): If the TTCrx is reset, then the DCM core within the CSM-1

FPGA must be reset to restart the lo cal clocks. Exit from this state is normally to the

IDLE_STATE (see the Power Up Sequ ence detail below).

RESET_GOL ( 3): Entry to this state toggles the reset_b pin of the GOL chip for approx-

imately 400ns. Exit from this state is to the WAIT_LOCK state.

WAIT_LOCK (4): Wait while the GOL chip re-synchronizes and re-locks itself. This

wait is approximately 800ms long. Exit f rom this state is to the IDLE_STATE.

SAMPLE_AMT_PHASE (6): Entry to this stat e initiates a sampling of the phase of the

data and strobe signals returning from each mezzanine board. At the end of the sample

period, the CSM-1 clock phase most appr opriate for acquiring the serial bit stream from

each active mezzanine board will be know n and selected for use. Exit from this state is

to the IDLE_STATE.

AMT_RESET 9

WAIT_PROM 10

TRY_RESET_ERROR 11

TABLE 13. CSM-1 States

State Code

JTAG Programming

CSM-1 Design Development at UoM 21

--> IMPORTANT NOTE: The FP GA “assumes” the serial clock is already JTAG

enabled from each active mezzanine board . If it is not, the CSM_error_bit of Table 9

will be set due to the failed phase samp ling.

JTAG_RESET (7): Toggle the JTAGTRST_b signal line to the TTCrx and GOL chips

for approximately 350ns. Exit fro m this state is to the IDLE_STATE. NOTE: The JTAG

instruction register of both the GOL and the TTCrx is loaded with the IDCODE instruc-

tion as a side-effect of this operation.

TTC_LOAD (8): Force a TTC PR OM load from the FPGA independent of the setting of

the “TTC use prom flag” JTAG bit. Othe r operations are identical to those of the

RESET_TTC state operating in a non-power -up sequence.

AMT_RESET (9): T oggle the mezzanine board AMT reset line for approximately

800ns. Exit from this state is to the ID LE_STATE.

TRY_RESET_ERROR (11): During the course of operations within the CSM-1 state

machine, an error could occur. This resu lts in setting the “CSM-1 Error” bit. Entry to the

TRY_RESET_ERROR state results in an attempt to determine and correct the cause of

the error. If the attempt succeeds the b it will be cleared. Exit from this state is to the

state appropriate to correcting the er ror. If more than one error exists, it could take two

or three entries to this state before th ey are all handled.

FPGA_RESET_ STATE (1): Entry to this state re-initializes the CSM-1 FPGA to its

power up state. Exit from this state is through the RESET_TTC and RESET_GOL states

as detailed below for the Power- Up Sequence of events.

All ot her values for the CSM-1 State are ignored, and treated as if they were the

IDLE_STATE.

Note that a JTAG request to enter a s tate will be ignored if the state machine is not

in an IDLE_STATE when the JTAG regist er is updated. No error will be reported in

this situation.

Power Up Sequence Following a board po wer up, we recommend the following sequence of states and oper-

ations before the FPGA is used for da ta acquisition. The first two operations are auto-

matically performed as a result of th e FPGA initialization sequence.

1. RESET_TTC. Note that all TTCrx initi alized during power up will have the same

address, which is hard-coded in the CSM- 1 initial programming strings. If individual

TTCrx addresses are desirable, then th e correct addresses should be set in the JTAG

strings of each CSM-1, in conjunction with a RESET_TTC command.

2. RESET_GOL

JTAG Programming

22 CSM-1 Design Development at UoM

3. Enable the connected se t of mezzanine boards, and enable the CSM-1 to JTAG pro-

gram them. The IDLE_STATE should be se lected while doing this, or it can be done at

the same time as the RESET_GOL state is selected.

4. JTAG program th e mezzanine boards, selecting the IDLE_STATE in the process. A

continuous return clock should be select ed.

5. SAMPLE_AMT_PHASE

6. Disable the mezzanine boards AMT clock return to the CSM-1 (IDLE_STATE).

7. Disable mezzanine board JTAG operati ons, and enable CSM-1 acquire data JTAG bit

(IDLE_STATE).