CSM-1users.fm

UMHE-02-10
UMHE-http://atlas.physics.lsa.umich.edu/docushare

UMHE-02-10

UMHE-http://atlas.physics.lsa.umich .edu/docushare

CSM-1 & CSM

Design Manual

Design of the Ch amber Service Module & Prototype 1 (Rev C)

University of Michigan

Ju ly 8, 2003

J. Ch apman, P. Binchi, R. Ball, T. Dai, & J. Hall

CSM-1 Design

Development at UoM

O VERVIEW OF P ROGRAM

The CSM-0 (a prototype CSM) is do cumented in specifications

on the University of

Michigan ATLAS web server Docush are. It was developed as a first step toward the

design of a production muon front-e nd readout multiplexer, initialization controller, tim-

ing distributor, and calib ration controller. Documents on the DocuShare server are

linked via ATLAS Electronics > MDT Electronics and are called “CSM-0 Development

and Users Manual” and “CSM-0 Desi gn Internals Document”. The general characteris-

tic of the design has not change d and is specified in numerous ATLAS muon publica-

tions as the near chamber part of the NIMROD

. The work to design the CSM began

with simulations of the d ataflow and were reported in the LEB99 meeting in Snow-

mass

and are available on the Docush are server. The simulations were performed in

VerilogHDL. A similar VerilogHDL de scription of the AMT-3

will when a similar

description of the ROD

is complete, provide a tool for continuously evaluating the

design as it matures. The prototyp e CSM-0 synthesized from the original VerilogHDL

code will serve as an active test f ixture for the design. This prototype has been built as a

6U VME module containing the CSM unit, a JTAG interface, an LHC clock emulation,

and a VME readable output FIFO. Since the timing and control functions of the LHC,

the TTCrx

chip and its associated driver components, were not available in the test

1. http://atlas.physics.lsa.umich.e du/docushare/default.htm

2 . http://umaxp1.physics.lsa.umich.edu/~chapman/atlas/nimrod.ps

3. Proceedings of the LEB99 confere nce at Snowmass (1999)

4. http://atlas.kek.jp/%7earaiy/amt1/index.html

5. http://www.nikhef.nl/pub/departm ents/et/atlas)mdt/index.html

6. http://www.cern.ch/Atlas/GROUPS/FRONTEND/Ttc1.htm

Overview of Program

CSM-1 Design Development at UoM 1

environments, thes e functions are designed into a separate Xilinx called the TTCem cir-

cuit. This additional Xilinx is required to accept the external trigger, to provide the

EVID and BCID, and to synchronize t he trigger with the free running clock that gener-

ates the simulated LHC crossing in tervals.

The differences between the CSM-0 and the final CSM unit mounted on chambers are

significant but the two units have a one to one correspondence. The final on-chamber

version will be simpler in th at it will not need to perform event building. Since the ROD

must form events from 6 CSM outpu t streams, there is no need to do this job twice. The

on-chamber CSM will therefore be ex pected to do simple time-division multiplexing of

data from the 18 TDCs. With this sc heme the TDC from which any particular unit of

data originates can be determined f rom the word position in the time sequence. If no

data is available from a sp ecific TDC, an idle position-holder will need to be sent. The

full event building version of th e CSM-0 can be thought of as a CSM/ROD pair that

handles only one chamber in contras t to the final system which will incorporate six sim-

ple time multiplexing CSMs feeding a six input ROD module (called an MROD) where

the event building will take pl ace. Thus, the on-chamber CSM will not have a trigger ID

FIFO, a deep input FIFO, an output data FIFO, or a word count FIFO. It will have an

optical output encoder and driver to send 32-bit words to the MROD module which

accepts its output and the output from 5 other CSM modules.

The overall flow of data from the chamber wires up to the modules in the RO D crate is

represented in Figure 1 on page 1. The simulation performed for the CSM/ROD con-

tained the 18 TDC inputs each with their own serial data and clock. It described the out-

put flow of data along the channe l destined for the ROD. Data units were collected from

the TDCs by the CSM and transmitted in turn to the ROD modules. Data is transmitted

from the CSM as 32-bit units serial ized and sent on a single fiber. The 32-bit data units

are either TDC headers, trailers, t ime digitizations, or various control words. The pri-

mary task of the prototype CSM, cal led the CSM-1, is time-division multiplexing. Data

from each TDC is sent as requested by the level 1 trigger when the front-end link from

the individual TDC becomes free, i.e. , when data from all previo us triggers are sent.

In dividual events are separated by header and trailer words. These header and trailer

words are se lectively enabled for transmission in the TDC. At least trailer words are

required to indicate the end of event. The simulation assumes that both headers and

trailers are sent for redundancy.

FIGURE 1. The Position of the CMS i n the muon data flow.

The time-division CSM collects da ta from each of 18 individual TDCs into individual

FIFOs, polls for data available in these 18 FIFOs, and if present simply forwards the

TDC data to the ROD module in th e time slot specified for that TDC. The time position

CSM

ROD

Mezzanine Card Chamber Service Modu le ReadOut Driver

Containi ng ROD Cards

18/CSM 6/ROD

ASD/TDC

In Faraday cage 1 per chamber 1 ROD Card/TgrTower

Design Specifications

2 CSM-1 Design Development at UoM

in the output stream defines the source of the data. As a means to guarantee that the time

positions are synchronized between the sending CSM and the receiving ROD module, a

spacer word is sent as the 19th data unit each cycle. To insure synchronization of the

sending and receiving clock, 2 optical link idle words (not to be confused with the idle

word used to flag “no data” fr om an individual TDC) are sent following the spacer

word. The basic flow of information is illustrated in Figure 2 on page 2.

FIGURE 2. Serial data flow and even t transmission by the CSM

D ESIGN S PECIFICATIONS

Module Subsections

The CSM will have six subsections . Figure 3 on page 4 illustrates these blocks, the

JTAG initialization, the trigger timing and control, the optical transmitter, the serial to

parallel receivers, the multiplexer , and the environment monitor. The list below further

defines the content of these blocks .

1. The JTAG init ialization block communicates to an external controller. This control-

ler can be any unit which adhere s to the JTAG standard. Opto-isolation circuits pro-

vide for local ground separation from the ground of the external JTAG controller.

Thus the external controller must supply its own power and ground to the CSM unit.

2. A second subsection contains the TTCrx chip and its optical receiver. It contains the

reset functions, the EVID and BC ID registers, the phase adjustments for the 40MHz

clock, the decoding of the LV L1 trigger, and the calibration functions. The CSM

makes standard use of the TTC syst em and requires no special interconnections.

TDC

10 9

910

CSM

HeaderDataTrailer

Spacer Word

Spacer Word

18 TDCs maximum

13 14 15 16117 23456

Time Multiplexed TDC Data Words

Idle Words

Design Specifications

CSM-1 Design Development at UoM 3

Since the CSM is co nfigured as a simple time division multiplexer, it does not use

the BCID and EVID registers.

3. The optical transmitter is b ased on the CERN-designed radiation-hard GOL chip and

an Infineon optical transceiver. This unit can accept 32-bit words at up to 40Mhz for

transmission along an optical fiber . Since we need only send 32-bit words at 25MHz

to remain ahead of the maximum ra te at which data can be received from all TDCs

(along with the spacer/idle words) , we will use the transmitter at 25Mhz. At the

MROD the 32-bit words are collected, the spacer word checked, the TDC empty

words removed, and the actual TDC d ata stored. The optical transmission idle words

do not appear at the receiver outp ut. They serve only as synchronization characters.

4. Input from the TDCs arrives as a serial bit stream that is assembled into 32-bit words

under the control of a sequence that seeks a start bit, assembles 32 data bits, tests a

parity bit, and outputs the 32-bit data word with an accompanying data available bit.

Since the entering data from the TDC arrives with arbitrary phase with respect to the

local copy of the 40MHz clock, a ph ase sampling circuit must be activated during

the initialization to select th e appropriate phase for sampling the incoming data.

After this sample period the b it clock from the TDCs can be disabled to reduce the

EM noise and mezzanine card po wer associated with generation of this clock.

5. An input multiplexer subs ection polls for data from the Serial to Parallel circuit at

40MHz, transmits found data to 18 individual FIFOs, one for each TDC. Once the

data is stored the data available b it is reset.

6. A second polling multiplexer scans the individual TDC FIFOs for data and if found

sends the 32-bit data unit to the o ptical transmitter. If no data is present in the FIFO

for the polled TDC, an empty TDC fl ag is transmitted for the TDC.

7. At the conclusion of the 18 step TDC poll, a spacer word is inserted to insure time

step synchronization between the CS M and the MROD. This word contains a special

code, D0000000, in hex. Two op tical link idle code words follow the spacer word.

These insure link synchronizatio n and do not appear as words to the MROD.

8. The TDC data words and their parity bits are modified so as to include parity infor-

mation both for the received parity from the TDC and to define independently the

outgoing parity. Since the TDC id entifier field (4-bits) of the TDC word is redun-

dant, given the data source to time division position relationship, part of this field is

overwritten with two parity related bits. The lower two bits remain unchanged. Bit

27 is set to contain a parity error flag if the incoming word from the TDC fails the

parity test done within the CS M. Bit 26 of the outgoing TDC word is set so that the

parity of the outgoing word is odd (including the Bit 27 error flag).

9. The final subsection provid es for voltage and temperature monitoring. The cable

from each mezzanine card provides a connection to its analog voltage regulator out-

put, its digital voltage regulator output, and to an on-board temperature sensor.

These 18 x 3 lines plus 3 lines fr om the CSM regulators and temperature sensor, are

routed to a 64 channel analog mu ltiplexer and ADC. The 64-channel ADC is a direct

copy of the ELMB version fa bricated on the back side of the CSM. The CSM-1 does

not contain the analog multiplexer. It will be added in the final design step.

Design Specifications

4 CSM-1 Design Development at UoM

The Block Diagr am

FIGURE 3. The Block Diagram of the CSM. The CSM-1 does not have the ELMB and

ELMB Mux. The opto-isolated JTAG connects through an adapter to various JTAG

sources.

Serial to Parallel

The TDC resets to an idle state fro m which it sends no data. The reset can be derived

from the TTCrx and also received by the CSM via JTAG. This reset erases any previous

activity on the input lines from the TDC. After a reset, the phase of the incoming data

from the TDCs must be sensed. This is done using the 4 phases of the 40MHz clock pro-

vided by the Xilinx Digital Clock Manager, DCM. This unit includes a DLL for clock

locking and also provides 0, 90 1 80, and 270 degree versions of the clock. To perform

the needed sampling of the phase of each TDCs arriving data bits, an automatic selec-

tion of the best choice between th e 4 phases is made. The clock phase is chosen so that

the data changes between 6 and 12 n anoseconds after the chosen clock phase. When the

sampling circuit stores its choi ce (2-3 cycles of the clock), normal data sensing can

begin. At this point the individu al clocks from each of the TDCs can be disabled saving

power and potential clock noise pickup at the mezzanine card.

In normal operation the start b it is sensed, 32 data bits are assembled into a shift register,

parity is tested, and a stop bit is demanded for each TDC data word. Parity is tested and

an error bit is saved for reportin g to the MROD. At this time the data ready flag is set

indicating that the 32-bit output register plus a parity flag contains data. This register

cannot change more than once ev ery 36 bit times since the data stream contains a start

bit, 32 data bits, parity bit, and 2 stop bits.

Input Acc umulation

The 32- bit data words arrive from each TDC based on the data present in that pa rticular

TDC and the av ailability of its output sequencer. Therefore, although all TDCs send

data for a given EVID before pro cessing the next, data from a given TDC does not have

a well defined timing relationshi p to data for the same EVID from other TDCs. The

CSM is designed to be a simple tim e division multiplexer and is not responsible for

event building. Thus, it can simp ly transmit the assembled 32-bit words to the MROD in

the next available time slot for th e particular TDC. Since the data arrives at a maximum

Tubes

Hedgehog

Mezzanine

TTCrx

MROD

100 Mbyte/s

Gigabit

TTC

TTC Fibre

Clk, L1A, Calib

Mux

JTAG

Monitor

ELMB

Mux

18 Mezzanine

Cards Maximum

64 ADCs

ELMB

CAN

Bus

ROB

TTCvi

Central

Control

CSM

Opto-isolated copper JTAG

Serial to Parallel

Xtmr

Ethernet

Not in CSM-1

Design Specifications

CSM-1 Design Development at UoM 5

rate of 1.143 MBits /s from each TDC and 21 words are transmitted for each polling

cycle, any rate of serial transmiss ion on the fiber to the MROD greater than 97MBytes/s

will not require any buffering w ithin the CSM. This data rate can easily be handled by

the GOL sending 32-bit words at 25 MHz or greater.

Multipl exer

It is expected th at both header and trailer words will be enabled in the TDC. These

header and trailer words each contain the EVID. Since the CSM does no test beyond

parity checking, these words ar e processed the same as any other data words from the

TDC. Data words can be se nt to the MROD whenever the link is up and the currently

polled TDC has data availabl e. The link will be unavailable from a reset until resynchro-

nization is established. The CS M will send data only when the link is active and must

therefore, be resynchronized when ever the link goes down. The amount of data lost will

generally be significant since res ynchronization requires many clock cycles. The loss of

synchronization is sensed at th e MROD, were a reset will be initiated. This reset will

probably need to set the CSM and TDC to idle over JTAG, disable triggers being sent

from the CSM to the TDCs, resynchro nize the link, and reestablish triggers. Lost data

for the interval required to do th is resynchronization needs to be flagged at the MROD

where the BCID and EVID for trig gers is available. Information defining the last event

fully transmitted to the MROD is av ailable in the EVID/BCID values within the data.

Fibre Protocol

The output of the CSM is 32-bit dat a units, either TDC data words, empty TDC codes

for TDCs that have no data fo r the current polling cycle, spacer words, or link idle

codes. These words are to be sent to a fiber encoder/driver. The low power CERN GOL

chip can accept 32-bit words at rates up to 40MHz. This is well above the 25MHz

required to remain ahead of the dat a arriving from the TDCs. The higher the rate chosen

for the CSM polling and hence for t he GOL transmission, the more empty TDC words

that will be sent. A choice of th e 25MHz transmission clock, is a natural since this rate

can be generated with commercial os cillator components. Data will flow uninhibited

unless the link goes down and a ful l reset is initiated. The code in the CSM-1 is cur-

rently arranged to send no data to the GOL until the first trigger is seen and to not send

data for time division cycles (19 word groups) when no data is available from any TDC.

Data Word Formats

The data words from the TDC are des cribed in the AMT-1 document

and summarized

in Table 1 on page 6. Note that th e header and trailer words from the TDC provide the

EVID and the BCID words. The location of these bits permits a check of the data arrival

consistency as described abov e. The TDC data words also contain an error bit that indi-

cates when data has been missed. Th e data miss bit is to be interpreted as a flag that data

was lost between the time of the last data unit without a flag set for a given TDC and the

7. http://atlas.kek.jp/%7earaiy/am t1/index.html

Design Specifications

6 CSM-1 Design Development at UoM

data units with flags for that TDC. The CSM defines two additional IDs, one for the

TDC empty word and one for the sp acer word. The IDs for these words are tentatively

defined in Table 2 on page 6. In th e time division multiplexing scheme each TDC word

is sent in a time slot defined in t erms of the spacer word position. Following the spacer

word, TDC unit 0 data is sent or if no data is available, an TDC empty word is sent. The

next time slot contains TDC unit 1 data if any is available. This process is described in a

note by Thei Wijnen

on the NIKHEF web page. The TDC fi eld in Table 1 on page 6 is

replaced by the 4-bit field [Bit 27 - 24] composed of [TDC parity error, odd parity, 2 bits

unchanged]. Where TDC parity erro r = 1 if the parity of the received TDC word was not

consistent with the parity bit received and odd parity was not set to make the overall 32-

bit word odd parity.

A special set of formats exist for the CSM-1 when operating in event building mode. In

this mode the CSM-1 generates ou tput comparable to that of the CSM-0. These special

outputs are defined in Table 3 on page 7. There are CSM headers and trailers defined,

wire encoding for TDC 16 and 17, an d a set of error conditions that flag incomplete

events, events with one or more par ity errors in transmission from one or more TDC,

TABLE 1. TDC Data Word IDs and con tents

ID Word Cont ents Depends on ID Type

31-28

9876543210

1010

TDC EVID BCID

1100

TDC EVID Word Count

0010

TDC Mask Flags

0011

TDC Channel T E Coarse Time Fine Ti me

0100

TDC Channel Width Coarse Time Fine Time

0110

TDC Unused Errors

0111

TDC 0000 BCID

0111

TDC 0001 R L1 Occupancy

TABLE 2. CSM IDs for TDC empty (0 000) and Spacer word (1001)

ID Word Contents Depends on ID T ype

31-28

9876543210

0000

0100 0

1101

0000 0

8. http://...lookup Thei Wijnen no te on the data stream for TDM

Design Specificati ons

CSM-1 Design Development at UoM 7

and the conditions that caused the error. The wire encoding used in the CSM-0 is also

used here. See th e CSM-0 Users manual

for details of this enco ding.

For the CSM-1 event builder, the error words beginning with ID and subID 0x0101 and

0x0 the meanings of the remaining b its are given in Table 4 on page 7 below.

TABLE 3. Special CSM-1 Output Wo rd Formats in Event Builder Mode

ID Word Contents Depends on ID T ype

31-28

9876543210

CSM-1 event builder output format s defined to date

0101 100 1

EVID BCID

0101 1011

EVID Word Count

0101 1101

a. This subID flags an abnormal ev ent termination which is preceeded by the error word

with subID zero.

EVID Word Count

0101 0000

Error Code Abnormal TDC

0101 0010

b. This CSM sub-type is a pad word, and appears only when event buffers are padded to a

multiple of 64(256) words. These fo llow the CSM trailer (sub-type 1101 or 1011), and

the Word Count does not include the m in the total.

EVID Word Count

1010

TDC EVID BCID

1100

TDC EVID Word Count

0011

Wire Number T E Coarse Time Fine Ti me

0100

Wire Number Width Coarse Time Fine Time

9. http://atla s.physics.lsa.umich.edu/docushare/dscgi/ds.py/View/Collection-207

TABLE 4. Flag Bits for CSM-1 Ev ent Builder Errors

Error or Flag Bit Posn

Abort was due to missing header/tra iler (hdr_trl_abort)

Abort was due to missin g header (miss_hdr_abort)

Abort was due to dpram over threshold

Abort was due to a timeout wa iting for data

Flag -- BCID matching is disabl ed

Flag -- EVID matching is disable

Flags for AMTs with both good an d complete data

17-0

The CSM Module Specifications

8 CSM-1 Design Development at UoM

T HE CSM M ODULE S PECIFICATIONS

Component Details

The TTCrx

The TTCrx has the following functio ns:

• The TTCrx receives fibre data from the central timing and control logic, regenerates

the 40MHz LHC clock, receives th e commands, and presents these commands

including the LVL1 trigger to exter nal logic. It maintains counters for EVID and

BCID that are presented to ex ternal pins along with the LVL1 accept and other com-

mands.

• It provides for independent phas e adjustments of two copies of the 40MHz clock.

• It provides for calibration puls es coordinated with LVL1 triggers appropriately

delayed from the calibration timin g.

The timing and calibra tion role of the TTC system is illustrated in Figure on page 9. In

this figure, the role of the TTCrx for timing and calibration pulsing is shown. In addi-

tion, the role of the JTAG in es tablishing the parameters for the test calibration within

the CSM and mezzanine cards is illu strated. The TTCrx provides two clock outputs

each with its own phase adjust ment down to the 104ps least count. One of these phases

will be adjusted for gating of comm ands from the TTCrx including the LVL1 accepts

and resets. The second phase will b e used to initiate the calibration pulse, permitting

adjustable timing to the TDCs in 104ps steps.

The CSM-1 acts in response to a subset of both long and short format TTC commands.

The only short format command to which it responds, besides the EVID and BCID

resets, is that which initiates a calibration trigger sequence for the attached mezzanine

boards. This command has user bits 3 and 6 set on, and all other bits set off or ignored

(010010xx). The two least signif icant bits are ignored as they correspond to the EVID

and BCID clear strobes. A docum ent specifically concerned with calibration

is avail-

able.

10.CSM-1 Calibration Triggers, http ://atlas.physics.lsa.umich.edu/docushare/dscgi/ds.py/View/

Collection-214

The CSM Module Specifications

CSM-1 Design Development at UoM 9

Three Individual ly Addressed Commands (IAC), or long format commands, initiate

CSM-1 activity. These are detailed in Table 5 on page 9 below, and are discussed at

length in Reference on page 8.

The three commands detailed in Table 5 on page 9 are “external” TTC commands, i.e.,

they are relayed to the circ uitry attached to the external busses and lines of the TTCrx.

Each TTCrx also responds to seve ral “internal” commands, sometimes with detectable

external actions. In particular, th e CSM-1 monitors for the response to the internal

ERDUMP and CRDUMP commands, which result in the external presentation by the

TTCrx of the contents of 10 of its 20 8-bit registers. This content is latched by the CSM-

1 and included in the JTAG bit-st ream shifted out of the chip. See the section on JTAG

programming in this manual for more details.

available.The Ro le of the TTCrx and JTAG Systems for Calibration

The Serial to Parallel Units

In the CSM-0 th e serial to parallel conversion of data from the TDCs was done in three

small Xilinx FPGAs, each process ing 6 channels of TDC data. This same arrangement

could be implemented in the final C SM. However, the availability of more logic and

higher pin counts in newer FPGAs makes it possible to place all channels in a large chip

along with logic for the multiplexe r. The presence of DLL clock circuits with multiple

phase outputs further aids this pro cess by enabling incoming data sensing at the phase

appropriate channel by channel. Th e serial to parallel circuits implement the logic to:

• Autosense the individual TDC da ta streams and select the appropriate 1of 4 possible

clock phase to sample the arrivi ng data. Disable the TDC clock after sampling

through the JTAG programming of the AMT-3.

• Reset to a quies cent state expecting a logic true start bit as the first data element

from each TDC.

• After a start bit is received, begin assembly of the next 32-bits into a register.

TABLE 5. Individually Addressed Co mmands recognized by the CSM-1

SubAddress Data Bits Action

xxx Initiate TTC reset

Strobe Duration Set the duration of the calibration strobe to the

given number of 25ns clock tics

Strobe Duration Set the duration of the calibration strobe, as

above, and initiate the strobe itse lf.

TTCvi

Set

orbit

pulse

time

N ticks

TTCrx

CSM

ASD

Setup data

for pulse on

JTAG lines

delay

L1A

synchronized pulse

pulse

JTAGsetup

Mk II

MROD

Crossing

L1A

to CSM

The CSM Module Specifications

10 CSM-1 Design Development at UoM

• Compute and test the parity of the 32-bit string, setting an error flag if a fau lt is seen.

• Transfer th e 32-bits to a holding register upon receipt of the stop bit, set a “data

ready” flag indicating data present, and return to the quiescent state.

• Reset the “data ready” flag upon receipt of an “acknowledge” signal from the poll-

ing multiplexer.

The JTAG Interface

The JTAG signals co nnect to the CSM at a programmable PROM, and then to the

FPGA, passing next to the GOL chip , and finally to the TTCrx. From there it returns to

the FPGA on user handled pins. The JTAG chain is distributed to the mezzanine cards

as defined by the enable bits for i ndividual TDCs. Connection of the JTAG bus to the

mezzanine cards is totally controll ed in the FPGA code and can be modified as required.

The Virtex-II series of Xilinx chip s used for the CSM implementation support numerous

I/O levels and standards which include LVDS and differential PECL. The chips pow-

erup and initialize to an active TA P (JTAG) control for downloading configuration data.

To provide external control of this bootstrap procedure, external circuitry must be able

to force this “first state” fo r configuration to occur. The external JTAG can be provided

by any commercial controller or by the DCS system. One means of forcing the “first

state” will be to cycle the power . The TTCrx system could also generate a global reset.

Careful evaluation of the startup o peration must be done to insure that no frozen states

exists other than through a true ha rdware device failure. This JTAG link will support the

following functions:

• Initialization of the FPGA code w ithin the Xilinx chips.

• Initialization of the parameters of the CSM and ASD/TDC cards

• Initialization of the parameters of the TTCrx.

• Contro lling the run/reset/resynchronization of the CSM, TDC, and ASD

The Polling Multiplexer

The polling multiplex er is actually two polling multiplexers, one that scans for data

ready flags from the 18 serial to parallel units and enters data found into 18 distinct

FIFOs and a second that scans for F IFO data and if found places it into the time division

sequence of the output optical tran smitter. The polling rate of the first multiplexer is

based on the 40MHz LHC clock and th e rate of the second is 25MHz as set to guarantee

to empty the FIFO faster than it can fill. The first multiplexer sequences through the

steps:

• Reset to polling address to TDC 0 and to idle mode (not active).

• If in active mode, examine the “data ready” flag for the currently addressed TDC.

• If data is available, send it to the FIFO for the addressed channel and reset the flag.

• Increment the TDC addressed and l oop.

The second multip lexer has 21 steps. Eighteen of these steps are associated with the 18

FIFOs that hold data from the individual TDC. One, 18, defines the state when the

The CSM Module Specifications

CSM-1 Design Development at UoM 11

spacer word is to be transmitted. The remaining two correspond to optical fiber idle

words. The sequence through the steps:

• Reset the polling to address 18 and to idle mode (not active).

• If the polling multiplexer addr ess is 18 output a TDM spacer word to the GOL chip.

• If the multiplexer address is 19 or 20 send a Gigabit Ethernet idle code.

• If the address is 0-17, test the FIFO for data in the addressed channel.

• If no data is present in the FIFO , send a TDC empty word.

• If data is available for FIFO (TDC) addressed, send an “acknowledge” to the FIFO

to declar e that the word has been taken and send the word to the GOL chip.

• Increment the polling address mod ulo 21 to advance to the next step.

The Fibre Sequencer

The GOL chip accepts 32-b it data words from the polling multiplexer and a 2-bit write

control that:

• Sends a Gigabit Ethernet idle, if the 2-bit control is 00. The 32-bit data is ignored.

• Sends the 32-bit data presen ted to the GOL, if the control is 01.

• Sends an Extend code, if the cont rol is 10.

• Sends a trans mit error code, if the control is 11.

The DCS Analog Monitor

The 64-channel analog monitor is a direct copy of the 64-channel ELMB multiplexer

temperatures of the mezzanine cards and of the CSM itself. It is powered by the CSM

3.3 volts via a switched capacitor regulator that outputs 5 volts for the unit. Connections

to the ELMB processor are optical isolated using the same components as used for the

ELMB multiplexer that monitors cham ber conditions. The location of the monitor on

the CSM is chosen to minimize th e interchange of signals between the CSM and the

DCS circuitry. Only 5 opto-isolated signals are needed to provide JTAG to the CSM. In

addition, a second set of 8 I/O lin es from the ELMB processor will be extended to the

CSM and opto-isolated. The extensi on provides for monitoring 64 differential signals or

128 connections. Of these 57 diff erential signals are anticipated, 3 from each mezzanine

card and 3 from the CSM. Figure 4 on page 12 illustrates the relationship of the DCS to

the CSM mounted portions of the DCS .

The CSM Module Specifications

12 CSM-1 Design Development at UoM

FIGURE 4. Relationship of the DCS elements on the CSM to the DCS System

CSM Interconnects

Figure 5 on page 12 shows the layou t of the CSM. An arrangement with all passive

components on a motherboard whos e role is to provide attachment to the 18 mezzanine

cards via their 40-pin connectors.

FIGURE 5. The Layout and Interco nnection of the CSM

Th ese signals must be routed to various subsections according to the list below.

• Mezzanine to CSM Moth erboard x 18 max

8 JTAG (L VDS), 8 Data/CLK (LVDS) = 16 lines

8 Analog PWR, 8 Digital PWR, 6 Sen se, 2 Calibration = lines total

CAN

signals

CAN Power

ELMB Power

T sensors

DCS Motherboard

Credit Card

B field meter

regulator

CAN

driver

CAN interface

+ micro

controllers

ELMB

SPI

64 Channel

ADC

64 Channel

Mux/ADC

CSM SPI

Digital I/O

CSM

Power

57 Analog

Channels

SPI

CSM

JTAG

Cross Plate

18 connectors (40 pin) Blue at nea r end , Red at far

TTC Receiver CSM Multiplexer

Gigibit Ethernet

ELMB Mux

140 Pin

Interconnect

ELMB JTAG & SPI

Note: For the small

chambers these headers

are not needed and board

can be shorter.

Note: ELMB Mux is copy of 64 channel DCS mux

to MROD

24 40 ?

The CSM-1 Prototype Development

CSM-1 Design Development at UoM 13

• CSM to CSM Motherboard - 3 x 140 pins = 420 pins

8 JTAG (LVDS), 10 Data/CLK (LVDS) x 18 mezzanine cards = 324 lines

6 Sense, 8 Digital PWR, 8 ELMB PWR = 22 lines

• ELMB Mux to CSM Motherboard

6 Sense x 18 Mezzan ine cards, 6 Sense x 1 CSM = 114 lines

5 JTAG I/O, 5 ELMB SPI, & 3 Spares = 13 lines

The Layout, as seen from above, is shown in Figure 6 on page 13. Note that for cham-

bers not requiring more than 12 TDCs, that the motherboard can be reduced in size to

the area required by the activ e components. This will entail the design of 2 different

motherboards but affords the op tion of a short unit for small chambers.

FIGURE 6. The CSM-1 and Motherboa rd.

T HE CSM-1 P ROTOTYPE D EVELOPMENT

Design Elements

The CSM-1 has as it goal the rapid design of a module more closely aligned with the

final CSM than is the CSM-0. It presumes an early version of the MROD and the use of

fiber connections to the trigger, timing, and control and to the data acquisition flow. The

fiber interconnect more realistical ly represents the clock noise, data noise, and ground-

ing environment of the final conf iguration. The only justification for the intermediate

design of a second prototype is exp ediency. The goal is to have a fiber connected design

that can reside on chamber usin g the same interconnect motherboard, TTC system, and

TTC optical input

GOL optical output

DCS connection for JTAG

& 8-bit I/O communication

Alternate JTAG

via Jumpers

5Volt power

18 Mezzanine I/Os

on 40-pin ribbons

2 rows (near seen)

9 on each side in

Mounting plate

Standoff

The CSM-1 Prototype Development

14 CSM-1 Design Development at UoM

th e opto-isolated JTAG connection anticipated in the final CSM. The items to be

included in the CSM but not present in the CSM-1, are the analog multiplexer for volt-

age and temperature sensing, an d the final TTC system with calibration programming.

The connections to do calib ration will be in place in the CSM resident FPGA, but per-

haps not from the TTCvi programmi ng. The TTC system is described on the CERN web

pages at CERN

where the TTCvi Mark I is desc ribed. The specifics of the CSM-1 are:

• The main multiplexing task of th e CSM-1 has been compressed from 4 FPGAs into

one using a scheme of clock phase s ampling to remove the need for many indepen-

dent clock networks, using a larg e fine pitch ball grid array chip, and using a LVDS

compatible FPGA from Xilinx, th e XC2V1000FPBG456. Prototype testing of the

optical fiber output to the MR OD will be done using the CERN GOL chip and the

Infineon V23818-K305-L57. The fina l design will likely use a simple laser diode in

place of the transceiver since the input channel is unused in the CSM to MROD link.

• The CSM-1 requires clocking an d event triggering. For the CSM-1 design the Trig-

ger, Timing, and Control will be ba sed on the CERN developed TTC system. Our

transition from the internal simu lation of the trigger and timing to the externally gen-

erated functions implies and integr ation of the CERN chip into the module and the

software integration of the TTCvi m odule into our environment. We plan to use the

Mark I version of the TTCvi in itially and switch to the Mark II version in 2003.

• The initialization of the ASD/TDC mezzanine cards is done via JTAG protocol

which in the case of the CSM-0 is b uilt into the VME functions of the CSM-0 mod-

ule. In the CSM-1 no VME connectio ns is available. The CSM-0 will use an opto-

isolated JTAG function as will th e final CSM. However, the final CSM will have a

DCS connection and a CAN bus interf ace to provide the programming for the JTAG.

To make progress toward this final goal, the opto-isolated JTAG will be driven from

a commercial module.

• The CSM will interconnect to th e ASD/TDC modules via shielded ribbon cables that

leave the Faraday cages and converg e at the spacer plate of the MDT. At this loca-

tion they will attach to a passiv e interconnect on which the active CSM will reside.

To make use of the current CSM-0 wi th the new interconnect board, a MiniAdapter

board has been constructed with th e same circuit design as the adapter currently in

use with the CSM-0, but design ed to attach to the passive interconnect. Following

use with the CSM-0, a simple repl acement of this new adapter with the CSM-1 will

convert a chamber from CSM-0 readou t to the MROD readout via the CSM-1.

• The CSM-1 will not contain the an alog multiplexer nor will it interact with the DCS

system initially.

• The CSM-1 will not contain any Single Bit Upset detection or correction code. Since

this code is firmware, it is an ticipated that work on this code will follow the certifi-

cation of the design in the first half of 2003.

11.TTC system web pages, http://ttc.web.cern.ch/TTC/intro.html

JTAG Programming

CSM-1 Design Development at UoM 15

JTAG P ROGRAMMING

CSM-1 JTAG

The CSM-1 has a JTAG string which is a composite of several bit strings for the TTCrx

the TTC RO status, and the CSM modu le RO status. These strings are defined in the

CSM VerilogHDL code and can be modi fied rather easily. The current sizes of the

JTAG elements are represented in Ta ble 6 on page 15 along with the overall order of

these strings in the JTAG sequence.

The TTCrx string is defi ned in the TTCrx manual

. In the CSM, this string is loaded

via JTAG and pr esented to the TTCrx chip following a reset signal. This reset initiates

the timing sequence for loading the TTCrx from PROM with the FPGA taking the role

of the PROM. The CSM parameter stri ng of 72 bits provides RW control for the CSM,

the GOL, and TTCrx as shown in Tabl e 7 on page 15. The CSM status bits are RO and

represent the version number of the CSM code, the state of various DLL lock lines, and

a few error flags.

TABLE 6. JTAG String Lengths & Po sition

String Name # bits Bit posn

CSM_pa rm_len

72 206

TTCrx_init_len

80 278

CSM_status_len

28 0

AMT_parity_err_len

18 28

TTC_readback_len

160 46

12.Timing receiver ASIC (TTCrx) Re ference Manual, http://ttc.web.cern.ch/TTC/intro.html

TABLE 7. CSM JTAG Control Bit Definitions

Field Bit posn

TDC enable lo w bit

TDC enable length

Include mezzanine cards in JTAG fla g

CSM enable trigger bit

Mezzanine command delay pipe length low bit

Mezzanine command delay pipe length , length

Spare bits low bit

Spare bits length

GOL differential input selection flag

GOL ld0 bit

GOL ld1 bit

JTAG Programming

16 CSM-1 Design Development at UoM

The TTCrx readb ack bits hold the contents of the 20 I

C addressed 8-bit registers of the

TTCrx. Two fpga build ver sions are possible. In the default version, one I

C register is

read every 400ms, covering all 20 registers. In the second version, only the first ten reg-

isters are available. These 10 regi sters are accessed passively via the ERDump and

CRDump commands of the TTC rx. (See Table 8 on page 16) The CSM status register

GOL pll bit

GOL laser bit

GOL negative edge selection bit

GOL mode selection bit

GOL tdi bit

GOL address low bit

GOL address length

TTC tdi bit

TTC use prom flag bit

CSM next state low bit

CSM next state length

CSM take command action

CSM BCID no match bit (event builde r only)

CSM EVID no match bit (event build er only)

CSM send all AMT types bit (event builder only)

CSM no header or trailers bit (e vent builder only)

CSM disable TTC and GOL JTAG bit

Spare bit 1

Spare bit 2

Spare bit 3

Spare bit 4

Spare bit 5

Spare bit 6

Spare bit 7

Spare bit 8

Spare bit 9

Spare bit 10

TABLE 8. TTCrx Readback Register

Field bit posn

TTCrx registers 9 to 0

TTCrx registers 19 to 10 ( I

C access only)

TABLE 7. CSM JTAG Control Bit Definitions

Field Bit posn

JTAG Programming

CSM-1 Design Development at UoM 17

bits reflect th is difference, with the first column describing the ERDump/CRDump situ-

ation, and the second describing the I

C situation. (See Table 10 on page 17) The

ERDump/CRDump vers ion will eventually be phased out.

The AMT parity error bits are clear ed when data acquisition is enabled and accumulated

from the parity error calculation for each data word separately for each of the AMT

chips (each mezzanine card). A stop and restart of the data acquisition begins another

set of 18 accumulated OR bits.

The CSM status register b it definitions are indicated in Table 10 on page 17 using the

same convention as provided in th e parameters file. In this case the bits are RO with the

data sent via the JTAG output strea m being ignored.

TABL E 9.AMT Parity Error Flags

Field bit po sn

TDC 0 Flag

TDC 1 Flag

TDC 2 Flag

TDC 3 Flag

TDC 4 Flag

TDC 5 Flag

TDC 6 Flag

TDC 7 Flag

TDC 8 Flag

TDC 9 Flag

TDC 10 Flag

TDC 11 Flag

TDC 12 Flag

TDC 13 Flag

TDC 14 Flag

TDC 15 Flag

TDC 16 Flag

TDC 17 Flag

TABLE 10. CSM Status Register Bit Definitions

Field b it posn

CSM version nu mber low bit CSM version number low bit

GOL ready bit GOL ready bit

TTC ready bit TTC ready bit

LHC clock locked bit LHC clock lock ed bit

XMT internal clock lock bit (FPGA) XMT internal clock lock bit (FPGA)

XMT external clock lock bit (GOL) X MT external clock lock bit (GOL)

JTAG Programming

18 CSM-1 Design Development at UoM

CSM states

The CSM-1 operates in st ates defined currently according to Table 11 on page 18. Most

of the states have to do with initi alization. Once running the CSM-1 requires no inter-

vention as long as synchronization is maintained and errors are rare. Since the CSM-1

continues to be under development, one should expect the number of states, parameters,

and status bits to be fluid. Check regularly for updates of this manual by examining the

Docushare pages for new versions.

Unused OR (for fooling sy nthesizer) Unused OR (for fooling synthesizer)

TTC prom load error flag TTC prom load error flag

CSM state low bit CSM state lo w bit

Sample phase error bit Sample phas e error bit

C Operation Failure I

C Operation Failure

Spare = 0 TTC I

C compare error

TTC dump compare error bit Spa re = 0

CSM error bit CSM error bit

TABLE 10. CSM Status Register Bit Definitions

Field b it posn

TABLE 11. CSM-1 States

State Code

IDLE_STATE 0

FPGA_RESET_STATE 1

RESET_TTC 2

RESET_GOL 3

WAIT_LOCK 4

DCM_RESET_CLK25 5

SAMPLE_AMT_PHASE 6

JTAG_RESET 7

TTC_LOAD 8

AMT_RESET 9

WAIT_PROM 10

TRY_RESET_ERROR 11

JTAG Programming

CSM-1 Design Development at UoM 19

The CSM-1

Operational State

Machine

The states detailed in Table 11 on page 18 can be individually entered, but a few are

really only useful when automati cally entered as a result of selecting some other opera-

tion.

This section attempts to document the actions taken as a result of selecting the various

states. Following each state name i s the decimal value of the enumerated type to which

it corresponds, a'la Table 11 on pa ge 18.

IDLE_STATE (0): Th is is the normal state in which the FPGA operates. All other states

eventually flow back to this one fo llowing the completion of any actions taken in those

other states. Issued JTAG Operatio nal State changes will only be made if this is the cur-

rent state when the JTAG command is received.

RESET_TTC (2): Entry to this state toggles the Reset_b pin of the TTCrx chip for

approximately 400ns. Exit from this state at the end of the reset period depends upon the

setting TTC_use_prom_bit of th e CSM parameters JTAG register. If the TTC should

use the FPGA as its load prom, th en that is the next action taken (state WAIT_PROM).

Otherwise, reset of the DCM co re controlling the re-generation of the 25ns LHC clock

within the FPGA is undertaken (stat e DCM_RESET_CLK25).

WAIT_P ROM (10): Wait while the TTCrx chip uses the CSM-1 FPGA as its load prom.

Exit from this st ate is to the DCM_RESET_CLK25 state.

DCM_RESET_CLK25 (5): If the TTCrx is reset, then the DCM core within the CSM-1

FPGA must be reset to restart th e local clocks. Exit from this state is normally to the

IDLE_STATE (see the Power Up Sequ ence detail below).

RESET_ GOL (3): Entry to this state toggles the reset_b pin of the GOL chip for appro x-

imately 400ns. Exit fr om this state is to the WAIT_LOCK state.

WAIT_LOCK (4): Wait while the GOL chip re-synchronizes and re-locks itself. This

wait is approximately 800ms long . Exit from this state is to the IDLE_STATE.

SAMPLE_AMT_PHASE (6): Entry to this state initiates a sampling of the phase of the

data and strobe signals returnin g from each mezzanine board. At the end of the sample

period, the CSM-1 clock phase most appropriate for acquiring the serial bit stream from

each active mezzanine board will be known and selected for use. Exit from this state is

to the IDLE_STATE.

--> IMPORTANT NOTE: The FPGA “assum es” the serial clock is already JTAG

enabled fro m each active mezzanine board. If it is not, the CSM_error_bit of Table 9

will be set due to the fa iled phase sampling.

JTAG_ RESET (7): Toggle the JTAGTRST_b signal line to the TTCrx and GOL chips

for approximately 350ns. Exit from this state is to the IDLE_STATE. NOTE: The JTAG

instruction register of both the GO L and the TTCrx is loaded with the IDCODE instruc-

tion as a side-effect of this opera tion.

JTAG Programming

20 CSM-1 Design Development at UoM

TTC_LOAD (8 ): Force a TTC PROM load from the FPGA independent of the setting of

the “TTC use prom flag” JTAG bit. O ther operations are identical to those of the

RESET_TTC state operating in a non -power-up sequence.

AMT_RE SET (9): Toggle the mezzanine board AMT reset line for approximately

800ns. Exit from this state is to the IDLE_STATE.

TRY_RE SET_ERROR (11): During the course of operations within the CSM-1 state

machine, an error could occur. This results in setting the “CSM-1 Error” bit. Entry to the

TRY_RESET_ERROR state results in an attempt to determine and correct the cause of

the error. If the attempt succeed s the bit will be cleared. Exit from this state is to the

state appropriate to correcting th e error. If more than one error exists, it could take two

or three entries to this state befo re they are all handled.

FPGA_RESET_STATE (1): Entry to this state re-initializes the CSM-1 FPGA to its

power up state. Exit fr om this state is through the RESET_TTC and RESET_GOL states

as detailed below for the Power- Up Sequence of events.

All other values for the CSM-1 State are ignored, and treated as if they were the

IDLE_STA TE.

Note that a JTAG re quest to enter a state will be ignored if the state machine is not

in an IDLE_STATE when the JT AG register is updated. No error will be reported i n

this situation.

Power Up Sequence Following a bo ard power up, we recommend the following sequence of states and oper-

ations before the FPGA is used f or data acquisition. The first two operations are auto-

matically performed as a result of the FPGA initialization sequence.

1. RESET_TTC. Note that all TTCr x initialized during power up will have the same

address, which is hard-coded in the CSM-1 initial programming strings. If individual

TTCrx addresses are desira ble, then the correct addresses should be set in the JTAG

strings of each CSM-1, in conj unction with a RESET_TTC command.

2. RESET_GOL

3. Enable the connected set of mezzanine boards, and enable the CSM-1 to JTAG pro-

gram them. The IDLE_STATE should be selected while doing this, or it can be done at

the same time as the RESET_GOL st ate is selected.

4. JT AG program the mezzanine boards, selecting the IDLE_STATE in the process. A

continuous return clock should be selected.

5. SA MPLE_AMT_PHASE

6. Disable the mezzanine boards AMT clock return to the CSM-1 (IDLE_STATE).

7. Disable mezzanine board JTAG op erations, and enable CSM-1 acquire data JTAG bit

(IDLE_STATE).