1

LNL

SADIRC2000SADIRC2000

Resoconto 2000 eResoconto 2000 eRichieste LNL per il 2001Richieste LNL per il 2001

L. BertiL. Berti 30% 30%M. Biasotto 100%M. Biasotto 100%M. GulminiM. Gulmini 50% 50%G. MaronG. Maron 50% 50%N. TonioloN. Toniolo 30% 30%

Le percentuali sono condivise con INFN-GRIDLe percentuali sono condivise con INFN-GRID

2

LNL

40 MHz

105 Hz

102 Hz

100 Tbyte/s

100 Gbyte/s

100 Mbyte/s

Level 1Trigger

Event Manager

Detector Frontend

Event Builder

Computing Services

Controls

Readout

Filter

Collision rate 40 MHzLevel-1 Maximum trigger rate 100 kHzAverage event size 1 MbyteNo. of In-Out units (200-5000 byte/event) 1000Event builder (512-512 switch) bandwidth 500 Gbit/sEvent filter computing power 5 106 MIPSData production Tbyte/dayNo. of readout crates 250No. of electronic boards 10000

Collision rate 40 MHzLevel-1 Maximum trigger rate 100 kHzAverage event size 1 MbyteNo. of In-Out units (200-5000 byte/event) 1000Event builder (512-512 switch) bandwidth 500 Gbit/sEvent filter computing power 5 106 MIPSData production Tbyte/dayNo. of readout crates 250No. of electronic boards 10000

Units

Units

Future DAQ SystemsFuture DAQ Systems

3

LNLSadirc2000 Event Builder DemonstratorsSadirc2000 Event Builder Demonstrators

• Square Event Builder (15x15): Square Event Builder (15x15): June 2000 (June 2000 (OKOK))

• Asymmetric Event Builder (4x40):Asymmetric Event Builder (4x40): November 2000November 2000

4

LNLHardware components for the Square EBHardware components for the Square EB

SysKonnect SK-9821 1000BaseT adapter

Foundrynet FastIron II+ 1000BaseT switch Supermicro PIIIDMEmotherboard

Intel 840 chipset

Pentium III 600 Mhz

64/66 PCI BUS

5

LNLSoftware components for the Square EBSoftware components for the Square EB

• vxWorks real time o/s

• C++ software for emulation of the Event Builder components ( EVM, RU, BU )

• Standard vxWorks driver for the SysKonnect adapter

6

LNLError recoveryError recovery

• Error recovery procedure implemented

• See: Samim Erhan, Wolfgang Schleifer, Nick Sinanis ‘System Error Analysis’ TriDAS Review 11 May 2000

7

LNLError recoveryError recovery

BU

send

RUsend timer

start

cache

send( retry )

start

cancel

timeout

cache

BU

allocate

EVMallocate timer

start

confirm

allocate start

cancel

timeout

confirm

BU – EVM communication BU – RU communication

8

LNLSquare Event builder layoutSquare Event builder layout

• RUs and BUs distributed in all the switch slots

• Part of the traffic localized in the slot

• Minimize the switch backplane utilization

1 15141312111098765432

1 15141312111098765432

EVM

RUs

BUs

Slot 1 Slot 2 Slot 3 Slot 4

9

LNLSwitch packets loss in special conditionSwitch packets loss in special condition

First packet forwarded correctly

Some packets lost

( 10-30 packets )

All the following packetsforwarded correctly

• The switch lose some packets in particular conditions

• When the transmission restart after a pause >60 Secs

• Probably related to the MAC address table management

• Problem solved by the Error recovery procedure

10

LNLTest conditionsTest conditions

• BDN+BCN in the same network• No Readout commands from EVM to RU• Clear/Allocate optimization: the clear

command is sent together an allocate for a new event

• No command or event aggregation: each packet transport a command or a data frame relative to a single event

11

LNLTest conditionsTest conditions

• Full data transfer: data moved from/to the pc memory

• Tests performed in the 400-4000 Bytes range

12

LNLThroughput vs number of allocated eventsThroughput vs number of allocated events

15 x 15

13

LNLThroughput vs event sizeThroughput vs event size

15 x 1

1 x 15

15 x 15

14

LNLEvent rate vs event sizeEvent rate vs event size

15 x 1

1 x 15

15 x 15

15

LNLBU balancingBU balancing

15 x 15

16

LNLEvent builder scalingEvent builder scaling

15 x 15

17

LNLAggregate throughputAggregate throughput

15 x 15

18

LNLConclusions for the Square Event BuilderConclusions for the Square Event Builder

• Event Builder Demonstrator 15x15 based on copper Gigabit Ethernet interfaces tested

• Good performances and scalability

• Actual limit is in the Readout Unit (?)

• Error recovery work fine, systematic tests need

19

LNLNNext Steps for 2000ext Steps for 2000

• Asymmetric Event Builder (4x40)

• Event Builder Protocol Simulations

20

LNLNext Steps for 2001: InfiniBand (I)Next Steps for 2001: InfiniBand (I)

TODAYTODAY

21

LNLInfiniBand (II)InfiniBand (II)

Legacy host architectureLegacy host architecture

The Infiniband ModelThe Infiniband Model

22

LNLInfiniBand (III)InfiniBand (III)

• What?What?– Initial single link signaling rate of 2.5Gbaud Initial single link signaling rate of 2.5Gbaud

• Means unidirectional transfer rate of 250MB/sec with a Means unidirectional transfer rate of 250MB/sec with a theoretical full duplex rate of 500MB/sectheoretical full duplex rate of 500MB/sec

– Initial support for single, 4, and 12 wide link widthsInitial support for single, 4, and 12 wide link widths

– Point to point switched fabricPoint to point switched fabric

– Message based with multicasting supportMessage based with multicasting support

MultiMultiStageStageSwitchSwitch

LinksLinksTCATCA

I/O ControllerI/O Controller

MemoryMemoryControllerController

PCI-X PCI-X HostHost

BridgesBridges

CPUCPU CPUCPU CPUCPU CPUCPU

HCAHCALinkLink

Fibre ChannelFibre Channel

SCSISCSI

Gig. EthernetGig. Ethernet

MemoryMemory

HCAHCA- Host Channel Adapter- Host Channel AdapterTCATCA - Target Channel Adapter - Target Channel Adapter

23

LNL SADIRC Requests for 2001SADIRC Requests for 2001

• Simple test system (4 servers + Storage Area Network + Simple test system (4 servers + Storage Area Network + Network) forNetwork) for 20012001 is possibleis possible

• Early access to the productsEarly access to the products

• Test BedsTest Beds

• Requests 2001 (valuations) Requests 2001 (valuations) • 4 servers4 servers 50 Ml50 Ml• 1 IBA Switch1 IBA Switch 20 Ml20 Ml• IBA AdaptersIBA Adapters 10 Ml10 Ml

24

LNL SADIRC2000 Global RequestsSADIRC2000 Global Requests

AAnnnnoo IInnvveennttaarriioo CCoonnssuummoo MMiissss.. IInntteerrnnoo MMiissss.. EEsstteerroo

22000011 8800 MMLL 88 MMll IINNFFNN--GGRRIIDD IINNFFNN--GGRRIIDD