G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 11

LNL

INFN-GRID-WP2.4:INFN-GRID-WP2.4:Computing Fabric Computing Fabric & Mass Storage& Mass Storage

LNLLNLL. BertiL. BertiM. BiasottoM. BiasottoM. GulminiM. GulminiG. MaronG. MaronN. TonioloN. Toniolo

PadovaPadovaS. BalsamoS. BalsamoM. BellatoM. BellatoF. CostaF. CostaR. FerrariR. FerrariM. MichelottoM. MichelottoI. SaccarolaI. SaccarolaS. VenturaS. Ventura

CNAFCNAFA. ChiericiA. ChiericiL. Dell’AgnelloL. Dell’AgnelloF. GiacominiF. GiacominiP. MatteuzziP. MatteuzziC. VistoliC. VistoliS. ZaniS. Zani

LecceLecceG. AloisioG. Aloisio M. CafaroM. Cafaro Z. ZzzzZ. Zzzz L. DepaolisL. Depaolis S. Campeggio S. Campeggio E. FasanelliE. Fasanelli

TorinoTorinoA. ForteA. Forte

GenovaGenovaG. ChiolaG. ChiolaG. CiaccioG. Ciaccio

Roma 1Roma 1D. AnzellottiD. AnzellottiC. BattistaC. BattistaM. De RossiM. De RossiF. MarzanoF. MarzanoS. FalcianoS. FalcianoA. SpanuA. SpanuE. ValenteE. Valente

BolognaBolognaG.P. SiroliG.P. SiroliP. MazzantiP. Mazzanti

CataniaCataniaC. RoccaC. RoccaE. CangianoE. CangianoTecnologoTecnologo

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 22

LNLTerminologyTerminology

PCPC

LANLAN

PC network PC network + midleware+ midleware

= PC Farm/Fabric= PC Farm/Fabric PC ClusterPC Cluster

LALANN

LALANN

LALANN

LALANN

LALANN

LALANN

PC Farm network PC Farm network + midleware+ midleware

= = GRIDGRID

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 33

LNLWhy this WP?Why this WP?

Commodity components like PCs and LANs are mature to form Commodity components like PCs and LANs are mature to form inexpensive and powerful Computing Fabricinexpensive and powerful Computing Fabric

Computing Fabrics located in different sites are integrating to Computing Fabrics located in different sites are integrating to form a Computational/Data GRIDform a Computational/Data GRID

ButBut

– How to design a fabric of 1000s nodes balancing computing How to design a fabric of 1000s nodes balancing computing power and efficient storage access?power and efficient storage access?

– How to control and monitor the basic system components?How to control and monitor the basic system components?

– How to “publish” the monitored values to the GRID monitor How to “publish” the monitored values to the GRID monitor system?system?

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 44

LNLComputing Fabric WPComputing Fabric WP

This WP wants to address the mentioned problems adding a technological This WP wants to address the mentioned problems adding a technological tracking task to follow and to test, with real use case, the evolution of the basic tracking task to follow and to test, with real use case, the evolution of the basic constituents of a fabric.constituents of a fabric.

WP Break DownWP Break Down– Fabric Design (INFNGrid Wp 2.4)Fabric Design (INFNGrid Wp 2.4)

Overall architecture and fabric setupOverall architecture and fabric setup LAN and SAN (System Area Network) technologiesLAN and SAN (System Area Network) technologies Communication protocols for high speed network fabricsCommunication protocols for high speed network fabrics Storage Systems Storage Systems Microprocessor TechnologyMicroprocessor Technology

– Fabric Management (DataGrid WP4)Fabric Management (DataGrid WP4) Configuration management and automatic software installationConfiguration management and automatic software installation System monitoringSystem monitoring Dynamic System PartitionDynamic System Partition Problem managementProblem management

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 55

LNLFabric Design - INFNGrid WP 2.4 - InstitutionsFabric Design - INFNGrid WP 2.4 - Institutions

Dipartimento di Scienze dell’Informazione - Universita’ di Dipartimento di Scienze dell’Informazione - Universita’ di VeneziaVenezia

Dipartimento di Scienze dell’informazione - Universita’ di Dipartimento di Scienze dell’informazione - Universita’ di GenovaGenova

Ingegneria Informatica – Universita’ di LecceIngegneria Informatica – Universita’ di Lecce

8 sezioni INFN, 1 Laboratorio Nazionale8 sezioni INFN, 1 Laboratorio Nazionale

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 66

LNLFabric Management - DataGrid WP4 - InstitutionsFabric Management - DataGrid WP4 - Institutions

CERNCERN

Konrad Zuse Zentrum (Berlin)Konrad Zuse Zentrum (Berlin)

Kirchhoff Institute (Heidelberg)Kirchhoff Institute (Heidelberg)

IN2P3 (Lyon)IN2P3 (Lyon)

INFNINFN

NikhefNikhef

RALRAL

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 77

LNLFabric Design Detailed Program (2000-2001)Fabric Design Detailed Program (2000-2001)

Fabric ArchitectureFabric Architecture– Network topologiesNetwork topologies– Data Server connections and network file systemsData Server connections and network file systems– System break downSystem break down

Interconnection NetworksInterconnection Networks– 100/1000 Ethernet100/1000 Ethernet– MyrinetMyrinet– InfinibandInfiniband

Communication protocols for high speed network fabricsCommunication protocols for high speed network fabrics

Storage SystemsStorage Systems– Ultra SCSI (160/320/…)Ultra SCSI (160/320/…)– Ultra and Serial ATAUltra and Serial ATA– Storage Area Network (SAN)Storage Area Network (SAN)

Fibre Channel Fibre Channel Microprocessor TechnologyMicroprocessor Technology

– Dual slim processorsDual slim processors– IA64IA64

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 88

LNLFabric Design Deliverables Fabric Design Deliverables

DDeelliivveerraabbllee DDeessccrriippttiioonn DDeelliivveerryy DDaattee DD22..44..11 RReeqquuiirreemmeennttss ggaatthheerriinngg ffrroomm

ffaabbrriicc uusseerrss 33 MMoonntthhss

DD22..44..22 SSuurrvveeyy ooff eexxiissttiinngg ffaabbrriicc//cclluusstteerr aarrcchhiitteeccttuurreess,, aarrcchhiitteeccttuurraall iissssuueess ttoo ddeessiiggnn aa ffaabbrriicc ooff ccoommmmooddiittyy ppccss

66 ++ 66 MMoonntthhss

DD22..44..33 IINNFFNN ttoooollkkiitt ttoo ddeessiiggnn aanndd rreeaalliizzee llooccaall ffaabbrriiccss

1122 MMoonntthhss

DD22..44..44 IInntteeggrraattiioonn ooff tthhee DDaattaaGGrriidd ffaabbrriicc mmaannaaggeemmeenntt ssooffttwwaarree ttoooollss

2244 MMoonntthhss

DD22..44..55 IInntteeggrraattiioonn ooff tthhee FFaabbrriicc MMaannaaggeemmeenntt iinnttoo tthhee GGRRIIDD eennvviirroonnmmeenntt

3366 MMoonntthhss

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 99

LNLFabric Design MilestoneFabric Design Milestone

MMiilleessttoonnee DDeessccrriippttiioonn SSIITTEE SSttaarrtt DDaattee

EEnndd DDaattee

MM22..44..11 NNFFSS ccoonnnneeccttiioonn ttooppoollooggyy:: ddiisskk sseerrvveerr bbaasseedd

LLNNLL 99//22000000 33//22000011

MM22..44..22 NNFFSS ccoonnnneeccttiioonn ttooppoollooggyy:: ddiissttrriibbuutteedd ddiisskkss

CCNNAAFF//BBOO 99//22000000 33//22000011

MM22..44..33 MMiiccrroopprroocceessssoorr TTeecchhnnoollooggyy:: dduuaall sslliimm pprroocceessssoorrss

CCNNAAFF//BBOO 99//22000000 33//22000011

MM22..44..44 SSttoorraaggee SSyysstteemmss:: FFiibbrree CChhaannnneell -- SSCCSSII UUllttrraa 116600//332200

PPAADDOOVVAA 99//22000000 33//22000011

MM22..44..55 SSttoorraaggee SSyysstteemmss:: SSeerriiaall AATTAA PPAADDOOVVAA 55//22000011 77//22000011 MM22..44..66 MMiiccrroopprroocceessssoorr TTeecchhnnoollooggyy:: IIAA6644 PPAADDOOVVAA 1111//22000000 33//22000011 MM22..44..77 IInntteerrccoonnnneeccttiioonn NNeettwwoorrkkss:: MMyyrriinneett

++ IIAA6644 LLEECCCCEE 99//22000000 1122//22000011

MM22..44..88 IInntteerrccoonnnneeccttiioonn NNeettwwoorrkkss:: GGiiggaabbiitt EEtthheerrnneett aanndd eeffffiicciieenntt ccoommmmuunniiccaattiioonn pprroottooccoollss

GGEENNOOVVAA//RROOMMAA11

99//22000000 55//22000011

MM22..44..99 SSeettuupp ooff aa cclluusstteerr ooff mmoorree tthhaann 110000 nnooddeess

LLNNLL 11//22000011 1122//22000011

MM22..44..1100 IInntteerrccoonnnneeccttiioonn NNeettwwoorrkkss:: IInnffiinniibbaanndd

LLNNLL 55//22000011 1122//22000011

MM22..44..1111 IInnssttaallllaattiioonn ooff tthhee IINNFFNN ffaabbrriiccss AAllll 22000022 22000033 MM22..44..1122 TTIIEERR 00 IInntteeggrraattiioonn AAllll 22000033 22000033

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1010

LNL DATA GRID Fabric Management WP4DATA GRID Fabric Management WP4

Automatic Software Installation and Management (OS and Automatic Software Installation and Management (OS and Applications)Applications)

Configuration ManagementConfiguration Management

System MonitoringSystem Monitoring

Problem ManagementProblem Management

Local Authorization Services and Grid IntegrationLocal Authorization Services and Grid Integration

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1111

LNLWP4 Fabric Managemet DeliverablesWP4 Fabric Managemet Deliverables

Requirements document and survey of existing tools and Requirements document and survey of existing tools and technologies (month 6)technologies (month 6)

A configuration and installation management demonstrated to A configuration and installation management demonstrated to work on a cluster of more than 100 nodes (month 12)work on a cluster of more than 100 nodes (month 12)

A fully deployed service level monitoring system for a computer A fully deployed service level monitoring system for a computer centre. Hooks to provide remote requests for meta information centre. Hooks to provide remote requests for meta information like policies and quality measures, to allow schedule decisions like policies and quality measures, to allow schedule decisions (month 24)(month 24)

A fully integrated system to accept remote resource requests in A fully integrated system to accept remote resource requests in the form of tape mounts r jobs to run and provide monitoring the form of tape mounts r jobs to run and provide monitoring information about progress og requests, and final accounting information about progress og requests, and final accounting report back to the sender (month 36)report back to the sender (month 36)

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1212

LNLNFS Disk Servers Based Farm (Legnaro)NFS Disk Servers Based Farm (Legnaro)

Disk Server:Disk Server:-Dual PIII 800 MHzDual PIII 800 MHz- Mem 512 MBMem 512 MB- SCSI AdaptersSCSI Adapters

Ethernet Switch:Ethernet Switch:- 10 FE10 FE- 1 Uplink GE1 Uplink GE

Computational Nodes:Computational Nodes:- Dual PIII 800 MHz (40+40 SI95)Dual PIII 800 MHz (40+40 SI95)-Mem 512 MBMem 512 MB

N1N1SSWWIITTCCHH

N10N10

GEGE

FEFE

N2N2

Farm ModuleFarm Module

LNL is testing this farm module (with PIII at 450 MHz)LNL is testing this farm module (with PIII at 450 MHz)

DISKDISKSERVERSERVER

This Farm has been fundedThis Farm has been fundedBy Comm Calcolo for LNLBy Comm Calcolo for LNLOff-line computation (gr. 2/3)Off-line computation (gr. 2/3)

Requests for 2000:Requests for 2000:- 2 Raid SCSI Controllers - 2 Raid SCSI Controllers 10 Ml10 Ml

ANISANISBootBoot

ServerServer

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1313

LNLLNL PC Farm (phase I – May 200)LNL PC Farm (phase I – May 200)

N1N1SSWWIITTCCHH

N10N10

GEGE

FEFE

N2N2

Farm Module IFarm Module I

DISKDISKSERVERSERVER

FrontEndFrontEnd(ANIS)(ANIS)

ASUS P3BFASUS P3BF440 BX chipset440 BX chipset1 PIII 450 MHz1 PIII 450 MHzPCI 33/32PCI 33/32

SuperMicro PIIIDMESuperMicro PIIIDME840 chipset840 chipset1-2 PIII 600 MHz1-2 PIII 600 MHzPCI 33/32-66/64PCI 33/32-66/64

ExtremeNetworksExtremeNetworksSummit 48Summit 4848 Fe48 Fe2 Ge2 Ge

2 ULTRA 160 SCSI2 ULTRA 160 SCSIADAPTEC 39160ADAPTEC 39160

Problemi con Problemi con Memoria SDRAMMemoria SDRAM

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1414

LNL

DISKDISKSERVERSERVER

LNL PC Farm (phase II – Nov 2000)LNL PC Farm (phase II – Nov 2000)

N1N1SSWWIITTCCHH

N15N15

GEGE

FEFE

N2N2

Farm Module IFarm Module I

DISKDISKSERVERSERVER

SuperMicro 370DLESuperMicro 370DLEServerWorks III LE chipsetServerWorks III LE chipset2 PIII 800 MHz2 PIII 800 MHz512 Mbyte 512 Mbyte PCI 33/32-66/64PCI 33/32-66/64Case minitowerCase minitower

ExtremeNetworksExtremeNetworksSummit 48Summit 4848 Fe48 Fe2 Ge2 Ge

Enclosure Compaq 4354REnclosure Compaq 4354R14 dischi 36GB SCSI U314 dischi 36GB SCSI U3500 Gbyte Storage Array500 Gbyte Storage Array

Raid Controller U3Raid Controller U3CompaqCompaqSmartArray 5300SmartArray 5300

Case Test: SuperMicro SC810Case Test: SuperMicro SC810

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1515

LNL Low Price 1U case example: SM SC810Low Price 1U case example: SM SC810

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1616

LNLCMS Fabric at LNL (2001)CMS Fabric at LNL (2001)

N1N1N10N10F

EF

EN2N2

Farm ModuleFarm Module

N1N1N10N10

FE

FE

N2N2

Farm ModuleFarm Module

N1N1N10N10

FE

FE

N2N2

Farm ModuleFarm Module

N1N1N10N10

FE

FE

N2N2

Farm ModuleFarm Module

DISKDISKSERVERSERVER

DISKDISKSERVERSERVER

DISKDISKSERVERSERVER

DISKDISKSERVERSERVER

GigaEthernet SwitchGigaEthernet Switch

SWITCHSWITCH SWITCHSWITCH SWITCHSWITCH SWITCHSWITCH

HP2524

HP8000 FastIron 4000

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1717

LNLLNL FarmLNL Farm

32 PCs, PIII 600 MHz, 3 Farm modules, 32 PCs, PIII 600 MHz, 3 Farm modules, 120 Gbps GE/Cu switch, 120 Gbps GE/Cu switch, 500 Gbyte disk500 Gbyte diskUsed for:Used for:- Sadirc/CMS Event Builder prototypesSadirc/CMS Event Builder prototypes- Data Analysis Production for LNL expsData Analysis Production for LNL exps- NFS and topological tests - NFS and topological tests

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1818

LNLNFS Distributed Disks based Farm (CNAF)NFS Distributed Disks based Farm (CNAF)

SSWWIITTCCHH

N10N10

FEFE

N2N2

Farm ModuleFarm Module

ANISANISBootBoot

ServerServer

N1N1

- Topology testsTopology tests- dual slim processors (rack mounted) testsdual slim processors (rack mounted) tests- remote file systems (nfs, ams, etc. ) testsremote file systems (nfs, ams, etc. ) tests

Requests for 2000:Requests for 2000:- 10 dual slim processors10 dual slim processors 90 Ml 90 Ml- Fast ethernet switchFast ethernet switch 10 Ml 10 Ml- 10 SCSI disks 36/72 GB 10 SCSI disks 36/72 GB 20 Ml 20 Ml ------------

120 Ml120 Ml

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1919

LNLStorage Systems: FC and SCSI road mapStorage Systems: FC and SCSI road map

SCSISCSI

Ultra3/Ultra160Ultra3/Ultra160160 MB/sec160 MB/sec

Ultra2Ultra280 MB/sec80 MB/sec

Wide UltraWide Ultra40 MB/sec40 MB/sec

Ultra320Ultra320320 MB/sec320 MB/sec

Ultra640Ultra640640 MB/sec640 MB/sec

20042004

20032003

20022002

20012001

20002000

19991999

19981998

19971997

19961996

19951995

Fibre ChannelFibre Channel

1Gb1Gb100 MB/sec100 MB/sec

2Gb2Gb200 MB/sec200 MB/sec

400MB/sec (full duplex)400MB/sec (full duplex)

8/10Gb8/10Gb1000 MB/sec1000 MB/sec

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2020

LNLStorage Systems: Fiber Channel/SCSI evaluation (PD)Storage Systems: Fiber Channel/SCSI evaluation (PD)

N1N1FFCC

SSWWTT

N4N4

FCFCN2N2

N3N3

N1N1

Arbitrated loop (AL) FCArbitrated loop (AL) FC

Switched FCSwitched FC

Requests for 2000Requests for 2000- 4 Disk Servers4 Disk Servers 30 Ml30 Ml- 1 8 ports FC Switch 1 8 ports FC Switch 20 Ml20 Ml- 4 FC adapters 15 Ml4 FC adapters 15 Ml- 2 Raid Arrays 30 Ml2 Raid Arrays 30 Ml- 4 Ultra/1604 Ultra/160 SCSI disksSCSI disks 8 Ml 8 Ml ------------

103 Ml103 Ml

N1N1

SCSISCSI

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2121

LNLStorage Systems: Serial ATA (Padova)Storage Systems: Serial ATA (Padova)

N1N1

SerialSerialATAATA

Requests for 2001Requests for 2001- 4 ATA Cntr.4 ATA Cntr. 5 Ml5 Ml

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2222

LNLInterconnection Networks: Gigabit Ethernet (Genova)Interconnection Networks: Gigabit Ethernet (Genova)

Communication Protocols for high speed networks:Communication Protocols for high speed networks:

- Communication System GAMMACommunication System GAMMA

- Use of Programmable GigaEthernet NICsUse of Programmable GigaEthernet NICs

- 1000 Base T GigaEthernet (on copper) reliability evaluation1000 Base T GigaEthernet (on copper) reliability evaluation

- Implementation of efficient parallel/distributed RAID systemsImplementation of efficient parallel/distributed RAID systems

Requests for 2000Requests for 2000- 8 PC8 PC 24 Ml24 Ml- 1 12 ports Geth Switch 1 12 ports Geth Switch 30 Ml30 Ml- 8 GEth NICs8 GEth NICs 16 Ml16 Ml- ------------TOTALTOTAL 70 Ml70 Ml

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2323

LNLInterconnection Networks: MyrinetInterconnection Networks: Myrinet

Full-duplex 2.5+2.5 Gigabit/second links, switch ports, and Full-duplex 2.5+2.5 Gigabit/second links, switch ports, and interface portsinterface ports

Flow control, error control, and “heartbeat” continuity Flow control, error control, and “heartbeat” continuity monitoring on every linkmonitoring on every link

Low latency, cut-through crossbar switchesLow latency, cut-through crossbar switches

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2424

LNLInterconnection Networks: Myrinet (Lecce)Interconnection Networks: Myrinet (Lecce)

N1N1MMYYRRIINNEETT

N10N10

N2N2DISKDISK

SERVERSERVER

MMYYRRIINNEETT

N10N10

N2N2

N1N1Requests for 2000Requests for 2000- 10 Biproc Comp. Nodes10 Biproc Comp. Nodes 50 Ml50 Ml- 1 16 ports Myrinet Switch 1 16 ports Myrinet Switch 8 Ml 8 Ml- 10 Myrinet NICs10 Myrinet NICs 35 Ml35 Ml- 10 disks 72 GB 10 disks 72 GB 35 Ml 35 Ml- 1 disk server1 disk server 10 Ml10 Ml- Rack + cablesRack + cables 5 Ml 5 Ml- ------------TOTALTOTAL 143 Ml143 Ml

Myrinet based farmMyrinet based farm- Disk Server based - Disk Server based - Distributed disks based- Distributed disks based

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2525

LNLInterconnection Networks: InfiniBand (I)Interconnection Networks: InfiniBand (I)

TODAYTODAY

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2626

LNLInfiniBand (II)InfiniBand (II)

Legacy host architectureLegacy host architecture

The Infiniband ModelThe Infiniband Model

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2727

LNLInfiniBand (III)InfiniBand (III)

What?What?

– Initial single link signaling rate of 2.5Gbaud Initial single link signaling rate of 2.5Gbaud

Means unidirectional transfer rate of 250MB/sec with a Means unidirectional transfer rate of 250MB/sec with a theoretical full duplex rate of 500MB/sectheoretical full duplex rate of 500MB/sec

– Initial support for single, 4, and 12 wide link widthsInitial support for single, 4, and 12 wide link widths

– Point to point switched fabricPoint to point switched fabric

– Message based with multicasting supportMessage based with multicasting support

MultiMultiStageStageSwitchSwitch

LinksLinksTCATCA

I/O ControllerI/O Controller

MemoryMemoryControllerController

PCI-X PCI-X HostHost

BridgesBridges

CPUCPU CPUCPU CPUCPU CPUCPU

HCAHCALinkLink

Fibre ChannelFibre Channel

SCSISCSI

Gig. EthernetGig. Ethernet

MemoryMemory

HCAHCA- Host Channel Adapter- Host Channel AdapterTCATCA - Target Channel Adapter - Target Channel Adapter

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2828

LNLInterconnection Networks: InfiniBand (LNL)Interconnection Networks: InfiniBand (LNL)

Simple test system (4 servers + Storafe Area Network + Simple test system (4 servers + Storafe Area Network + Network) for Network) for 20012001 is possible is possible

Early access to the productsEarly access to the products

Test Beds Test Beds

Requests 2001 (valuations) Requests 2001 (valuations) Comm V (Sadirc2000)Comm V (Sadirc2000)• 4 servers4 servers 50 Ml50 Ml• 1 IBA Switch1 IBA Switch 20 Ml20 Ml• IBA AdaptersIBA Adapters 10 Ml10 Ml

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2929

LNLMicroprocessor Technology: Intel Itanium (IA64)Microprocessor Technology: Intel Itanium (IA64)

Requests for 2001Requests for 2001

– Padova: Investigation IA64Padova: Investigation IA64 4 PC IA644 PC IA64 40 Ml 40 Ml

– Lecce: IA64 in MyrinetLecce: IA64 in Myrinet 10 Dual IA6410 Dual IA64 100 Ml100 Ml 1 Disk Server +1 Disk Server +

disksdisks 16 Ml 16 Ml

G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 3030

LNLResourcesResources

SitesSites PersonsPersons FTEFTE

MaterialsMaterials

(Ml)(Ml)2000 20012000 2001

Ital. Trav.Ital. Trav.

(Ml)(Ml)2000 20012000 2001

Inter.TravInter.Trav

(Ml)(Ml)2000 20012000 2001

LegnaroLegnaro 55 2.62.6 1010 8080

PadovaPadova 77 2.12.1 103103 4545

CNAFCNAF 66 1.71.7 120120 --

BolognaBologna 22 0.60.6 -- --

LecceLecce 66 1.41.4 143143 116116

TorinoTorino 11 0.20.2 -- --

GenovaGenova 22 1.01.0 7070 --

CataniaCatania 33 0.70.7 -- --

Roma 1Roma 1 77 1.51.5 ** **

TOTALTOTAL 3939 11.811.8 446446 161161

241241