Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 0 Lezione 7b - 9 Dicembre 2009 Il...

Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 1

Lezione 7b - 9 Dicembre 2009

Il materiale didattico usato in questo corso è stato mutuato da quello utilizzato da Paolo Veronesi per il corso di Griglie Computazionali per la Laurea Specialistica in Informatica tenuto nell’anno accademico 2008/09 presso l’Università degli Studi di Ferrara.

Paolo [email protected], [email protected]://www.cnaf.infn.it/~pveronesi/unife/

Università degli Studi di Bari – Corso di Laurea Specialistica in Informatica

“Tecnologia dei Servizi “Grid e cloud computing” A.A. 2009/2010

Giorgio Pietro Maggi [email protected], http://www.ba.infn.it/~maggi


Defining the Grid

A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user.


The EGEE Project Aim of EGEE:

“to establish a seamless European Grid infrastructure for the support of the European Research Area (ERA)”

EGEE 1 April 2004 – 31 March 2007 71 partners in 27 countries, federated in regional Grids

EGEE-II 1 April 2007 – 30 April 2008 Expanded consortium

EGEE-III 1 May 2008 – 30 April 2010 Transition to sustainable model


Enabling Grid for E-sciencE project

ArcheologyAstronomyAstrophysicsCivil ProtectionComp. ChemistryEarth SciencesFinanceFusionGeophysicsHigh Energy PhysicsLife SciencesMultimediaMaterial Sciences…

>250 sites48 countries>50,000 CPUs>20 PetaBytes>10,000 users>150 VOs>150,000 jobs/day

Flagship Grid infrastructure project co-funded by the European Commission starting from April 2004Entering now in the 3° phase


Defining the Grid



EGEE Infrastructure


EGEE Infrastructures Production service

Scaling up the infrastructure with resource centres around the globe Stable, well-supported infrastructure, running only well-tested and reliable

middleware

Pre-production service Run in parallel with the production service (restricted nr of sites) First deployment of new versions of the gLite middleware Test-bed for applications and other external functionality

T-Infrastructure (Training&Education) Complete suite of Grid elements

and application (Testbed, CA, VO, monitoring, support, …)

Everyone can register and use GILDA for training and testing

20 sites on 3 continents


EGEE Operations Process Geographically distributed responsibility for

operations: There is no “central” operation Regional Operation Centers

Responsible or resource centers in their region Tools are developed/hosted at different sites:

GOC DB (RAL), SAM (CERN), GStat (Taipei), CIC Portal (Lyon)

Grid operator on duty 10 teams working in weekly rotation Crucial in improving site stability and management

Operations coordination Weekly operations meetings Regular ROC managers meetings Series of EGEE Operations Workshops

Procedures described in Operations Manual Introducing new sites Site downtime scheduling Suspending a site Escalation procedures; etc.

Highlights: Distributed operation Evolving and maturing procedures Procedures being in introduced into and shared with the related infrastructure projects


Central probes (SAM)

Local probes

Network monitoring

Doubled size and usage without impact on operations

Improved reliability through multi-level monitoring


EGEE operations

Operations Coord. Centre (OCC)

- management, oversight of all operational and support activities

Regional OperationsCentres (ROC)

- providing the core of the support infrastructure, each supporting a number of resource centres within its region

Resource Centres (RC)

- providing resources

(computing, storage, network…)

- At FZK, coordination and management of user support, single point of contact for users


Monitoring Visualization

11


The EGEE support infrastructure

•RC A

•RC B

•RC C

•RC A

•RC B

•RC C•ROC C•ROC BROC N

RC A

RC B

RC C

TPM

VO TPM CVO TPM B

VO TPM A

GGUS

Central

System

Middleware

supportMiddleware

supportMiddleware

support

Deployment

supportMiddleware

supportDeployment

support

VO Support

CVO Support

BVO Support

A

Middleware

supportMiddleware

supportMiddleware

support

•ROC C•ROC BROC N

Network Support

Network Support Other GridsOther GridsOther Grids

Other GridsOther GridsOther Grids

CODCIC

Portal


Defining the Grid



gLite Middleware Distribution

Combines components from different providers Condor and Globus (via VDT) LCG EGEE Others

Focus on providing a deployable MW distribution for EGEE production service

Middleware services + configuration tools

Follows a service oriented approach Usage of webservices where useful and possible performance-wise

Complemented by application-level servcies


Production Grid MiddlewareKey factors in EGEE Grid Middleware Development:

1. Strict software processUse industry standard software engineering methods Software configuration management, version control, defect tracking,

automatic build system, …

2. Conservative approach in what software to useAvoid “cutting-edge” software Deployment on over 200 sites cannot assume a homogenous

environment – middleware needs to work with many underlying software flavors

Avoid evolving standards Evolving standards change quickly (and sometime significantly cf.

OGSI vs. WSRF) – impossible to keep pace on > 200 sites

Long (and te

dious) path

from prototypes to

production


Certification

Functional Tests

Testbed Deployment

gLite Process

Development

Software Error Fixing

Integration

Deployment Packages

Integration Tests

Installation Guide, Release Notes, etc

Pre-Production

Scalability Tests

Pre-Production Deployment

Fail

Fail

Fail

Pass

Pass

Pass

Production Infrastructure

Problem

Directives

ExternalSoftware

Directives


gLite Software Process Technical Coordination Group (TCG)

gathers & prioritizes user requirementsfrom HEP, biomed, (industry), sites

gLite development is client-driven! Software from EGEE-JRA1 and other projects

JRA1 preview testbed (currently being set up) early exposure to users of “uncertified” components

SA3 Integration Team Ensures components are deployable and work Deployment Modules implemented high-level gLite node types

(WMS, CE, R-GMA Server, VOMS Server, FTS, etc) Build system now spun off into the ETICS project (Jan 2006)

SA3 Certification Team Merge of the JRA1 testing and SA1 certification teams Dedicated testbed; test release candidates and patches Develop test suites

SA1 Pre-Production System Scale tests by users


MiddlewareGlobus GT4 CondorAPST

PlatformInfrastructure

Unix Windows JVM TCP/IP MPI .Net Runtime

Environmental Sciences

Life & Pharmaceutical

Sciences

ApplicationsGeo Sciences

Building Software for the Grid

VPN SSH

Courtesy IBM


MiddlewareGlobus GT4 CondorAPST

PlatformInfrastructure

Unix Windows JVM TCP/IP MPI .Net Runtime

Environmental Sciences

Life & Pharmaceutical

Sciences

ApplicationsGeo Sciences

Building Software for the Grid

VPN SSH

Courtesy IBM,

Upper Middleware & Tools

Lower Middleware Bonds


Defining the Grid



EGEE Applications >270 VOs from several

scientific domains Astronomy & Astrophysics Civil Protection Computational Chemistry Comp. Fluid Dynamics Computer Science/Tools Condensed Matter Physics Earth Sciences Fusion High Energy Physics Life Sciences

Further applications under evaluation

Applications have moved from testing to routine and daily usage

~80-95% efficiency


The Future of Grids Increasing the number of infrastructure users by increasing

awareness Dissemination and outreach Training and education

Increasing the number of applications by improving application support and middleware functionality Improved usability through high level grid middleware extensions

Increasing the grid infrastructure Incubating related projects Ensuring interoperability between projects

Protecting user investments Towards a sustainable grid infrastructure


Grid Interoperability

Incubator for new Grid efforts world-wide Infrastructure and application efforts

Leading role in building world-wide Grids through interoperation efforts Bilateral: EGEE/OSG, EGEE/NDGF,

EGEE/NAREGI, EGEE/Unicore/DEISA Multilateral: Grid Interoperability Now

(GIN)

Experiences and requirements fed back into standardization process (OGF) Many EGEE members are area directors,

WG chairs, WG members

Contacts with industry strengthened Industry Days, Industry Task Force,

Business Associates Programme

GIN


EGEE working with related infrastructure projects

GIN


Evolution

European e-Infrastructure

Testbeds Utility ServiceRoutine Usage

National

Global


Need to prepare permanent, common Grid infrastructure Ensure the long-term sustainability of the European e-Infrastructure

independent of short project funding cycles Coordinate the integration and interaction between National Grid

Infrastructures (NGIs) Operate the production Grid infrastructure on a European level for a wide

range of scientific disciplines

Must be no gap in the support of the production

grid


Summary

Grids represent a powerful new tool for science

Today we have a window of opportunity to move grids from research prototypes to permanent production systems (as networks did a few years ago)

EGEE offers … … a mechanism for linking together people, resources and data of

many scientific community … a basic set of middleware for gridfying applications with

documentation, training and support … regular forums for linking with grid experts, other communities and

industry


gLite Middleware overview


Grid Middleware

When using a PC or workstation you Login with a username and

password (“Authentication”) Use rights given to you

(“Authorisation”) Run jobs Manage files: create them,

read/write, list directories Components are linked by a

bus Operating system One admin domain

• When using a Grid you– Login with digital credentials (“Authentication”)

– Use rights given you (“Authorisation”)

– Run jobs– Manage files: create them, read/write, list directories

• Services are linked by the Internet

• Middleware• Many admin domains


EGEE Project and gLite• Enabling Grids for E-sciencE (EGEE) is the largest multi-disciplinary grid

infrastructure in the world– Brings together more than 120 European organisations – Consists of 250 sites in 48 countries and more than 68,000 CPUs – Is available to some 8,000 users 24 hours a day, 7 days a week– Processes more than 150,000 jobs per day from different scientific domains

• gLite is the middleware powering the EGEE infrastructure and many other related projects

– Is an integrated set of components designed to enable resource sharing among different institutions

– Pulls together contributions from many other projects, including LCG and VDT– Enable users with a large set of services


The “global” grid

e-Infrastructures adopting gLite

e-Infrastructures interoperable or in pro-gress to be made interoperable with gLite


Additional Infrastructures: GILDA• EGEE provides a training infrastructure: GILDA (Grid INFN

Laboratory for Dissemination Activities)– Runs the entire gLite stack protocols– Used to demonstrate EGEE grid technology project– Supports beginner and expert training courses on gLite

• Adopted by several Grid projects worldwide

• Own Certification Authority

• Available 365 days for everyone !

• Used in the ISSGC schools series

• Since 2007 other middleware than gLite are tested on GILDA


•20 sites in 3 continents•> 11000 certificates issued, >20% renewed at least once•> 250 courses, training events, official university curricula•> 2,000,000 hits on the web site from >100 different countries •> 4.5 TB of training material downloaded from the web site

The GILDA t-Infrastructure (https://gilda.ct.infn.it)


gLite in the Grid “ecosystem”

. . .

LCG

EGEE

Used in

USA EU

NextGrid DEISAGridCC

Future grids

EDG

Globus MyProxyCondor ...

VDT

DataTAG

CrossGrid ...

OSG, …

SRM

…

interactive


The Middleware structure• Applications have access both to

Higher-level Grid Services and to Foundation Grid Middleware

• Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory

• Foundation Grid Middleware are actually developed in EGEE

– Must be complete and robust– Should allow interoperation with other major

grid infrastructures– Should not assume the use of Higher-Level

Grid Services


gLite infrastructure

Workload Management System (WMS)Data Management


Typical Job workflow

JDL

Logging &Book-keeping

ResourceBroker

Job SubmissionService

StorageElement

ComputingComputingElementElement

Information Service

Job Status

ReplicaCatalog

Job SubmitEvent

Input Sandbox

JDL

Job

Input Sandbox

Output Sandbox

Output Sandbox

User Interface

Author.Service

voms-proxy-init

GSI data acc/trans

f

glite-job-submit myjob.jdlMyjob.jdl

Executable = “gridTest”;StdError = “stderr.log”;StdOutput = “stdout.log”;InputSandbox = {“/home/joda/test/gridTest”};OutputSandbox = {“stderr.log”, “stdout.log”};InputData = “lfn:testbed0-00019”;DataAccessProtocol = “gridftp”;Requirements = other.Architecture==“INTEL” && \

other.OpSys==“LINUX”;Rank = “other.GlueHostBenchmarkSF00”;


Security System


gLite Security

• Authentication based on X.509 PKI infrastructure– Certificate Authorities (CA) issue (long lived) certificates

identifying individuals (much like a passport)– Trust between CAs and sites is established (offline)– In order to reduce vulnerability, Grid user identification

is done by (short lived) proxies of their certificates

• Proxies can– Be delegated to a service such that it can act on the

user’s behalf– Include additional attributes (like VO information via the

VO Membership Service VOMS)– Be stored in an external proxy store (MyProxy) – Be renewed (in case they are about to expire)


Which CA are trusted in LCG/EGEE?

http://www.eugridpma.org/

“The EUGridPMA is the international organization to coordinate the trust fabric for e-Science grid authentication in Europe. It collaborates with the regional peers APGridPMA for the Asia-Pacific and The Americas Grid PMA in the International Grid Trust Federation. The charter document defines the group's objective, scope and operation. It is the basis for the guidelines documents on the accreditation procedure, the Authentication profile for X.509 secured "classic" certification authorities and other IGTF recognised Profiles. “

In LCG/EGEE CA are installed on machine trough rpms.In LCG/EGEE CA are installed on machine trough rpms.


Grid resources (A)

Grid resources (B)

Conventional grid security

Certification Authority (CA)BobCert request

User Interface (UI)

Bob´s Grid certificate

Sysadmin A :- Create user “grid1“- Map Bob´s certificate to “grid01“

Sysadmin B :- Create user “user001“- Map Bob´s certificate to “user001“

- Single sign-on- Delegation through proxy certificate

- Manual user “mapping“- No info about VOs

grid-proxy-init


gLite: VOMS

Virtual Organization Membership Service (VOMS)

EGEE/gLite enhancement for VO management

Provides information on user's relationship with Virtual Organization (VO)Membership

Group membership

Roles of user

Multiple VOUser can register to multiple VOs and create an aggregate proxy

Access ressources in every registered VO

Backward compatibilityExtra VO related information in users proxy certificate

Users proxy can still be used with non VOMS-aware services

7 Maggio 2009 – Paolo Veronesi Griglie Computazionali - Lezione 007 42


gLite: VOMS - Web interface Requires a valid certificate from a

recognized CA imported on the browser

VO user can

Query membership details

Register himself in the VONeeds a valid certificate

Track his requests VO manager can

Handle requests from users

Administer the VO Everybody can

Get information about the VO


Grid resources (A)

Grid resources (B)

gLite – Enhanced security in gLite

Certification Authority (CA)BobCert request

User Interface (UI)

Bob´s Grid certificate

VO Database

VO Service

VO Manager

VO membership request

VO

VO AccountPool

VO AccountPool

Automatic mappingfor Bob

Automatic mappingfor Bob

voms-proxy-init


LCAS & LCMAPS• At resources level, authorization info is extracted

from the proxy and processed by LCAS and LCMAPS

• Local Centre Authorization Service (LCAS)– Checks if the user is authorized– Checks if the user is banned at the site

• Local Credential Mapping Service (LCMAPS)– Map remote credentials to local credentials (eg. different

UNIX uid/gid)– Map also VOMS group and roles (full support of FQAN)

enables privileges separations

Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 0 Lezione 7b - 9 Dicembre 2009 Il...

Documents

Transcript of Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 0 Lezione 7b - 9 Dicembre 2009 Il...