M. De Cecco - Lucidi del corso di Robotica e Sensor Fusion Microsoft Kinect IR PATTERN PROJECTO R...

M. De Cecco - Lucidi del corso di Robotica e Sensor Fusion

Microsoft Kinect

IR PATTERN PROJECTO

R

RGB CAMERA

IR CAMERA


Microsoft Kinect: How it works?


Microsoft Kinect


Microsoft Kinect: Data

• points3D_Irsys.m: 3xN matrix of 3D points measured and associated RGB color

• calibData.m : intrinsic and extrinsic parameters of IR and RGB camera

• imRGB.m : image captured by RGB camera


Microsoft Kinect: Data

points3D_IRsys: 3xN matrix of 3D points expressed in IR camera reference system

X x1 x… xn

Y y1 y… yn

Z z1 z… zn

Number of points N should be 640x480 = 307200!


Microsoft Kinect: Calibration Data

calibData.m : intrinsic and extrinsic parameters of IR and RGB camera

•K_rgb : 3x3 : intrinsic parameters matrix of RGB camera•kc_rgb : 1x5 distortion paramteres of RGB camera•K_ir : 3x3 : intrinsic parameters matrix of IR camera•kc_ir : 1x5 distortion parameters of IR camera•rotRGB2IR : 3x3 rotation matrix from RGB to IR reference system•trasRGB2IR : 1x3translation vector of IR reference system respect to RGB


x

y

z

SC

x

y

z

OCCD

SCCD

OC

Reminder

OP

Il modello geometrico (ideale) della telecamera impiega il modello pinhole (stenoscopio), per cui entrano in gioco i seguenti parametri:• il centro della prospettiva (Oc): corrisponde al secondo punto nodale del sistema ottico;• il piano immagine (p): il piano coincidente con lo schermo sul quale si forma l’immagine;• l’asse principale: la retta coincidente con l’asse ottico (zc);• il punto principale (OP), intersezione dell’asse ottico con il piano immagine;• la distanza focale della telecamera (f): misura la distanza tra il centro della prospettiva ed il punto principale

p

f

NOTA: nel sistema CCD la colonna corrisponde alla x, la riga alla y

0

0

0 1/ 0 /

1 0 0

0 0 1 0

0 0 0 1 1

C C CCDCCD

s y s colonna

x rigaP T P

f


Reference System: From Camera to CCD

K_rgb : intrinsic parameters matrix of RGB cameraK_ir : intrinsic parameters matrix of IR camera

100

0

0

0

0

yfy

xfx

K

fx , fy : focal length (in pixel) of the camerax0, y0: coordinate of principal point

x

y

z

SRGB

x

y

z

OCCD

SCCD

OC

OP

p

1x0

y0

CP

P


nRGBccd PKkcP

1cam

cam

cam

cam

cam

cam

cam

cam

cam

cam

n PzPz

PzPx

PzPz

PzPz

PzPx

P

Homogeneous coordinate:

Alredy compensated in the image given by Kinect

x

y

z

SRGB

x

y

z

OCCD

SCCD

OC

OP

p

1

x0

y0

Pccd

NOTE: the x coordinate in CCD reference system is the column and the y is the row

Expressed in pixel

Reference System: From Camera to CCD

nP


Refernece System: From IR to RGB

RGBZ

RGBX

RGBY

RGB

IRX

IRY

IRZ

IR

x

y

z

O

Pp

CCDIRP

R T

)(1 TPRP IRRGB

IRtrasTGBT

IRrotRGBR

where

2

2

:


Microsoft Kinect: RGB image

Distorsion alredy compensated


Things to do:

Steps:• Associate 3D points in IR sys with RGB info• Identify ground plane and eliminate it (Ransac)• Found the position of two cubes (SQ and

Ransac)• Chose the blue (or red) cube

Goal: find in 3d space the blue (or red) cube (dimension 10x10x10 cm )

M. De Cecco - Lucidi del corso di Robotica e Sensor Fusion Microsoft Kinect IR PATTERN PROJECTO R...

Documents

Transcript of M. De Cecco - Lucidi del corso di Robotica e Sensor Fusion Microsoft Kinect IR PATTERN PROJECTO R...