Body and gestures:models, algorithms , applications

Corso di Interazione uomo-macchina II

Prof. Giuseppe Boccignone

Dipartimento di Scienze dell’InformazioneUniversità di Milano

boccignone@dsi.unimi.ithttp://homes.dsi.unimi.it/~boccignone/l

DeMeuse (1987)

Non verbal behavior//De Meuse

A. Vinciarelli, M. Pantic, H. Bourlard, Social Signal Processing: Survey of an Emerging Domain,Image and Vision Computing (2008)

Posture and gesture interaction

Posture and gesture interaction

• There are two main challenges in recognizing posture and gestures:

• detecting the body parts involved in the gesture (e.g. hands)

• addressed by selecting appropriate visual features: these include, e.g., histograms of oriented gradients , optical flow, spatio-temporal salient points and space-time volumes .

• modeling the temporal dynamic of the gesture

• addressed by using techniques such as Dynamic Time Warping , Hidden Markov Models, and Conditional Random Fields .

Analysing postures and gestures

• The primary goal of gesture recognition research is to create a system which can identify specific human gestures and use them to convey information or for device control.

• A gesture may be defined as a physical movement of the hands, arms, face, and body with the intent to convey information or meaning.

• Gesture recognition, then, consists not only of the tracking of human movement, but also the interpretation of that movement as semantically meaningful commands

Analysing postures and gestures

• Like in the case of gestures, machine recognition of walking style (or gait) has been investigated as well, but only for purposes different from SSP, namely recognition and identification in biometric applications

• The common approach is to segment the silhouette of the human body into individual components (legs, arms, trunk, etc.), and then to represent their geometry during walking through

• vectors of distances , symmetry operators , geometric features of body and stride (e.g. distance between head and feets or pelvis)

Analysing postures and gestures

Analysing postures and gestures//application areas

• Automatic posture recognition has been addressed in few works, mostly aiming at

• surveillance

• activity recognition

• Few works where the posture is recognized as a social signal

• to estimate the interest level of children learning to use computers

• to recognize affective state of people

Analysing postures and gestures

http://www-prima.imag.fr/

Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.

Body gestures and postures//Generative model

Bodygesture

(mental conceptof)

Bodyposture

(limb motion)

Visual images

G P V

Tpg Tvp

G

P

V

P(V, P, G)=P(V | P) P(P | G) P(G)

Body gestures and postures//Generative model

Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.

G

P

V

P(G | V) =P(V | G) P(G)

P(V )

P(G) !H P(V | P) P(P | G)

!G !H P(G) P(V | P) P(P | G)

=

Body gestures and postures//Generative model: inference

Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.

pointing

G

P

V

Body gestures and postures//Generative model: more complex model

Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.

pointing

body

walking golf swing

Body gestures and postures//Generative model

Gt

Pt

Vt

Gt+1

Pt+1

Vt+1

P(Gt+1 | Vt+1) "

P(Vt+1 | Gt+1) P(Gt+1 | Vt) "

P(Vt+1 | Gt+1) !Gt P(Gt+1 | Gt) P(Gt | Vt)

body

Body gestures and postures//Generative model: inference

Gt

Pt

Vt

Gt+1

Pt+1

Vt+1

body

Body gestures and postures//Generative model: inference

Body gestures and postures//Generative model: architecture

Body gestures and postures//Generative model: body models

tracked body parts indexed by different colors

Body gestures and postures//limb segmentation

tracked body parts indexed by different colors

Body gestures and postures//limb segmentation

Body gestures and postures//Body-part parameterization

ellipse convex hull

Body gestures and postures//Limb pose estimation: head

Body gestures and postures//Limb pose estimation: arm

Body gestures and postures//Limb pose estimation: leg

Body gestures and postures//Body-part parameterization

Body gestures and postures//Estimating body posture

Hand gestures

• Taxonomy of hand gestures for HCI

• Visual interpretation of hand gestures can help in achieving the ease and naturalness desired for Human Computer Interaction (HCI).

Hand gestures

• Classical use in HCI:

• In a computer controlled environment one wants to use the human hand to perform tasks that mimic both the natural use of the hand as a manipulator, and its use in human-machine communication

Hand gestures

• Classical use in HCI:

• In a computer controlled environment one wants to use the human hand to perform tasks that mimic both the natural use of the hand as a manipulator, and its use in human-machine communication

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

G

H

V

P(V, H, G)=P(V | H) P(H | G) P(G)

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

G

H

V

P(G | V) =P(V | G) P(G)

P(V )

P(G) !H P(V | H) P(H | G)

!G !H P(G) P(V | H) P(H | G)

=

Hand gestures//Generative model

Hand gestures//Generative model

Gt

Ht

Vt

Gt+1

Ht+1

Vt+1

P(Gt+1 | Vt+1) "

P(Vt+1 | Gt+1) P(Gt+1 | Vt) "

P(Vt+1 | Gt+1) !Gt P(Gt+1 | Gt) P(Gt | Vt)

Hand gestures//Generative model

• 3D hand model-based models of gestures use articulated models of the human hand and arm to estimate the hand and arm movement parameters. Such movements are later recognized as gestures.

• Appearance-based models directly link the appearance of the hand and arm movements in visual images to specific gestures

Hand gestures//Generative model: spatial models

Hand gestures//Generative model: spatial models

Hand gestures//Generative model: spatial models

3D Textured volumetric model

3D wireframe volumetric model.

3D skeletal model

Binary silhouette.

Contour

Gestures for augmented reality

Kinect style://http://www.openni.org/

Kinect style://http://www.openni.org/

Kinect style://http://www.openni.org/

The Pascal Challengehttp://gesture.chalearn.org/data

• portraying a single user in front of a fixed camera, interacting with a computer by performing gestures!to

• play a game,

• remotely control appliances or robots, or

• learn to perform!gestures!from an educational software

The Pascal Challengehttp://gesture.chalearn.org/data

• collected a large dataset of 50,000 gestures with KinectTM

• provide MatlabTM code to browse though the data and process it

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

• Results show that emotion classification using the two modalities achieves better recognition accuracy in general, outperforming the classification using the face modality only

• using expressive body information adds accuracy to the emotion recognition based on the face alone.

• early fusion seems to achieve a better recognition accuracy compared to late fusion.

Multimodal emotion recognition//Gunes & Piccardi