Body and gestures (2):models, algorithms , applications

Corso di Interazione uomo-macchina II

Prof. Giuseppe Boccignone

Dipartimento di Scienze dell’InformazioneUniversità di Milano

boccignone@dsi.unimi.ithttp://homes.dsi.unimi.it/~boccignone/l

DeMeuse (1987)

Non verbal behavior//De Meuse

A. Vinciarelli, M. Pantic, H. Bourlard, Social Signal Processing: Survey of an Emerging Domain,Image and Vision Computing (2008)

Posture and gesture interaction

Posture and gesture interaction

• There are two main challenges in recognizing posture and gestures:

• detecting the body parts involved in the gesture (e.g. hands)

• addressed by selecting appropriate visual features: these include, e.g., histograms of oriented gradients , optical flow, spatio-temporal salient points and space-time volumes .

• modeling the temporal dynamic of the gesture

• addressed by using techniques such as Dynamic Time Warping , Hidden Markov Models, and Conditional Random Fields .

Analysing postures and gestures

• The primary goal of gesture recognition research is to create a system which can identify specific human gestures and use them to convey information or for device control.

• A gesture may be defined as a physical movement of the hands, arms, face, and body with the intent to convey information or meaning.

• Gesture recognition, then, consists not only of the tracking of human movement, but also the interpretation of that movement as semantically meaningful commands

Analysing postures and gestures

• Like in the case of gestures, machine recognition of walking style (or gait) has been investigated as well, but only for purposes different from SSP, namely recognition and identification in biometric applications

• The common approach is to segment the silhouette of the human body into individual components (legs, arms, trunk, etc.), and then to represent their geometry during walking through

• vectors of distances , symmetry operators , geometric features of body and stride (e.g. distance between head and feets or pelvis)

Analysing postures and gestures

Analysing postures and gestures//application areas

• Automatic posture recognition has been addressed in few works, mostly aiming at

• surveillance

• activity recognition

• Few works where the posture is recognized as a social signal

• to estimate the interest level of children learning to use computers

• to recognize affective state of people

Analysing postures and gestures

http://www-prima.imag.fr/

Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.

Body gestures and postures//Generative model

Bodygesture

(mental conceptof)

Bodyposture

(limb motion)

Visual images

G P V

Tpg Tvp

G

P

V

P(V, P, G)=P(V | P) P(P | G) P(G)

Body gestures and postures//Generative model

Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.

G

P

V

P(G | V) =P(V | G) P(G)

P(V )

P(G) ∑H P(V | P) P(P | G)

∑G ∑H P(G) P(V | P) P(P | G)

=

Body gestures and postures//Generative model: inference

Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.

pointing

G

P

V

Body gestures and postures//Generative model: more complex model

Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.

pointing

body

walking golf swing

Body gestures and postures//Generative model

Gt

Pt

Vt

Gt+1

Pt+1

Vt+1

P(Gt+1 | Vt+1) ≈

P(Vt+1 | Gt+1) P(Gt+1 | Vt) ≈

P(Vt+1 | Gt+1) ∑Gt P(Gt+1 | Gt) P(Gt | Vt)

body

Body gestures and postures//Generative model: inference

Gt

Pt

Vt

Gt+1

Pt+1

Vt+1

body

Body gestures and postures//Generative model: inference

Body gestures and postures//Generative model: architecture

Body gestures and postures//Generative model: body models

tracked body parts indexed by different colors

Body gestures and postures//limb segmentation

tracked body parts indexed by different colors

Body gestures and postures//limb segmentation

Body gestures and postures//Body-part parameterization

ellipse convex hull

Body gestures and postures//Limb pose estimation: head

Body gestures and postures//Limb pose estimation: arm

Body gestures and postures//Limb pose estimation: leg

Body gestures and postures//Body-part parameterization

Body gestures and postures//Estimating body posture

Hand gestures

• Taxonomy of hand gestures for HCI

• Visual interpretation of hand gestures can help in achieving the ease and naturalness desired for Human Computer Interaction (HCI).

Hand gestures

• Classical use in HCI:

• In a computer controlled environment one wants to use the human hand to perform tasks that mimic both the natural use of the hand as a manipulator, and its use in human-machine communication

Hand gestures

• Classical use in HCI:

• In a computer controlled environment one wants to use the human hand to perform tasks that mimic both the natural use of the hand as a manipulator, and its use in human-machine communication

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

G

H

V

P(V, H, G)=P(V | H) P(H | G) P(G)

Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.

Hand gestures//Generative model

G

H

V

P(G | V) =P(V | G) P(G)

P(V )

P(G) ∑H P(V | H) P(H | G)

∑G ∑H P(G) P(V | H) P(H | G)

=

Hand gestures//Generative model

Hand gestures//Generative model

Gt

Ht

Vt

Gt+1

Ht+1

Vt+1

P(Gt+1 | Vt+1) ≈

P(Vt+1 | Gt+1) P(Gt+1 | Vt) ≈

P(Vt+1 | Gt+1) ∑Gt P(Gt+1 | Gt) P(Gt | Vt)

Hand gestures//Generative model

• 3D hand model-based models of gestures use articulated models of the human hand and arm to estimate the hand and arm movement parameters. Such movements are later recognized as gestures.

• Appearance-based models directly link the appearance of the hand and arm movements in visual images to specific gestures

Hand gestures//Generative model: spatial models

Hand gestures//Generative model: spatial models

Hand gestures//Generative model: spatial models

3D Textured volumetric model

3D wireframe volumetric model.

3D skeletal model

Binary silhouette.

Contour

Gestures for augmented reality

Kinect style://http://www.openni.org/

Kinect style://http://www.openni.org/

Kinect style://http://www.openni.org/

OpenInterface//http://www.openinterface.org/platform/

Recognizing Affective Dimensions from BodyPosture

• Behavioral studies have shown that posture can communicate discrete emotion categories as well as affective dimensions.

• In the affective computing field, while models for the automatic recognition of discrete emotion categories from posture have been proposed, there are no models for the automatic recognition of affective dimensions from static posture.

• A study by Andrea Kleinsmith and Nadia Bianchi-Berthouze

Recognizing Affective Dimensions from BodyPosture (Kleinsmith and Bianchi-Berthouze)

Recognizing Affective Dimensions from BodyPosture

Recognizing Affective Dimensions from BodyPosture

Recognizing Affective Dimensions from BodyPosture

Recognizing Affective Dimensions from BodyPosture

Recognizing Affective Dimensions from BodyPosture

Recognizing Affective Dimensions from BodyPosture

Recognizing Affective Dimensions from BodyPosture

Recognizing Affective Dimensions from BodyPosture

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

Multimodal emotion recognition//Gunes & Piccardi

• Results show that emotion classification using the two modalities achieves better recognition accuracy in general, outperforming the classification using the face modality only

• using expressive body information adds accuracy to the emotion recognition based on the face alone.

• early fusion seems to achieve a better recognition accuracy compared to late fusion.

Multimodal emotion recognition//Gunes & Piccardi

Anthropomorphic Embodied ConversationalAgents (Cowell)

• Interaction with a computer should be as easy as interacting withother people, taking advantage of the multimodal nature ofhuman communication

• Focus revolved around behaviors that portray a credible fac-ade, thereby helping embodied conversational agents (ECAs) to form a successful cooperative dyad with users

Anthropomorphic Embodied ConversationalAgents (Cowell)

Anthropomorphic Embodied ConversationalAgents (Cowell): design suggestions

Non verbalbehavior

Anthropomorphic Embodied ConversationalAgents (Cowell): design suggestions

Anthropomorphic Embodied ConversationalAgents (Cowell): applications

The mobile device landscape