Mapping Movement to Video Effects & Sound with Kinect 1

This class is for video and installation artists, musicians, dancers and others who are interested in creating video or audio effects using the movement of the body. You will learn how to capture data from the kinect and use motion tracking methods to map movement into parameters that can affect video and audio.  

Have you ever wanted to modulate the speed of a movie as you move from left to right in front of a video wall? Do you want to mix two videos and pan the audio as you move in space? Perhaps you want to create an interactive orchestra where different positions in space trigger a variety of instruments to play.  Or you are a musician who wants to create an interactive dance that generates music through movement. If so, this class is for you.

This class will be hands on, with the students following along with the instructor. 

Since the OpenNI framework has been discontinued, mac users and owners of Kinect 1 have had limited resources to take creative advantage of their interfaces.  There are, however, available objects that enable mac users to capture kinect 1 data in max/msp. 

You will learn you how to combine 3D data from the kinect and 2D data from the cv.jit library to create interactive audio-visual experiences with the movement of the body.  We will use Synapse to capture 3D Kinect data and Jean Marc Pelletier’s cv.jit library to refine our motion tracking. Once the motion tracking is complete we will monitor motion in 3D space and map body movement data to different video and audio parameters.  Using the movement of your body you will control video effects such as speed, looping, mix videos with the jit.qt.movie object attributes, and manipulate objects such as the jit.gl.videoplane by applying transformations and visual effects. You will learn how to trigger audio samples and affect parameters such as volume and panning with your movement. 

Topics to be covered in class:

• Capturing depth Data with Kinect 1 and the jit.synapse object
• Apply motion tracking methods using the cv.jit library and extract X,Y,Z data
• Calibrate motion tracking data to match the input range required for video and audio manipulation
• Map motion tracking data to parameters of video and audio effects
• Manipulate a Quicktime movie and Videoplanes with movement

Required for class:

• This class will use the Synapse object to capture depth data from the Kinect 1 camera and Jean Marc Pelletier’s cv.jit library for motion tracking.
• It is recommended that you download and install the cv.jit library and the jit.synapse object prior to class.
• Students will need to have a Kinect 1 camera.
• Some experience with MAX/MSP/Jitter is recommended but not required.