Kinect v2 SDK C++ - 1. Kinect Basics

Goals: Learn how to initialize a Kinect and get RGB data from it.
Source: View Source Download: 1_Basics.zip

Overview

We have two real pieces of Kinect-specific code. I will go over these in some detail, and give a fairly high level overview of the display code.

Includes, Constants, and Globals
Kinect Code
Windowing, Event Handling, and Main Loop
Display via OpenGL

Includes, Constants, and Globals

Includes

Mostly self explanatory: Kinect.h is the main Kinect header.

You need to include Ole2.h and Windows.h for the Kinect includes to work correctly. Don't forget to include the relevant code for your windowing system and OpenGL.
GLUT SDL

#include <Windows.h> #include <Ole2.h> #include <gl/GL.h> #include <gl/GLU.h> #include <gl/glut.h> #include <Kinect.h>

#include <Windows.h> #include <Ole2.h> #include <SDL_opengl.h> #include <SDL.h> #include <Kinect.h>

GLUT	SDL
#include <Windows.h> #include <Ole2.h> #include <gl/GL.h> #include <gl/GLU.h> #include <gl/glut.h> #include <Kinect.h>	#include <Windows.h> #include <Ole2.h> #include <SDL_opengl.h> #include <SDL.h> #include <Kinect.h>

Constants and Global Variables

We define the width and height as 1920*1080, since these are the Kinect color camera input dimensions.

Note that the data array will hold a copy of the image we get from the Kinect, so that we can use it as a texture. Experienced OpenGL users may want to use a Frame Buffer Object instead.

#define width 1920
#define height 1080

// OpenGL Variables
GLuint textureId;              // ID of the texture to contain Kinect RGB Data
GLubyte data[width*height*4];  // BGRA array containing the texture data

// Kinect variables
IKinectSensor* sensor;         // Kinect sensor
IColorFrameReader* reader;     // Kinect color data source

Kinect Code

Kinect Initialization

This is our first real Kinect-specific code. The initKinect() function initializes a Kinect sensor for use. This consists of two parts: First we find an attached Kinect sensor, then we initialize it and prepare to read data from it.

bool initKinect() {
    if (FAILED(GetDefaultKinectSensor(&sensor))) {
        return false;
    }
    if (sensor) {
        sensor->Open();

        IColorFrameSource* framesource = NULL;
        sensor->get_ColorFrameSource(&framesource);
        framesource->OpenReader(&reader);
        if (framesource) {
            framesource->Release();
            framesource = NULL;
        }
        return true;
    } else {
        return false;
    }
}

Things to note:

Normally, we'd be a bit more careful about return values for all of these functions; however, for brevity we will omit some of them.
Note the general pattern of data stream requesting:
1. Make a framesource of the appropriate type (Color, Depth, Body, etc.)
2. Request the framesource from the sensor interface
3. Open the reader from the framesource
4. The framesource can safely be released
5. Request data frames from the reader

Getting an RGB frame from the Kinect

void getKinectData(GLubyte* dest) {
    IColorFrame* frame = NULL;
    if (SUCCEEDED(reader->AcquireLatestFrame(&frame))) {
        frame->CopyConvertedFrameDataToArray(width*height*4, data, ColorImageFormat_Bgra);
    }
    if (frame) frame->Release();
}

This function is very simple. We poll for a frame from the data source, and if one is available, we can copy it into our texture array in the appropriate format. Don't forget to release the frame afterward!

Note that the raw color frame is probably in a YUV format or similar, so the conversion to a usable RGB/BGR format does involve a bit of processing. You can also access the raw data using frame->AccessUnderlyingBuffer, which we will use in the depth tutorial.

Metadata about the frame can be accessed as well. Here's a list of some things you might be interested in:

Camera settings, such as exposure and gain, can be accessed from an IColorCameraSettings

    IColorCameraSettings* camerasettings;
    frame->get_ColorCameraSettings(&camerasettings);
    float gain;
    TIMESPAN exposure;
    camerasettings->get_Gain(&gain);
    camerasettings->get_ExposureTime(&exposure);
    // ...etc.

Dimensions and field of view can be accessed from an IFrameDescription:

    IFrameDescription* description;
    frame->get_FrameDescription(&description);
    int height, width;
    float xfov, yfov;
    description->get_Height(&height);
    description->get_Width(&width);
    description->get_HorizontalFieldOfView(&xfov);
    description->get_VerticalFieldOfView(&yfov);
    // ...etc.

This also applies to Depth and IR frames.

See https://msdn.microsoft.com/en-us/library/microsoft.kinect.kinect.icolorframe.aspx for details.
That's all the Kinect code! The rest is just how to get it onscreen.

Windowing, Event Handling, and Main Loop

This section explains the GLUT- or SDL-specific code, consisting of window initialization, event handling, and the main update loop.

The initialization code is specific to which implementation (SDL or GLUT) is used. It simply initializes a window using the appropriate API, returning false on failure. The GLUT version also sets up a main loop by specifying that the draw() function be called every loop iteration.

The main loop is started in the execute() function. In GLUT, the loop is handled behind the scenes, so all we need to do is call the glutMainLoop() function. In SDL we write our own loop. Within each loop, we draw any new frames to the screen; this processing is done in the drawKinect() function.

There are many references online for both GLUT and SDL if you want to do more complex window and loop management or learn more about these functions.

GLUT

void draw() {
   drawKinectData();
   glutSwapBuffers();
}

void execute() {
    glutMainLoop();
}

bool init(int argc, char* argv[]) {
    glutInit(&argc, argv);
    glutInitDisplayMode(GLUT_DEPTH | GLUT_DOUBLE | GLUT_RGBA);
    glutInitWindowSize(width,height);
    glutCreateWindow("Kinect SDK Tutorial");
    glutDisplayFunc(draw);
    glutIdleFunc(draw);
    return true;
}

SDL

void execute() {
    SDL_Event ev;
    bool running = true;
    while (running) {
        while (SDL_PollEvent(&ev)) {
            if (ev.type == SDL_QUIT) running = false;
        }
        drawKinectData();
        SDL_GL_SwapBuffers();
    }
}

bool init(int argc, char* argv[]) {
    SDL_Init(SDL_INIT_EVERYTHING);
    SDL_Surface* screen =
        SDL_SetVideoMode(width, height, 32, SDL_HWSURFACE | SDL_GL_DOUBLEBUFFER | SDL_OPENGL);
    return screen;
}

Display via OpenGL

Initialization

Three steps, as described in the code - Setting up the texture to contain our image frame, preparing OpenGL for drawing our texture, and setting up a camera viewpoint (using an orthographic projection for 2D images).

    // Initialize textures
    glGenTextures(1, &textureId);
    glBindTexture(GL_TEXTURE_2D, textureId);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, width, height,
                 0, GL_BGRA, GL_UNSIGNED_BYTE, (GLvoid*) data);
    glBindTexture(GL_TEXTURE_2D, 0);

    // OpenGL setup
    glClearColor(0,0,0,0);
    glClearDepth(1.0f);
    glEnable(GL_TEXTURE_2D);

    // Camera setup
    glViewport(0, 0, width, height);
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    glOrtho(0, width, height, 0, 1, -1);
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();

Obviously, we should wrap this up in a function. I just put it into main() for brevity.

int main(int argc, char* argv[]) {
    if (!init(argc, argv)) return 1;
    if (!initKinect()) return 1;

    /* ...OpenGL texture and camera initialization... */

    // Main loop
    execute();
    return 0;
}

Drawing a frame to screen

This is very standard code. We first copy the kinect data into our own buffer, then specify that our texture will use that buffer.

void drawKinectData() {
    glBindTexture(GL_TEXTURE_2D, textureId);
    getKinectData(data);
    glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, width, height, GL_BGRA, GL_UNSIGNED_BYTE, (GLvoid*)data);

Then, we draw a rectangle that is textured with our frame.

    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
    glBegin(GL_QUADS);
        glTexCoord2f(0.0f, 0.0f);
        glVertex3f(0, 0, 0);
        glTexCoord2f(1.0f, 0.0f);
        glVertex3f(width, 0, 0);
        glTexCoord2f(1.0f, 1.0f);
        glVertex3f(width, height, 0.0f);
        glTexCoord2f(0.0f, 1.0f);
        glVertex3f(0, height, 0.0f);
    glEnd();
}

The End! Build and run, making sure that your Kinect is plugged in. You should see a window containing a video stream of what your Kinect sees.

Previous: Setup

Next: Kinect Depth Data