devblog

Here’s the source and everything for what I’ve done so far:

http://dl.dropbox.com/u/12829356/gsnyderKinectFinalProject.zip

Working with the Kinect this semester has been really fun, and a really great learning experience. It’s interesting to work with hardware that isn’t fully supported by it’s developer – you need more creativity to accomplish things, since not all the features you need are readily available/supported. For example, while I have no doubt in my mind that the official SDK release will contain a listener that can support user connected/user lost reactions, however for this project we had to use the fact that a VRPN sever is created by FAAST to implement the same thing.

It’s also a lot more rewarding when you get something to work, for that reason. Though I am still disappointed that ultimately not everything was implemented as planned. I’m not quite sure why certain gestures didn’t work – for example, Jump or any of the NITE ones – however I have a feeling that it was a combination of both FAAST’s shortcomings and various lighting/vision issues.

If I were to keep working on this project (which I’d like to, given enough time this summer), I would start by learning how OpenNI works and program my own gesture listener of sorts. I like the fact that in the KinectUserRecognizer can send key presses directly to the Flash window, regardless if it is the active window on screen. That way you don’t run into the issue of the user accidentally turning on gesture recognition but forgetting to re-click the Flash window. From there, I would also work on implementing some sort of coordinate system that is compatible with iHart code (or even better, figure out a way to get the computer to see the Kinect as a webcam, as opposed to merely a Kinect). I think it’s silly that the two aren’t compatible out of the box.

That being said, I’m really happy with the progress I made this semester and look forward to continuing this in the future. To anybody who reads this…have an awesome summer!

I’m so close!

I tried plugging everything together for the first time and almost everything works. I’m having trouble reading the game-restart gesture though.

Originally I thought it would be cool to have people jump if they wanted to play again, however it’s looking like that’s not working. It could just be my current set up – the kinect is unable to see my feet right now, after all – but I’d rather use something that works 100% of the time than something that could work if the environment is correct.

I’m thinking I’ll change things so that the user has to raise both of their hands in the air if they want to replay. That should work, hopefully!

I’ve been thinking a lot about how GestureEvents work, and have determined that while my system works, it isn’t that intuitive.

As of right now in order to get a specific event working one must know the name of that specific event and call it as such – for example, if I wanted to listen for a left arm moving forwards, I’d have to call that specific event.

I think conceptually it makes more sense though to pick a limb to listen to, and then from there choose what action you want to listen to as well. That way, especially for something like the hokey pokey, you could have an array of actions you want to listen for along with an array of limbs you wish to listen to – then after informing the listener of which limb/action pair you want, it magically takes care of the rest.

I’m not quite sure how to accomplish this, however. I tried implementing another constructor but I got some error about it not being in the <script/> file, which I’m assuming was caused by the fact that I’m extending an already-existant Flash class to create my own. From here, I’m thinking of including a function  in the gesture manager called interpretGesture – based off of the parameters you give it, it’ll return the correct gesture.

Other than that though, things are starting to wrap up nicely. I should be done with plenty of time to spare.

…I’m stuck.

I’ve managed to get a bulk of the flash file all taken care of, however I can’t go much further from where I am without knowing for sure whether or not there’s a person standing in front of the camera.

I figured I can make flash spoof a calibration screen of sorts, or even better, display the FAAST screen alongside the .fla itself so people can get an idea of how the kinect sees them and use that for feedback as to whether or not the kinect is actually calibrating. The .fla would just show the user how to stand in order to start the calibration process.

I’ve done some research and found that FAAST broadcasts a user’s skeleton information via a VRPN (Virtual Reality Peripheral Network) server, which might theoretically be possible for Flash to read. I’ve been digging around trying to find a way for Flash to listen in to this server, but so far I haven’t had any luck. I’ll keep searching for something though, hopefully something will pop up!

Spent the week getting ready for NEUCS. My poster presentation went amazingly, and OpenMRS did a great job too – we won an award for best project to most likely have a positive impact on society! And my fellow mohos managed to take three out of the five awards presented that day.

Hurray Mount Holyoke! We’re awesome

(I…couldn’t think of a witty title for this :/)

After seeing what worked/what didn’t work in the demo last week, I started expanding on the Hokey Pokey.

I’ve decided that, for the final project, people should only have to do part of the hokey pokey (since I doubt people will have the attention span to actually play through the whole thing). After examining/playing around with FAAST a bit more, I’ve decided that only these parts (or limbs, as I’m now going to refer to them as) should be included:

Left arm, right arm, left leg, right leg, head.

I want to include more but I don’t think FAAST could support it…I’m already fudging things as it is. There’s no good gesture for measuring when people “shake it all around”, which is something that I’m currently looking for a solution for. I’m trying to tweak the solution I had for last week (a really sensitive turn left gesture) but given the fact that people were somewhat confused and didn’t shake in a way that triggered the event, I think that should be changed.

For now, I think I’ll implement a timer that’ll trigger the next screen after a certain amount of time passes. As long as people are really into it, nobody will notice, right…?

(also: it’s april fools! best. day. ever.)

I got my first demo working!

It’s the hokey pokey. I would post a link to it here…but you’d need a kinect/faast in order to get it working. If there’s interest maybe I’ll make a YouTube video of somebody else doing it later.

For whatever reason, I’m having a hard time detecting NITE gestures: mainly wave. I wanted to use it for when you “shake [your arm] all around” but it didn’t work. I ended up fudging it by making a very very very sensitive turn left gesture that’ll go off when the person even thinks about turning left…like what kinda happens when someone shakes their left arm like they really mean it. I’m hoping to figure out what this problem is soon and fix it, since waving would be a nice gesture to support.

Furthermore, this demo is slightly annoying since somebody has to be at the computer at all times. Which works if it’s in an environment like the CS Luncheon where one can talk to people and use the downtime between demos to talk and discuss what is actually going on, but would not work in a hallway/set it and forget it type of demo. I’m thinking the only solution to this would to be have Flash open a port listener and listen for when FAAST starts broadcasting the user’s skeleton. While this wouldn’t be the greatest of solutions, it would at least allow some sort of detection as to when a user is present…either that, or maybe try using iHart’s face tracking assuming iHart is compatible with the kinect. Something to explore over the next few weeks as I perfect this demo, I suppose.

Last time I posted here, I had gotten the GestureEvent/GestureManager classes working. Since then I’ve tweaked them a bit so they support different types of gestures, instead of only leaning left.

From the flash side of things it’s really easy to add a new gesture – all you have to do is define a new constant variable (like LEAN_LEFT) and add it to GestureManager’s keylistener. From there, all you have to do to envoke the listener is use the following code:

gesMan.addEventListener(GestureEvent.LEFT_ARM_FORWARD, eventListener);

…it’s pretty straight forward.

FAAST, however, is a different story. While implementing gestures is fairly simple, sensitivity is somewhat of an issue. It’s hard to get gestures so that they’re perfectly balanced between not being overly sensitive/not being sensitive enough. I’m going to try and program a tech demo for the CS Luncheon next week, so we’ll see how that goes.