Making a guitar for the Kinect SDK

Moto is my graduate project on my university course. It involves playing air instruments on the Kinect by mapping them to your body. But of course no band would be complete without at least one guitarist, so that’s what I’ll show you now.

How it functions

Now, the way a guitar works in Moto is much like how they work in other music games like Guitar Hero – You’ve got five zones which, when hit, produce a different sound. It’s like a dumbed down version of a real guitar. Trying to emulate a guitar would be ridiculous of course as, well, you might as well just play a guitar.

In principle the further away your hand is from where you strum the lower the note. So, in Moto, five zones will be set up exactly for that. But the problem is this – How do you know where their hand is without any physical buttons to push? The answer? Pythagoras.

Pythagoras 101

a2 + b2 = c2

You remember that from school, right? The length of the hypotenuse squared is equal to the square of the other two sides. Triangles and that.

Here’s me demonstrating what I mean. Hello.

Just to confuse you, I’m using x, y and z. It still means the same thing. x2 + y2 = z2

So, for arguments sake, say x is 40cm long and y is 30cm long.  302 x 402 = 900 x 1600 = 2500. The square root of 2500 is 50. So z is 50cm. Simples.

Taking that with more of a view of the Kinect SDK, say I’m strumming at a point where x = 0 and y = 0 – the co-ordinate (0,0) – and my other hand is at the point (-30, -40). The same thing still applies. –302 x –402 = 502. When you square a negative number, it becomes positive. The length is still 50cm, regardless of direction.

This rule still works whether I’m close or far away from the point where I strum. Z – the line – gets shorter and longer. It’s from that I can work out exactly where my other hand is in relation to the strumming hand.

However, what happens if the player rocks out too hard? They’re not always going to be facing the Kinect head on and might have their hand further away from the Kinect than the strumming hand. Depending how far they turn round, the x value for the other hand is going to get closer and closer to the strumming hand, which will make the z length shorter. Solution? 3D Space.

Yeah. I know. It gets confusing. This diagram doesn’t really help much either, but hear me out.

True, more working out is needed, but instead we’re going to get a better handle on how far the two are apart. All we need to do, though, is do Pythagoras twice. Once to get the length of x in a bigger triangle, which then gives us the z we are looking for.

Consider the picture example above. Our strumming hand is closer to the Kinect than our other hand. So we’ve drawn a cuboid around to connect these points. The first triangle we’re going to look at uses x1 and y1 to get z1. x1 is the difference in x between the strumming hand and the other hand, and y1 is the difference in depth between the strumming hand and the other hand. This gives us z1 – the base of our bigger triangle.

When we’ve worked out z1, this gives us the base of our second triangle, which we’ve labeled x2. y2 is the difference in height from the strumming hand and the other hand. Doing Pythagoras’ theorem on this gives us z2 – the length we want.

I know it’s not straight forward, but it could be a whole lot worse. But let’s work out an example. The strumming hand is at (0, 0, 1) and the other hand is at (40, 60, 1.5).

(0 – 40)2 + (1 – 1.5)2 = –402 + –2.52 = sqrt(1600 + 6.25) = sqrt(1606.25) ≈ 40.08

So 40.08cm is the length of z1, so that’s also what x2 is.

40.082 + (0 – 60)2 = 40.082 + –602 = 1606.25 + 3600 = sqrt(4906.25) ≈ 70.04

So z2 – the distance between both our hands - is 70.04cm or there about. Phew.

It seems a whole lot of effort to get that tiny bit of information, but from there we can work out what note to actually play.

C-sharpen up

Within Moto I’ve already got a few functions predefined so I can use them for many things like hand tracking. We’re just going to set one up to work out Pythagoras so, if we need it again, we can use it.

private double doPythag(double a, double b)
{
  double c = Math.Sqrt(Math.Pow(a, 2) + Math.Pow(b, 2));
  returnc;
}

So now doing two lots of Pythagoras isn’t really a problem. You give it a and b (without squaring them first) and it’ll pump out the square root of c2 – c. Sweet.

The rest of the guitar essentially borrows code from the drums side of things – it defines an area that it waits for either hand to enter and, when it does, play a sound. But we’ve only got one box, the strumming area called strumArea which, you guessed it, is where we strum with our right hand by default.

double[] strumAreaStart = new double[3];
double[] strumAreaEnd = new double[3];
bool insideStrumArea = false;

internal void defineStrumArea() {
  double strumSize = 0.4; //Size of strum area edges (in metres)

  strumAreaStart = new double[] { Convert.ToDouble(skeleton.Joints[JointType.HipCenter].Position.X) - (strumSize / 2), Convert.ToDouble(skeleton.Joints[JointType.HipCenter].Position.Y) - (strumSize / 2),Convert.ToDouble(skeleton.Joints[JointType.HipCenter].Position.Z) - strumSize };

  strumAreaEnd = new double[] { strumAreaStart[0] + (strumSize / 2), strumAreaStart[1] +(strumSize / 2), strumAreaStart[2] + strumSize };

  SetStrumPosition(strum1);
}

We’re setting up our strum area at the moment. This function needs to run on every new frame as it’s based on the player’s hip position and so needs to constantly know where it should be checking. It’s just defining numbers which the X, Y and Z values of the hand will need to fall into. setStrumPosition() is positioning a graphic on screen. It’s not all that important really.

Next comes the number crunching – the stuff we work out at the start, but now in glorious code! We’re separating that into its own little function for the sake of clean looking code.

internal double checkNeckDist(Joint hand)
{
  Joint hip = skeleton.Joints[JointType.HipCenter];
  double xLength = doPythag((hip.Position.Z - hand.Position.Z), (hip.Position.X - hand.Position.X));

  double neckDist = doPythag(xLength, (hip.Position.Y - hand.Position.Y));

  return neckDist;
}

All this is doing is doing what we worked out earlier. If you need a refresher, hit it up above. :)

Lastly, tying it all together, is another function we are running on every frame. This is checking whether the supplied Joint lies between the X, Y and Z values that we’ve outlined earlier.

internal void checkStrum(Joint joint)
{
  //Did the player strum just then?

  double posX = joint.Position.X;
  double posY = joint.Position.Y;
  double posZ = joint.Position.Z;

  if (strumAreaStart[0] < posX && strumAreaEnd[0] > posX && strumAreaStart[1] < posY &&strumAreaEnd[1] > posY && strumAreaStart[2] < posZ && strumAreaEnd[2] > posZ)
    {
      if (!insideStrumArea)
      {
        if (instrumentInUse == 1)
        {
          //Normal guitar stance
          strumGuitar(checkNeckDist(skeleton.Joints[JointType.HandLeft]));
        }
        else if (instrumentInUse == 2)
        {
          //Lefty stance
          strumGuitar(checkNeckDist(skeleton.Joints[JointType.HandRight]));
        }
        insideStrumArea = true;
      }
    }
    else
    {
      insideStrumArea = false;
    }
}

instrumentsInUse is a variable within Moto which, funnily enough, tells the program which instrument the player is currently playing. 1 is Guitar and 2 is Lefty Guitar, where the player strums with their left hand instead. We are passing strumGuitar the value we’re generating in terms of distance from the hip to the hand. The bigger the number, the lower the sound should be. That’s all strumGuitar is doing.

That’s it really. Not all that complex. That’s how it works! :)