An example of a misbehaving IK handle due to mis-aligned joints.
bad_leg_IK_video good_leg_ik.aviBefore continuing, if your mesh is skinned please save your skin weights and detach the skin from the skeleton. We are going to be moving joints, and adjusting joint orients will ruin your mesh if you left it skinned.
The reason the IKhandle is getting confused is because it only wants to bend the joint in one axis, but if the middle joint (knee joint) is out of alignment in two axis then the IK solver won't work correctly.
In the side view of the above image the knee is forward, this is good because it tells the IK handle that when you move the ankle you want the knee to bend forward. In the front view the good leg is perfectly straight so it will bend correctly, the bad leg is bent in a second axis so it will not work with the ik handle.
So, how do we fix this?
1.First hide any meshes by going to Show>Polygons in the current panel your working in.
2. We want to draw a straight line from the first joint to the third joint using the Create>EP Curve Tool and open options for this tool by clicking on the box next to the name. Set curve degree to 1 ( linear).
3. Now hold down the V key (snap to point) and click on the first joint and the third joint.
You should now have a straight line between the two joints. This will be our guide that we willl snap our knee joint to.
4.Select the knee joint and press the insert key this will cause only this joint to move and not any of it's children.
Now hold down the C Key (snap to curve) and move the knee joint over the curve until it snaps to it.
5.Once you have the knee joint where you want it on the curve we're going to let go of the c key and move the knee forward in only one axis.
Yay, now your joints are aligned in one axis, and by moving the knee slightly forward it knows which way to bend as well.
6. Now since we moved the knee joint we need to fix our joint orients, so pick the leg joints and use the skeleton>orient joints tool to re-orient them.