Understand that MKV/AVI is not a video format. It's a container format into which video and audio streams can be placed.
For example, you can have an AVI and an MKV file, each using the exact same DivX video stream. The X-Box will find one as easy to play as the other.
So why are MKV files typically jerky? Because MKV supports more sophisticated video codecs, such as x264, while AVI does not. x264 is not what you call X-Box friendly.
Lesson number two: 720x408 is not
HD. You know how the higher display modes are referred to as 720p and 1080i? Well, the numbers refer to the amount of ROWS in the video image (as opposed to the number of columns). Your video has 408 rows; it's tiny (standard NTSC low-def is 480 rows, PAL is 576). 720p is usually 1280x720.
You might be able to get satisfactory results by trying a different video player. XBMC has two built in. If you select your video and hit the white button, you can use the "Play Using" option to choose between them.
MPlayer is the default for most videos, as it supports more codecs and provides better picture quality, subtitle and streaming support.
DVDPlayer, on the other hand, seems to provide better frame rates then MPlayer is capable of.
Worth taking a read through this thread