I think I figured it all out. I'd like to post my solution even though i'm the only one posting on this thread in hopes that in the future someone might come along this in a search engine and it will help them.
First of all, the reason the xbox wouldn't play the original files video was because of the reference frames that I mentioned earlier. I download a free piece of software called MediaInfo
and opened up the file. That program showed me that the original mkv had 8 reference frames and xbox is limited to 5. Don't ask me what ref. frames are because I don't know, but I do know that xbox files can only have 5. Unfortunately this meant I had to re-encode. I downloaded another free piece of software called Handbrake
and re-encoded the mkv into another mkv except I went over to the Advanced tab and changed the reference frames to 2. Again I don't know what benefit using more or less has, but after all this was said and done the quality is perfect 1080p so i'm not complaining.
The encode took about 2 hours (which is much improved over 10 hours previously) and once finished I remuxed the file again to a .ts to be played with XBMC. Unfortunately the .ts file still played very choppy and crappy, but this time the picture did work! So the ref. frames thing was the issue. Now back to the playback, as mentioned before, for me the stuttering and choppiness was horrible and unwatchable so I found another, even more convenient solution and that was to download Haali Media Splitter
which enables Media Center to see and play mkv files. From here I could officially open and play my 1080p video file in XBMC! Unfortunately there was still constant stuttering every couple minutes, but it was barely noticeable.
I did some more research and found that the xbox works best when 1080p files are being transferred at no more than 10 mbps. Really I don't know much about how to change that, but I did find a solution from another thread located here: http://cybernetnews.com/media-center-exten...treaming-video/
Basically, I had to enable flow control on my network settings and to do that on Windows 7 I went to the Control Panel, then clicked on Network and Sharing Center, then clicked on my Local Area Connection, then Properties, then Configure, Clicked the Advanced Tab, went down to Flow Control and chose enabled from the drop down and then also went down to Speed and Duplex and changed it to 100 mbps Full Duplex rather than leaving that on its auto setting. Since then I have watched half of a full 1080p movie and haven't noticed any skipping and stuttering.
Since my method required a re-encode anyways, and that process is pretty short it may be worth while to just downgrade the mkv to 720p at that point, but video files are really the only 1080p content I watch and it would be a waste of my fancy tv not to take advantage of that option. Personally, I doubt I would notice much of a different in quality though.
Hope that helps someone out. I'll post more if I run into any other problems, otherwise assume I have fixed my issue.