So if the source video (avi itself) is in a high resolution it very well may be choking the Xbox. The XBox has no problem scaling a video to 720p/1080i from a lesser resolution source but playing native HD files is just a bit much at times. 1080i native files have never worked. Some people have had success with 720p or close to that point resolutions, maybe mess with the cache a bit (i.e. lower it for more memory access) or expiriment with both cache and playing from the HD or a wired LAN vs. wireless if you are streaming the file.
My setup has upscaling to 1080i to match my television. I am routinely able to play 960x528 high resolution files with AC3 sound that I download from Bittorent. Basically anything here that says HR.HDTV in the title: http://10mbit.com:69...?filter=TV Show
. They play absolutely perfectly unless I downsize them into a tiny window (i.e. the display button on the remote) and start browsing around in which case I've gotten stuck twice.
So that should get you moving in the right direction. I use XBMC version 1.1, some of the newer releases might be a tad more efficient in memory and might give you an edge, I know upgrading to 1.1 allowed 5 megapixel pictures to be handled in a slideshow at 1080i where before I used to have to back out each time under much older versions. To my knowledge XBMC doesn't make use of more than 64MB of Ram so even if you upgrade the XBox to 128MB I don't believe it will utilize it.
Some stuff to consider.