As mentioned earlier, the GPU (graphics hardware) is locked in otherOS. Theres not enough power to render via software. You might get some less demanding codecs running close to smooth, but XViD -> h264 would never work well enough for it to be worth it. Theres far better solutions already out there, i.e popcorn hour or other NMT-based media solutions.
Ofcourse it would be great to have a working mediacenter on PS3 but I doubt it will work well enough to be worth it.