My ps3 is connected to my pc monitor using the vd-z3 transcoder(dip switches 2, 5 on) . I'm using the ps2 component cables(since the ps3 only comes with composite ) I've got my PS3 display settings set to use component and 720p because I'm using a proview monitor with maximum resolution of 720p. I can see the XMB and play Resistance Fall of Man with no problems(although it is waaaaaaay too dark, even with brightness turned up, but I gave up trying to figure that one out awhile ago) but when I put in Talladega Nights the screen just goes blank and flashes "mode not supported" every once in awhile. I can still hear audio, I just don't get a picture. I think its cause its trying to play the movie at 1080p and my monitor doesnt support it, but I can't find a resolution setting for movie playback. I have no idea how to get it to play on my monitor, anybody know what I'm doing wrong?

your monitor is not hdcp compatible i think..do a research on hdcp n find out if you monitor is hdcp compatible on you manual/specifications

I don't think it has anything to do with hdcp compatibility of component.

The problem is your transcoder cannot do 1080p, and it looks like the movie is trying to play in that resolution.

Uncheck the "1080p" option in your video settings.