Most likely if you are someone who bought an XCM case, chances are you have met the toughness it takes to remove the faceplate.
I was sitting here thinking about the problem and found an easy fix.

Just take an Xacto to each of the latch holes on your case. (7 of them)

What you are trying to do is shave any excess plastic from the holes. Use your stock case as a comparison. Use the corner of something to check the hole, if it sits level in the hole, you are set.
Once you get all 7 cut to the right amount, put the faceplate back on and take it off. If there still is more force than there should be, check your holes in the area of the force. Try shaving extra off.
This fix is a lot easier if you still have your stock case lying around to check your holes.