One thing that I would try is to make sure that you actually let X handle the selection. If you have set mouse=a in your vimrc file then vim will interpret the selection as its visual mode. If this is the case, try holding Shift when selecting.
Your terminal emulator may have its own mechanism for copying and pasting, for example gnome-terminal uses Ctrl Shift c for copying and Ctrl Shift v for pasting (as Morlock stated in his answer). You can use that instead.