With multiple desktop environments, you just choose the one you want at
login, and that's the one that will manage your X session. You could
remove parts of gnome, if you're careful not to remove any gnome
libraries which are needed by the system, but if disk space isn't tight
you can leave them all there.