I have done it. This home directory I am using now was once for 64-bit Gutsy.
Now it is home for 32-bit Hardy. I have also done this in the opposite direction
for SimpyMEPIS 7.0. If I remember correctly the MEPIS one messed up my x
settings, but I just manually edited xorg.conf.
If you are worried about any possible crashes or faulty settings then you can
use the same partition and just change the user name. Then drag and drop folders
from the old user's home folder into the new, incrementally. The main ones to
worry about are your documents which is no problem, bookmarks (again no
problem), mailboxes (no problem), etc. The ones most likely to give problems are
KDE and Gnome. The biggest problem was that KDE and Gnome menus had lots of
useless programs which can be manually edited out or they can be re-installed if
there are equivalents in 64/32-bit.