I have this happen in Kubuntu. The keyring involved is KDE Wallet. It is
for the storage of passwords and encryption keys. Once it is opened and the
keyring password is given then it will not ask for the keyring again until
you logout. I do not use XFCE so can't tell what the keyring service is. In
KDE you can shut off Wallet in the settings and the nag goes away. It is in
System Settings/ Account Details. I am not sure what keyring service in
XFCE is, but GNOME has one. I suspect that it is the gnome-keyring.

docs.xfce.org/xfce/xfce4-session/preferences

I would check to see if the Launch GNOME Services on Startup box is checked
and try unchecking it. See the above link to find that in the XFCE
Preferences.

I would note that keyring managers are important because they store
encryption keys and are useful in backing up and restoring passwords. To
each his own. :)