I travel quite a lot and do not always have access to all my keys. Although I do not often have to access the Xen shell, it would be very difficult if both were key only. It makes absolute sense to make port 22 key only, but if 922 were password as well it would be helpful. I use nonstandard ports for access to SSH on my VPS and as soon as I changed it I noticed a big change in my fail2ban logs.
One other thing I have done is to set up ipset, No one should ever attempt to connect to port 22 on my machines so I have iptables add any that do to an ipset which is then blocked from any port. But that may not be possible for the Bitfolk set up