You'd have to supply to the Panel some list of net blocks that
>     you will SSH from and then there'd be a button to punch holes in
>     the firewall for SSH from those net blocks for 6 hours (for
>     example).

>     There would have to be a limit on the size of the net blocks.
>     Let's say a /16 for IPv4 and a /32 for IPv6.

Perhaps a slight modification to the this option:

On some systems I set up  a system where nftables opens the ssh port temporarily and only after successful authentication via wget/https to either port 24 or port 443.

I use port 24 because it is not one I usually use and less likely to be fiddled with nat/broken firewalls.

The server uses the IP address 'wget' is coming from to open port 22 to that source ip. (v4 and/or v6).
I can think of scenarios where weird nat/firewalls break that, but I have not encountered such network in the past 5 odd years when I started using it.
The timeouts I am using is 3 minutes until initial connection is made and 1 hour after that.

When I travel and are on dynamic IPs, I sometimes start a shell script to wget every 2 minutes to keep my IP white-listed.

FWIW I also use wireguard and openvpn as backup methods. (I am aware of some networks where this algorithm would fail, I just don't happen to be on any of those recently).

Conrad