Neither am I for most things. I can do Apache via ssh with no problem (apart from the one time pointing the logfiles at the wrong place for one site meaning that Apache refuses to start, but everyone does that once... don't they?) for example. 

But I am not totally confident that I will not drop something talking to MySQL that way, and phpmysqladmin or whatever it is called is hideous with far more exploits over the years. 

What I have done, now, is restrict webmin to just respond to one IP address. If I ever need it while I am away, I can use ssh and edit the conf file. 

Ian