[bitfolk] The immutability of backups, revisited

Hello, This is a long email about BitFolk's free backups service. If you don't use that then you probably want to skip it. In mid-2010 I asked what you thought should be done about the backups when a customer accidentally backs up up some large amount of data that they didn't mean to: http://lists.bitfolk.com/lurker/message/20100528.061800.6814d4ed.en.html The TL;DR version of that thread is: - I don't want customers to have write access to their backups because a) It means the backups can't be trusted after a compromise; b) If they're not writable they can't be trashed accidentally in some other way; c) I don't want backup space used for general purpose storage. - The default backup schedule keeps things for 6 months. That means that once something is backed up, it is there for 6 months in backups even if you delete the file from your VPS's disk (or exclude from backups) as soon as you notice. When someone blows all their backup quota on some data they didn't mean to back up[1], they probably don't want to have to buy more disk space in order to keep their backups working. I imagine they would rather have the accidentally backed up data deleted. But they can't because they don't have write access. - I don't want BitFolk support to be poking around in people's backup data trying to delete things. The potential for mistakes and misunderstandings is too great. - I proposed one method of dealing with the situation which would be that BF support would delete entire iterations of backups until the usage went below quota again. If you're quick it might only be the last 4 hours, or the last day, but this is still rather harsh. - Other proposals were based around BitFolk developing tools to do limited file operations on the backups. This could be done, but it seems like a lot of work for a relatively rare situation. I'm still hoping for a better idea. Since that thread took place we've had a bit more experience with the issue. In most cases as long as we could see that the disk usage had gone down and would eventually drop below the quota once some of it cycled off the end, it was easier to just not enforce quota. In light of this, one new proposal that I have is that the disk space for backups should be a lot cheaper. BitFolk has an abundance of spare disk space that isn't selling. I don't think I want to reduce the cost of general purpose disk space (it's already quite low), but it seems only fair that disk space for backups -- which is of limited use -- should be cheaper. Hopefully this will encourage larger quotas for backups and make accidental backups a lesser issue, since it will impact you less to just have them cycle off the end. My next proposal is that it might be an acceptable risk to allow customers, via panel.bitfolk.com, to temporarily grant themselves write access to their backups. I'm suggesting it remains writable for just a few hours and then it automatically goes read-only again. Since that would be logged it would be possible to still be confident about the sanctity of the backups after a compromise. Do these proposals sound reasonable, or at least better than what we have now? Does anyone have any other ideas? Cheers, Andy [1] At this point it is tempting to ask, "why does the backup facility even let a customer back up more data than their quota allows?" The answer is that it is relatively expensive to calculate how much disk space has actually been used for backups, so it's not possible to do so at every backup run. However, even if it were possible, it doesn't really get us anywhere with this problem: once you back up a huge amount of stuff that you didn't mean to then you're still in this predicament whether that is 150% of your quota, 100% or 95%. You don't want it there, using up space for 6 months. So let's leave that issue for another day. It is documented: https://tools.bitfolk.com/redmine/issues/38