Re: [bitfolk] predicted network usage / "Predicted to exceed transfer quota"

Hi Conrad, On Fri, Feb 25, 2022 at 04:08:32PM +0000, Conrad Wood wrote: Yes. The prediction algorithm was explained here: https://lists.bitfolk.com/lurker/message/20220223.204720.d86cf277.en.html The two things (emailed notifications about data transfer, and the figures on the panel about data transfer) have their predictions calculated differently. For the email notifications: - First it does a lightweight prediction by taking the sum of data transferred this period, averaging it to a per-second value and then multiplying that by the number of seconds in 30 days. - If: - This prediction indicates a state transition, either from "predicted okay" to "predicted over" or the reverse, AND - You're in the first 15 days of the period, THEN: a more heavyweight prediction is done by adding up the actual figures for the last 30 days. That is then used as the prediction. For the panel: - Just uses the simple lightweight prediction. Reasoning: If we're going to send a notification about a state change, either to tell you to worry or to tell you not to worry, it would be better to use more accurate figures for that prediction. The lightweight prediction might be using only a small amount of data. In your case right now about 3 days' worth to guess about a 30 day period. If you look back at the last email notification sent to you, you'll see that it predicted you'll go over again this period, and its predicted figure matches what our Prometheus is now telling you, because that's always how it's worked. It's true that the simple prediction for you is right now returning a much lower figure. You can look back in your bandwidth graph and see why that is: There was a prolonged period of high usage a while ago, but within the last 30 days. If we ignore that period then of course the prediction will come out much lower. Is it correct to ignore that period? Who knows, we just have to pick an algorithm, you could argue either way. So why not always use the more accurate prediction on the panel as well? The reason is historical. The figures are stored in a regular SQL database and summing up 30 days of metrics from that source takes an appreciable amount of time. This setup pre-dates Prometheus or any other sort of time series database we had going. So, we have only been doing it when it's considered important to do so, in the batch job that sends out email notifications about data transfer limits, and then the calculated predictions were thrown away. It wasn't considered feasible to do those calculations in a web page. By the time you are 15 days in to this reporting period the two figures will match. So, what you are seeing is expected but it doesn't mean we can't improve things. I do not want to calculate the official billing figures from Prometheus because I don't want to make Prometheus (or Grafana) an essential piece of infrastructure. I could store the calculated predictions back in the database and use those on the panel. It's always worked like this and no one has really noticed in more than 10 years, probably because hardly anyone reaches their limits and when they do it's usually nearer to the end of the reporting period. It should be pretty easy to store the calculated predictions though, I've just never thought of doing it before. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting