Re: [bitfolk] predicted network usage / "Predicted to exceed transfer quota"

Hi Conrad, On Wed, Feb 23, 2022 at 08:40:12AM +0000, Conrad Wood wrote: You have made this request before, about two years ago, so it's worth reading that thread as well: https://lists.bitfolk.com/lurker/message/20200214.095931.838386c0.en.html I think it's probably also worth looking at what say Linode offers here: https://www.linode.com/docs/api/linode-instances/#network-transfer-view which looks to be very similar to what you are asking for. The chief problem here is that I don't know how to make an API server, which is still the same problem as 2 years ago. Obviously I can learn, and I appreciate that the information you request is simple and that you'll probably be quite tolerant of any changes that needed to happen to the API, but it's still far outside my experience. The motivation to do this work is quite low as: - I think this particular thing is a very niche requirement (people hardly ever approach their data transfer quotas) so this benefits very few customers. - The best source of this data is from within your own VMs. - The rare times that people do go over their quota they tend to focus on BitFolk's measurements of this instead of looking into the behaviour of their own VM. I feel like anything I implement is just going to be a further distraction from the customer's own responsibilities. I encourage customers with their own monitoring to entirely switch off BitFolk's data transfer monitoring, monitor it themselves and put it in their alerting system. A simple rolling 30 day count of bytes transferred shouldn't be that difficult. The emailed warnings are from data calculated every 15 minutes. During the first 15 days of the time period under consideration, the prediction is the sum of the last 30 days. In the last 15 days of the time period the prediction is a simple average over days so far multiplied by 30 days, e.g. if you're 20 days in then the prediction for this 30 day period is the sum of the last 20 days divided by 20 multiplied by 30. The figures on the panel regarding count of octets in/out are updated every 5 minutes. The predicted values there are always just the current period's figures averaged and then multiplied, e.g. if you're currently 20 days in then it will be the sum for the last 20 days divided by 20 multiplied by 30. So, if you are wondering how come a warning can be sent very quickly after a new data transfer period begins, it is because for the first 15 days of that period the predicted total comes from the sum of the previous period. Like if you are one day in then the prediction is the sum of all bytes transferred today and for 29 days before it. No, and to get one will require a lot of work for me. Which I appreciate doesn't seem like much work to people who know about these things and perhaps do them for a living, but it is for me. Is there a problem here where you suspect that BitFolk's figures do not match with your own? If so, there is of course always the potential for a mistake, so we could talk about that in a support ticket. BitFolk's figures should be a little bit less than your own packet counters as we disregard traffic to other parts of BitFolk. It would be unexpected for BitFolk's figures to indicate MORE data transferred than your own figures. As a first step it would be best to look at the total count of bytes transferred between the two date points to see if they generally match up. Or is it that if you are warned by BitFolk that you are predicted to exceed your quota, you want to be able to assure yourself what the scale of that will be, or if you have solved the issue and so on? Again I would much rather you do that using your own figures. A simple count of bytes transferred from a given start time should generally match (bearing in mind BitFolk's under counting as mentioned above) and should be able to give you some idea. I can understand how if you're near the quota, have been warned that you're over the quota or have actually gone over the quota, that you might want a more real-time insight into what BitFolk considers the figures to be, without reloading the panel web page or waiting for an email about it. I am interested in what other customers think. Though as most people never get anywhere near their data transfer quota, before responding please do try to put yourselves in the position of someone who has just received the automated email that says they've gone over quota and every GB transferred is now costing them £0.06+VAT. Does the current facility of the panel web page, Grafana site and emailed notifications cut it then? Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting