23 Feb
2022
23 Feb
'22
8:40 a.m.
Hi,
Sometimes some of my services misbehave in weird and wonderful ways.
If that happens, they can use an awful lot of bandwidth.
Whilst there is some monitoring that I can do (and do), I prefer to use
the data "closer to source".
There seems to be some data, that triggers pretty quickly and sends out
an email from "xfer a_t bitfolk.com" with the heading "Predicted to
exceed transfer quota".
However, rather than email I'd like an API, so that I can route it to
my alerting system. I do overlook emails sometimes, especially computer
generated ones.
My brain sometimes scans and classifies them too quickly as "deal with
it later" and incorrectly flags the email as 'not important'.
Having the information in my alerting system means I can route them so
that they remind me/trigger me again etc.
My questions are:
1. How often is that prediction calculated?
2. Is there an API I could query which gives me the data from the
email?
2.1 I'd settle for a binary API (predicted overusage == true|false)
2.2 otherwise an api that gives me the following values:
- predicted usage in
- contracted usage in
- predicted usage out
- contracted usage out
- Actual out
- Actual in
Nothing fancy required - a simple 'get' request or so would do just
fine ;)
23 Feb
23 Feb
8:47 p.m.
Hi Conrad,
On Wed, Feb 23, 2022 at 08:40:12AM +0000, Conrad Wood wrote:
rather than email I'd like an API, so that I can
route it to
my alerting system.
You have made this request before, about two years ago, so it's
worth reading that thread as well:
https://lists.bitfolk.com/lurker/message/20200214.095931.838386c0.en.html
I think it's probably also worth looking at what say Linode offers
here:
https://www.linode.com/docs/api/linode-instances/#network-transfer-view
which looks to be very similar to what you are asking for.
The chief problem here is that I don't know how to make an API
server, which is still the same problem as 2 years ago.
Obviously I can learn, and I appreciate that the information you
request is simple and that you'll probably be quite tolerant of any
changes that needed to happen to the API, but it's still far outside
my experience.
The motivation to do this work is quite low as:
- I think this particular thing is a very niche requirement (people
hardly ever approach their data transfer quotas) so this benefits
very few customers.
- The best source of this data is from within your own VMs.
- The rare times that people do go over their quota they tend to
focus on BitFolk's measurements of this instead of looking into
the behaviour of their own VM. I feel like anything I implement is
just going to be a further distraction from the customer's own
responsibilities.
Having the information in my alerting system means I
can route them so
that they remind me/trigger me again etc.
I encourage customers with their own monitoring to entirely switch
off BitFolk's data transfer monitoring, monitor it themselves and
put it in their alerting system. A simple rolling 30 day count of
bytes transferred shouldn't be that difficult.
My questions are:
1. How often is that prediction calculated?
The emailed warnings are from data calculated every 15 minutes.
During the first 15 days of the time period under consideration, the
prediction is the sum of the last 30 days.
In the last 15 days of the time period the prediction is a simple
average over days so far multiplied by 30 days, e.g. if you're 20
days in then the prediction for this 30 day period is the sum of the
last 20 days divided by 20 multiplied by 30.
The figures on the panel regarding count of octets in/out are updated
every 5 minutes. The predicted values there are always just the
current period's figures averaged and then multiplied, e.g. if
you're currently 20 days in then it will be the sum for the last 20
days divided by 20 multiplied by 30.
So, if you are wondering how come a warning can be sent very quickly
after a new data transfer period begins, it is because for the first
15 days of that period the predicted total comes from the sum of the
previous period. Like if you are one day in then the prediction is
the sum of all bytes transferred today and for 29 days before it.
2. Is there an API I could query which gives me the
data from the
email?
No, and to get one will require a lot of work for me. Which I
appreciate doesn't seem like much work to people who know about
these things and perhaps do them for a living, but it is for me.
Is there a problem here where you suspect that BitFolk's figures do
not match with your own? If so, there is of course always the
potential for a mistake, so we could talk about that in a support
ticket.
BitFolk's figures should be a little bit less than your own packet
counters as we disregard traffic to other parts of BitFolk. It would
be unexpected for BitFolk's figures to indicate MORE data
transferred than your own figures.
As a first step it would be best to look at the total count of bytes
transferred between the two date points to see if they generally
match up.
Or is it that if you are warned by BitFolk that you are predicted to
exceed your quota, you want to be able to assure yourself what the
scale of that will be, or if you have solved the issue and so on?
Again I would much rather you do that using your own figures. A
simple count of bytes transferred from a given start time should
generally match (bearing in mind BitFolk's under counting as
mentioned above) and should be able to give you some idea.
I can understand how if you're near the quota, have been warned that
you're over the quota or have actually gone over the quota, that you
might want a more real-time insight into what BitFolk considers the
figures to be, without reloading the panel web page or waiting for
an email about it.
I am interested in what other customers think. Though as most people
never get anywhere near their data transfer quota, before responding
please do try to put yourselves in the position of someone who has
just received the automated email that says they've gone over quota
and every GB transferred is now costing them £0.06+VAT. Does the
current facility of the panel web page, Grafana site and emailed
notifications cut it then?
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
11:37 p.m.
On Wed, 23 Feb 2022 at 20:51, Andy Smith <andy(a)bitfolk.com> wrote:
- The rare times that people do go over their quota
they tend to
focus on BitFolk's measurements of this instead of looking into
the behaviour of their own VM. I feel like anything I implement is
just going to be a further distraction from the customer's own
responsibilities.
This matches my experience. I pay reasonable attention to the quota
emails but don't do any monitoring myself because I don't expect to be
at any risk of going over the quota. From a customer perspective
though, I expect the measurements made by bitfolk to be the gold
standard in terms of billing so my measurements would ultimately be
secondary.
I did have a small amount of overage, back when the outgoing bandwidth
limit was 400GB. The timeline of emails looked like this:
23 March 1/4 usage email - used 94GB, projected 375GB (I doubt I noticed this)
27 March Predicted to exceed email - used 206GB, projected 552GB
29 March Overage email - used 401GB
30 March 2/4 usage email - used 407GB, projected 812GB
7 April 3/4 usage email - used 411GB, projected 548GB
14 April 4/4 usage email - used 415GB
I got a potential projected 412GB overage down to 15GB overage based
on those emails, so I was happy. It might be that email isn't the best
medium for everybody though. I can't remember whether I made much use
of the (as then) cacti graphs, I think the numbers in the emails were
the key factors.
Cheers,
Roger
24 Feb
24 Feb
12:09 a.m.
Hi Roger,
On Wed, Feb 23, 2022 at 11:37:40PM +0000, Roger Light wrote:
On Wed, 23 Feb 2022 at 20:51, Andy Smith
<andy(a)bitfolk.com> wrote:
It reads like you actually don't agree with me. What I was trying to
say was that I expect customers to actually be able to measure
their own bandwidth usage, and BitFolk telling them that they are
near or over quota is just a courtesy.
What you are saying is that you don't expect to have to measure
this yourself, you would never be aware of it or investigate it
without a notification from BitFolk and therefore these
notifications are essential to you. You did get the notification and
were able to take action, but only because email notifications
are satisfactory for you.
If most people are the same then this means that receiving and being
able to act upon these notifications isn't just a courtesy, it is
essential. The fact that you were able to is great, but the fact
that others aren't is of concern to me.
Conrad was saying that they do have their own monitoring but that
the email notifications and the figures on the panel web page aren't
satisfactory because they are hard to be aware of (emails get
overlooked) and don't provide easy spot checks.
Is there any middle ground that could improve things for you,
Conrad, without me having to make an HTTP endpoint that gives these
figures?
For example, I could probably plug it in to our Icinga monitoring¹
so that it continually gave you a WARNING email when predicted to
exceed, turning into a CRITICAL when it actually has. Besides email,
our Icinga can also do Pushover² alerts, and I am open to adding
more kinds of alerts there.
Even if that does not work for Conrad, it might be a better
interface for more people so I would be interested in feed back on
that or other ideas.
One of my big problems with the API thing is that I feel like it
will have a very limited audience, very likely at this point to be
only Conrad. I currently manage four Linodes for various reasons and
although Linode has an API I've never felt a need to use it myself.
I monitor my own bandwidth usage. I do often find myself at odds
with others though.
Cheers,
Andy
¹ https://tools.bitfolk.com/wiki/Monitoring
² https://pushover.net/
--
https://bitfolk.com/ -- No-nonsense VPS hosting
- The rare times that people do go over their
quota they tend to
focus on BitFolk's measurements of this instead of looking into
the behaviour of their own VM. I feel like anything I implement is
just going to be a further distraction from the customer's own
responsibilities.
This matches my experience. I pay reasonable attention to the quota
emails but don't do any monitoring myself because I don't expect to be
at any risk of going over the quota. From a customer perspective
though, I expect the measurements made by bitfolk to be the gold
standard in terms of billing so my measurements would ultimately be
secondary.
8:58 a.m.
On Thu, 2022-02-24 at 00:09 +0000, Andy Smith wrote:
Hi Roger,
On Wed, Feb 23, 2022 at 11:37:40PM +0000, Roger Light wrote:
Hi,
I totally forgot I asked it before. Re-reading the thread from 2 years
ago, it seemed to have gone down quite a rabbit hole then.
My focus wasn't on API design or authentication mechanisms - I would
not expect that.
In response to "use the counters on your vps" - I think it is worth
diving into a few details I am dealing with:
1)
The charges incurred (which is what I care about), are not related to
bandwidth but to bytes transmitted/received within a given period.
This means, any tool that I set up needs to be customised for bitfolk.
Specifically, at the very least, it needs to be configured to match the
billing period _exactly_ and reset the predictions at the appropriate
dates.
2)
Also, there is a slight, and quite theoretical discrepancy on what can
be observed on the VPS vs what is counted on the border routers.
Any encapsulation that is stripped by the host or added by the host my
VPS runs on, won't be counted as part of the VPS counters.
3) Whilst the VPS reboots, it won't be able to count the traffic,
whereas bitfolks systems continue to count the traffic. Traffic
received/transmitted during that time cannot be counted in my own
monitoring.
4) Somewhat "invalid" packets that arrive in the host or router but
don't make it to my VPS because they are being filtered on the way for
whatever reason, won't be counted in my stats either. (thinking DoS)
5) In case of the VPS misbehaving, because of an upgrade or software
failure or external attack, the counters on the VPS are no longer a
*reliable* source (I did experience that case before, so for me this
isn't just theoretical).
I _do_ count the traffic, but for the reasons mentioned above it is
less than ideal. It does work well for *most* cases, but clearly once
every 2 years or so it does not :)
One of the ways I could see this working - without having to create a
new API - is to expose it as a metric in bitfolks grafana/prometheus.
if there were metrics (gauges) "traffic_predicted" and
"traffic_contracted", it would be trivial for me to query bitfolks'
prometheus to poll that regularly.
This would be a perfectly suitable API for my purposes and perhaps much
simpler to implement?
This would be preferable (IMHO) to a plugin in icinga, because
prometheus already has a well-defined API, icinga less so (afaik), that
I can use if I (or anyone else) wish(es) to do so ;)
Thanks Andy for your patience (again),
Conrad
On Wed, 23 Feb 2022 at 20:51, Andy Smith
<andy(a)bitfolk.com> wrote:
It reads like you actually don't agree with me. What I was trying to
say was that I expect customers to actually be able to measure
their own bandwidth usage, and BitFolk telling them that they are
near or over quota is just a courtesy.
What you are saying is that you don't expect to have to measure
this yourself, you would never be aware of it or investigate it
without a notification from BitFolk and therefore these
notifications are essential to you. You did get the notification and
were able to take action, but only because email notifications
are satisfactory for you.
If most people are the same then this means that receiving and being
able to act upon these notifications isn't just a courtesy, it is
essential. The fact that you were able to is great, but the fact
that others aren't is of concern to me.
Conrad was saying that they do have their own monitoring but that
the email notifications and the figures on the panel web page aren't
satisfactory because they are hard to be aware of (emails get
overlooked) and don't provide easy spot checks.
Is there any middle ground that could improve things for you,
Conrad, without me having to make an HTTP endpoint that gives these
figures?
For example, I could probably plug it in to our Icinga monitoring¹
so that it continually gave you a WARNING email when predicted to
exceed, turning into a CRITICAL when it actually has. Besides email,
our Icinga can also do Pushover² alerts, and I am open to adding
more kinds of alerts there.
Even if that does not work for Conrad, it might be a better
interface for more people so I would be interested in feed back on
that or other ideas.
One of my big problems with the API thing is that I feel like it
will have a very limited audience, very likely at this point to be
only Conrad. I currently manage four Linodes for various reasons and
although Linode has an API I've never felt a need to use it myself.
I monitor my own bandwidth usage. I do often find myself at odds
with others though.
Cheers,
Andy
¹ https://tools.bitfolk.com/wiki/Monitoring
² https://pushover.net/
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users - The rare times that people do go over their
quota they tend to
focus on BitFolk's measurements of this instead of looking into
the behaviour of their own VM. I feel like anything I implement
is
just going to be a further distraction from the customer's own
responsibilities.
This matches my experience. I pay reasonable attention to the quota
emails but don't do any monitoring myself because I don't expect to
be
at any risk of going over the quota. From a customer perspective
though, I expect the measurements made by bitfolk to be the gold
standard in terms of billing so my measurements would ultimately be
secondary.
1:39 p.m.
Hi Conrad,
On Thu, Feb 24, 2022 at 08:58:49AM +0000, Conrad Wood wrote:
One of the ways I could see this working - without
having to create a
new API - is to expose it as a metric in bitfolks grafana/prometheus.
Okay. I will have a think about this. If it can go in to Icinga then
it can go in to Prometheus.
if there were metrics (gauges)
"traffic_predicted" and
"traffic_contracted", it would be trivial for me to query bitfolks'
prometheus to poll that regularly.
Can do say:
data_transferred_in
data_transferred_out
data_predicted_in
data_predicted_out
data_contracted_in
data_contracted_out
I guess they should be in bytes as Prometheus usually wants the base
unit.
What about the start and end times of the current period? Provided
as unix timestamps?
Thanks Andy for your patience (again),
No worries. Thanks for bringing it up on the list as it's useful to
see what other people think too.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
2:19 p.m.
On Thu, 2022-02-24 at 13:39 +0000, Andy Smith wrote:
Hi Conrad,
On Thu, Feb 24, 2022 at 08:58:49AM +0000, Conrad Wood wrote:
Hi,
That would be very nice. The metrics you suggest will work for me.
I do scrape it from the website now as suggested by Tony (thanks again)
in this thread, but web-scraping does have a few caveats, so pulling
them from prometheus would be nicer.
As for the metrics, I now (as of this morning) put the result from the
web scrapes into my prometheus like so:
bitfolk_network_usage_bytes_current{vps="xxx",direction="in|out"}
bitfolk_network_usage_bytes_predicted{vps="xxx",direction="in|out"}
bitfolk_network_usage_bytes_max{vps="xxx",direction="in|out"}
The choice of labels means I only need a single alert for all VPS' I
monitor and for any direction (in and out):
"bitfolk_network_usage_bytes_predicted >
bitfolk_network_usage_bytes_max"
Of course, the "bitfolk_" prefix is entirely optional on bitfolks'
prometheus.
The above labels lend themselves quite well to alert on absent metrics
as well, should one want to do so.
I don't think the start and end period times are required, as long as
"predicted" and "max" values are provided.
Thanks,
Conrad
One of the ways I could see this working -
without having to create
a
new API - is to expose it as a metric in bitfolks
grafana/prometheus.
Okay. I will have a think about this. If it can go in to Icinga then
it can go in to Prometheus.
if there were metrics (gauges)
"traffic_predicted" and
"traffic_contracted", it would be trivial for me to query bitfolks'
prometheus to poll that regularly.
Can do say:
data_transferred_in
data_transferred_out
data_predicted_in
data_predicted_out
data_contracted_in
data_contracted_out
I guess they should be in bytes as Prometheus usually wants the base
unit.
What about the start and end times of the current period? Provided
as unix timestamps?
Thanks Andy for your patience (again),
No worries. Thanks for bringing it up on the list as it's useful to
see what other people think too.
Cheers,
Andy
2:40 p.m.
Hi Conrad,
On Thu, Feb 24, 2022 at 02:19:07PM +0000, Conrad Wood wrote:
As for the metrics, I now (as of this morning) put the
result from the
web scrapes into my prometheus like so:
bitfolk_network_usage_bytes_current{vps="xxx",direction="in|out"}
bitfolk_network_usage_bytes_predicted{vps="xxx",direction="in|out"}
bitfolk_network_usage_bytes_max{vps="xxx",direction="in|out"}
Okay, I will use this scheme with the direction as a label.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
25 Feb
25 Feb
9:57 a.m.
On Thu, Feb 24, 2022 at 02:40:13PM +0000, Andy Smith wrote:
On Thu, Feb 24, 2022 at 02:19:07PM +0000, Conrad Wood
wrote:
I've now added panels and out and in to everyone's grafana
dashboards using the metric names above, except I used the label
"guest" instead of "vps".
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
As for the metrics, I now (as of this morning)
put the result from the
web scrapes into my prometheus like so:
bitfolk_network_usage_bytes_current{vps="xxx",direction="in|out"}
bitfolk_network_usage_bytes_predicted{vps="xxx",direction="in|out"}
bitfolk_network_usage_bytes_max{vps="xxx",direction="in|out"}
Okay, I will use this scheme with the direction as a label.
4:08 p.m.
On Fri, 2022-02-25 at 09:57 +0000, Andy Smith wrote:
On Thu, Feb 24, 2022 at 02:40:13PM +0000, Andy Smith
wrote:
Hi,
I think I'm missing something. The "predicted" on the panel[1] looks
rather different to grafana.
The screenshots were taken about 5 seconds apart.
Is that expected?
Conrad
[1]https://panel.bitfolk.com/xfer/
On Thu, Feb 24, 2022 at 02:19:07PM +0000, Conrad
Wood wrote:
I've now added panels and out and in to everyone's grafana
dashboards using the metric names above, except I used the label
"guest" instead of "vps".
Cheers,
Andy As for the metrics, I now (as of this morning)
put the result
from the
web scrapes into my prometheus like so:
bitfolk_network_usage_bytes_current{vps="xxx",direction="in|out"}
bitfolk_network_usage_bytes_predicted{vps="xxx",direction="in|out
"}
bitfolk_network_usage_bytes_max{vps="xxx",direction="in|out"}
Okay, I will use this scheme with the direction as a label. 
8:47 p.m.
Hi Conrad,
On Fri, Feb 25, 2022 at 04:08:32PM +0000, Conrad Wood wrote:
I think I'm missing something. The
"predicted" on the panel[1] looks
rather different to grafana.
The screenshots were taken about 5 seconds apart.
Is that expected?
Yes. The prediction algorithm was explained here:
https://lists.bitfolk.com/lurker/message/20220223.204720.d86cf277.en.html
The two things (emailed notifications about data transfer, and the
figures on the panel about data transfer) have their predictions
calculated differently.
For the email notifications:
- First it does a lightweight prediction by taking the sum of data
transferred this period, averaging it to a per-second value and
then multiplying that by the number of seconds in 30 days.
- If:
- This prediction indicates a state transition, either from
"predicted okay" to "predicted over" or the reverse, AND
- You're in the first 15 days of the period, THEN:
a more heavyweight prediction is done by adding up the actual
figures for the last 30 days. That is then used as the prediction.
For the panel:
- Just uses the simple lightweight prediction.
Reasoning: If we're going to send a notification about a state
change, either to tell you to worry or to tell you not to worry, it
would be better to use more accurate figures for that prediction.
The lightweight prediction might be using only a small amount of
data. In your case right now about 3 days' worth to guess about a 30
day period.
If you look back at the last email notification sent to you, you'll
see that it predicted you'll go over again this period, and its
predicted figure matches what our Prometheus is now telling you,
because that's always how it's worked.
It's true that the simple prediction for you is right now returning
a much lower figure. You can look back in your bandwidth graph and
see why that is: There was a prolonged period of high usage a while
ago, but within the last 30 days. If we ignore that period then of
course the prediction will come out much lower. Is it correct to
ignore that period? Who knows, we just have to pick an algorithm,
you could argue either way.
So why not always use the more accurate prediction on the panel as
well?
The reason is historical. The figures are stored in a regular SQL
database and summing up 30 days of metrics from that source takes an
appreciable amount of time. This setup pre-dates Prometheus or any
other sort of time series database we had going.
So, we have only been doing it when it's considered important to do
so, in the batch job that sends out email notifications about data
transfer limits, and then the calculated predictions were thrown
away. It wasn't considered feasible to do those calculations in a
web page.
By the time you are 15 days in to this reporting period the two
figures will match.
So, what you are seeing is expected but it doesn't mean we can't
improve things.
I do not want to calculate the official billing figures from
Prometheus because I don't want to make Prometheus (or Grafana) an
essential piece of infrastructure.
I could store the calculated predictions back in the database and
use those on the panel.
It's always worked like this and no one has really noticed in more
than 10 years, probably because hardly anyone reaches their limits
and when they do it's usually nearer to the end of the reporting
period. It should be pretty easy to store the calculated predictions
though, I've just never thought of doing it before.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
27 Feb
27 Feb
12:56 p.m.
On Fri, 2022-02-25 at 20:47 +0000, Andy Smith wrote:
Hi Conrad,
On Fri, Feb 25, 2022 at 04:08:32PM +0000, Conrad Wood wrote:
Understood.
I suggest we let this run for a couple of months and observe.
I agree the prediction does not have to be super accurate, so I would
expect that what we have now should work well.
Thank you!
Conrad
I think I'm missing something. The
"predicted" on the panel[1]
looks
rather different to grafana.
The screenshots were taken about 5 seconds apart.
Is that expected?
Yes. The prediction algorithm was explained here:
https://lists.bitfolk.com/lurker/message/20220223.204720.d86cf277.en.html
The two things (emailed notifications about data transfer, and the
figures on the panel about data transfer) have their predictions
calculated differently.
For the email notifications:
- First it does a lightweight prediction by taking the sum of data
transferred this period, averaging it to a per-second value and
then multiplying that by the number of seconds in 30 days.
- If:
- This prediction indicates a state transition, either from
"predicted okay" to "predicted over" or the reverse, AND
- You're in the first 15 days of the period, THEN:
a more heavyweight prediction is done by adding up the actual
figures for the last 30 days. That is then used as the prediction.
For the panel:
- Just uses the simple lightweight prediction.
Reasoning: If we're going to send a notification about a state
change, either to tell you to worry or to tell you not to worry, it
would be better to use more accurate figures for that prediction.
The lightweight prediction might be using only a small amount of
data. In your case right now about 3 days' worth to guess about a 30
day period.
If you look back at the last email notification sent to you, you'll
see that it predicted you'll go over again this period, and its
predicted figure matches what our Prometheus is now telling you,
because that's always how it's worked.
It's true that the simple prediction for you is right now returning
a much lower figure. You can look back in your bandwidth graph and
see why that is: There was a prolonged period of high usage a while
ago, but within the last 30 days. If we ignore that period then of
course the prediction will come out much lower. Is it correct to
ignore that period? Who knows, we just have to pick an algorithm,
you could argue either way.
So why not always use the more accurate prediction on the panel as
well?
The reason is historical. The figures are stored in a regular SQL
database and summing up 30 days of metrics from that source takes an
appreciable amount of time. This setup pre-dates Prometheus or any
other sort of time series database we had going.
So, we have only been doing it when it's considered important to do
so, in the batch job that sends out email notifications about data
transfer limits, and then the calculated predictions were thrown
away. It wasn't considered feasible to do those calculations in a
web page.
By the time you are 15 days in to this reporting period the two
figures will match.
So, what you are seeing is expected but it doesn't mean we can't
improve things.
I do not want to calculate the official billing figures from
Prometheus because I don't want to make Prometheus (or Grafana) an
essential piece of infrastructure.
I could store the calculated predictions back in the database and
use those on the panel.
It's always worked like this and no one has really noticed in more
than 10 years, probably because hardly anyone reaches their limits
and when they do it's usually nearer to the end of the reporting
period. It should be pretty easy to store the calculated predictions
though, I've just never thought of doing it before.
Cheers,
Andy 
24 Feb
24 Feb
9:25 a.m.
Hi Andy,
On Thu, 24 Feb 2022 at 00:17, Andy Smith <andy(a)bitfolk.com> wrote:
It reads like you actually don't agree with me.
What I was trying to
say was that I expect customers to actually be able to measure
their own bandwidth usage, and BitFolk telling them that they are
near or over quota is just a courtesy.
What you are saying is that you don't expect to have to measure
this yourself, you would never be aware of it or investigate it
without a notification from BitFolk and therefore these
notifications are essential to you. You did get the notification and
were able to take action, but only because email notifications
are satisfactory for you.
I think you're in an interesting dilemma where the act of you sending
courtesy bandwidth usage emails leads to less bandwidth monitoring on
the user side. If a user is consistently getting messages that they
are using say 10% of the available bandwidth and with no real
expectation that there is going to be a sudden increase, then I can
entirely understand not doing any monitoring.
Regards,
Roger
10:11 a.m.
My 5p worth as a long time hard core geek and developer....
For me this is a non-question. There is an API already, on https://panel.bitfolk.com/xfer/
albeit in html.
To do an http request with authorisation and filter out the correct information is barely
any more work than implement any other API call into your current system.
Alternatively, have a rule for incoming emails that sends you an alert. (How? Email? ;-)
)
Trying to not sound rude, but everything is there, just not in your flavour of ice cream.
For Andy to implement anything else would just create redundant functionality, and you
surely would have to create a similar customisation at your end as with what is already
there. The web server(s) behind the panel surely wouldn't notice another call ever
fifth minute.
As for the courtesy email, that is my only monitoring of bandwidth usage. my machines
rarely even hit 1 GB transfer per month, so I don't see the need for it. I blindly
trust Andy and BitFolk to keep an eye on that for me. Probably not very polite of me. Do
have other systems set up for monitoring and protecting against an attack, but bandwidth
is not something I pay attention to there. Maybe I should, but so far I haven't seen
the need for it. I check the usage email when it comes in and if it looks wrong I look in
to it. This has happened once, and the usage was maybe 5GB instead of 1.
Kindest regards,
__
/ony
-------
Thursday, February 24, 2022, 12:09:44 AM, Andy wrote:
Hi Roger,
On Wed, Feb 23, 2022 at 11:37:40PM +0000, Roger Light
wrote:
> On Wed, 23 Feb 2022 at 20:51, Andy Smith <andy(a)bitfolk.com> wrote:
>
> > - The rare times that people do go over their quota they tend to
> > focus on BitFolk's measurements of this instead of looking into
> > the behaviour of their own VM. I feel like anything I implement is
> > just going to be a further distraction from the customer's own
> > responsibilities.
>
> This matches my experience. I pay reasonable attention to the quota
> emails but don't do any monitoring myself because I don't expect to be
> at any risk of going over the quota. From a customer perspective
> though, I expect the measurements made by bitfolk to be the gold
> standard in terms of billing so my measurements would ultimately be
> secondary.
It reads like you actually don't agree with me.
What I was trying to
say was that I expect customers to actually be able to measure
their own bandwidth usage, and BitFolk telling them that they are
near or over quota is just a courtesy.
What you are saying is that you don't expect to
have to measure
this yourself, you would never be aware of it or investigate it
without a notification from BitFolk and therefore these
notifications are essential to you. You did get the notification and
were able to take action, but only because email notifications
are satisfactory for you.
If most people are the same then this means that
receiving and being
able to act upon these notifications isn't just a courtesy, it is
essential. The fact that you were able to is great, but the fact
that others aren't is of concern to me.
Conrad was saying that they do have their own
monitoring but that
the email notifications and the figures on the panel web page aren't
satisfactory because they are hard to be aware of (emails get
overlooked) and don't provide easy spot checks.
Is there any middle ground that could improve things
for you,
Conrad, without me having to make an HTTP endpoint that gives these
figures?
For example, I could probably plug it in to our Icinga
monitoring¹
so that it continually gave you a WARNING email when predicted to
exceed, turning into a CRITICAL when it actually has. Besides email,
our Icinga can also do Pushover² alerts, and I am open to adding
more kinds of alerts there.
Even if that does not work for Conrad, it might be a
better
interface for more people so I would be interested in feed back on
that or other ideas.
One of my big problems with the API thing is that I
feel like it
will have a very limited audience, very likely at this point to be
only Conrad. I currently manage four Linodes for various reasons and
although Linode has an API I've never felt a need to use it myself.
I monitor my own bandwidth usage. I do often find myself at odds
with others though.
Cheers,
Andy
11:48 a.m.
On Thu, 2022-02-24 at 10:11 +0000, Tony Andersson wrote:
My 5p worth as a long time hard core geek and
developer....
For me this is a non-question. There is an API already, on
https://panel.bitfolk.com/xfer/ albeit in html.
To do an http request with authorisation and filter out the correct
information is barely any more work than implement any other API call
into your current system.
Thanks - good idea ;)
I've just checked and it really was quite simple to scrape.
(With the usual caveats of web-site scraping though. )
For now, that might just be good enough.
I'm sure I'll bring it up in 2 years again ;)
Thanks everyone.
Cheers,
Conrad
12:55 p.m.
Andy Smith said:
I am interested in what other customers think. Though as most people
never get anywhere near their data transfer quota, before responding
please do try to put yourselves in the position of someone who has
just received the automated email that says they've gone over quota
and every GB transferred is now costing them £0.06+VAT. Does the
current facility of the panel web page, Grafana site and emailed
notifications cut it then?
My 2p worth. I've never gone over (or got very near) the quota, so it's
not something I've thought too hard about.
For my own use, I think your emailed warnings are fine. They arrive on
my phone and I look at them in fairly short order. A pushover alert
would also arrive on my phone and get looked at. But I do "get" that
others might like to receive alerts via other methods.
Since Bitfolk are the ones who'd charge me if I went over, I think it's
not unreasonable to expect to be able to use your figures rather than
trying to handball my own and ensure they match. Getting a ballpark
figure isn't hard and I'd do that if I cared enough, but it's only ever
going to be an estimate of what you'd charge.
On a wider note, I wonder whether configurable throttling would be
handy. If someone does something that hammers my VPS, it'd be a comfort
to know that, at the point you're (imminently) going to start charging
me per GByte, I can have my bandwidth throttled down to limit how
quickly that charge goes up. This is a WIBNI
(https://en.wiktionary.org/wiki/WIBNI), nothing more. For all that lazy
me knows, you may already have this available!
Cheers,
Alun.
1:58 p.m.
New subject: Configurable traffic throttling (was Re: predicted network usage / "Predicted to exceed transfer quota")
Hello,
On Thu, Feb 24, 2022 at 12:55:44PM +0000, Alun Jones via users wrote:
On a wider note, I wonder whether configurable
throttling would be handy. If
someone does something that hammers my VPS, it'd be a comfort to know that,
at the point you're (imminently) going to start charging me per GByte, I can
have my bandwidth throttled down to limit how quickly that charge goes up.
Just on this point: it's not easy to throttle traffic that is sent
TO a customer (traffic police), because that is not in the control
of either BitFolk or the customer.
If someone is determined to send traffic to a BitFolk VM then a
filter has to be applied upstream at the far end of all of Jump's
transit and peering links to prevent that traffic getting on to
Jump's network and being chargeable to BitFolk and thus the
customer. Realistically this cannot be done except in cases of
denial of service and even then it's mostly done by dropping all
traffic to the destination IP, so that is a total outage for the
customer.
The good news is that most legitimate overage is outbound, because
the customer VM is serving something, so when the customer stops
doing that the traffic goes away.
Also outbound traffic is amenable to throttling and that can be done
inside the customer's VM or on the BitFolk side. Typically using the
"tc" command.
However, I have not seriously thought about offering that as a
setting on the panel because this situation happens rarely and
traffic throttling is quite complex. It feels like either it would
need to be simplified to the point of being quite poor and blunt
instrument, or else it would be dauntingly complex.
For example, you don't normally want to set a hard limit on port
speed. 2TB in 30 days?
$ units "2TB/30 days" "megabits/sec"
* 6.1728395
so set the port speed to 6Mbit/s and never go over quota? Well, yes,
but also then you never burst past 6Mbit/s even when you aren't
anywhere near your quota. Pretty sucky when it's easy to get near
100Mbit/s out of a VM for a single TCP conversation.
So then you start thinking, well, how about no throttling until say
75% of the quota is gone and then set the needed port speed so that
the last 25% can never be used.
Or use a token bucket filter to allow bursts but still maintain the
required average.
And so on.
With all of this achievable from within the customer's VM, I kind of
don't want to get into it from BitFolk's side. :)
When customers are predicted to go over their quota or have gone
over it and are having problems locating the cause of it, I do
generally offer to limit their port speed or shut off network
completely, in order to control their costs.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
2:23 p.m.
New subject: Configurable traffic throttling (was Re: predicted network usage / "Predicted to exceed transfer quota")
Andy Smith said:
Just on this point: it's not easy to throttle traffic that is sent
TO a customer (traffic police), because that is not in the control
of either BitFolk or the customer.
Indeed. Been there (at work) :-)
However, I have not seriously thought about offering
that as a
setting on the panel because this situation happens rarely and
traffic throttling is quite complex. It feels like either it would
need to be simplified to the point of being quite poor and blunt > instrument, or else
it would be dauntingly complex.
I was thinking in terms of a very blunt instrument indeed. e.g. "if over
quota, limit rate to (say) 2MBits", maybe with a tick box that says "I
acknowledge it this time around - remove the throttle until the next
accounting period starts". So, if (serving) something takes me over
quota, my liability is under 6p/hour until I do anything about it.
With all of this achievable from within the
customer's VM, I kind of
don't want to get into it from BitFolk's side. :)
Also perfectly sensible. Again, availability of your accounting data
would make the VM side of things easier.
I've been thinking a bit about how I could maintain my own stats, over
lunch. The easiest way of getting a raw count (in my setup) would just
be ip(6)tables rules in the INPUT, FORWARD and OUTPUT chains. Just use
the counters from those. But then I started thinking about when the VM
reboots (or I reload the firewall) and then started thinking about
recording to a database. Catching edge cases like this does make it less
trivial than just scraping your existing, definitive data from somewhere.
Cheers,
Alun.
2:44 p.m.
New subject: Configurable traffic throttling (was Re: predicted network usage / "Predicted to exceed transfer quota")
Hi Alun,
On Thu, Feb 24, 2022 at 02:23:54PM +0000, Alun Jones via users wrote:
I've been thinking a bit about how I could
maintain my own stats, over
lunch. The easiest way of getting a raw count (in my setup) would just be
ip(6)tables rules in the INPUT, FORWARD and OUTPUT chains. Just use the
counters from those.
That is in fact how BitFolk is doing it, because of having to
disregard internal data flows.
But then I started thinking about when the VM reboots
(or I reload
the firewall) and then started thinking about recording to a
database. Catching edge cases like this does make it less trivial
than just scraping your existing, definitive data from somewhere.
When people start monitoring anything they usually start to use
software that collects and persists all kinds of metrics, like
node_exporter or collectd or whatever, so that solves that issue for
them, and allows for monitoring all kinds of things besides
bandwidth.
It could perhaps be seen as overkill for one VM.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
3:03 p.m.
New subject: Configurable traffic throttling (was Re: predicted network usage / "Predicted to exceed transfer quota")
Andy Smith said:
[iptables counters]
That is in fact how BitFolk is doing it, because of
having to
disregard internal data flows.
I did fall a bit down the rabbit hole on this one and landed up on
"nfacct", which looks nice. You can request and clear counters as a
single atomic request and the userland side can export in various
machine readable forms. Makes the "get it into a backend database" part
very simple.
Cheers,
Alun.
28 Feb
28 Feb
8:16 a.m.
Wouldn't it be simpler to either a) fix your services so they don't
have such a colourful failure mode, or b) if that's not possible
monitor the instantaneous traffic and alert or even shut things down
if it stays too high for too long?
Gordon
On Wed, 23 Feb 2022 at 08:50, Conrad Wood <cnw(a)conradwood.net> wrote:
Hi,
Sometimes some of my services misbehave in weird and wonderful ways.
If that happens, they can use an awful lot of bandwidth.
Whilst there is some monitoring that I can do (and do), I prefer to use
the data "closer to source".
There seems to be some data, that triggers pretty quickly and sends out
an email from "xfer a_t bitfolk.com" with the heading "Predicted to
exceed transfer quota".
However, rather than email I'd like an API, so that I can route it to
my alerting system. I do overlook emails sometimes, especially computer
generated ones.
My brain sometimes scans and classifies them too quickly as "deal with
it later" and incorrectly flags the email as 'not important'.
Having the information in my alerting system means I can route them so
that they remind me/trigger me again etc.
My questions are:
1. How often is that prediction calculated?
2. Is there an API I could query which gives me the data from the
email?
2.1 I'd settle for a binary API (predicted overusage == true|false)
2.2 otherwise an api that gives me the following values:
- predicted usage in
- contracted usage in
- predicted usage out
- contracted usage out
- Actual out
- Actual in
Nothing fancy required - a simple 'get' request or so would do just
fine ;)
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
1027
days inactive
1032
days old
20 comments
7 participants
participants (7)
-
Alun Jones -
Andy Smith -
Conrad Wood -
Gordon Pearce -
Roger Light -
Roger Light -
Tony Andersson