BitFolk Announcements

announce@mailman.bitfolk.com

1 participants
265 discussions

Pushover notifications now available (beta)

by Andy Smith

Hi, I make use of Pushover myself: https://pushover.net/ A customer asked for non-email notifications from BitFolk's monitoring and I suggested Pushover for them too. I've now added the necessary bits to generalise it for any customer. If you would like Pushover notifications then please mail support(a)bitfolk.com to ask for them. When you ask please: - supply your Pushover User Key (visible in your Pushover dashboard) - state whether you want it for just host notifications (is the host down or unreachable?) or for service notifications as well (is service X on host Y in a non-OK state?). Services might generate quite a few notifications. If you only want SOME services to generate these notifications, you can let us know what those are too. Limitations: - Will only send a notification on a non-OK state, so you won't receive RECOVERY/OK notifications. This is in the interest of not deluging you with notifications. - Priority is always set to 1 (high), which means it will try to make a noise and vibrate your mobile device. You can override that in your mobile device. The full API does allow an emergency priority which would require you to acknowledge it, and three quieter options, but I'm not ready to support any of those yet. - You can't currently have Pushover notifications without email notifications too. - You can't tell monitoring to stop sending you Pushover notifications but carry on sending email ones. Anything you'd usually do to stop notifications will stop both kinds. If more than a few people request this then I will integrate it with the address book in the panel so you can get Pushover notifications by adding a contact with a Pushover key there. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 2 months

BitFolk's bank details changed (but the old ones will still work)

by Andy Smith

Hi, If you don't pay us by bank transfer or standing order then you can ignore this email. BitFolk's bank details have changed. The old ones should still work but next time it is convenient it would be good if you would update to the new ones. The new bank details are listed on every invoice generated after 1600Z today, and also in your Panel at: https://panel.bitfolk.com/account/invoices/#toc-bank-details Despite Barclays assuring me that I would still be able to log in and download a final statement after the old account was closed, it appears I cannot do that so I'm unable to see any bank payments made in the last couple of days. I expect I'll get a final paper copy of the statement through the post in the next week and will be able to mark those payments as settled then. So if you made a bank payment in the last few days but the invoice is still marked as outstanding, do not worry. Get in touch if you start to receive late payment reminders about it. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 2 months

2020-08-16 22:58Z - Host "elephant" locked up and required a power cycle

by Andy Smith

Hi, At approximately 22:58Z on 16 August host "elephant" became unresponsive and we received alerts for every customer/service on it. Its BMC serial console connected but was also unresponsive. After determining that it was still powered on but not responding, around 23:13Z I decided to power cycle it. This appears to have gone without incident and by around 23:17Z customer VMs were starting to be booted; this completed by around 23:31Z. If you are still experiencing problems with your service please in the first instance have a look at your Xen Shell console to see if you spot anything obvious. https://tools.bitfolk.com/wiki/Xen_Shell If that doesn't help, please open a support ticket. I still do not know what caused elephant to hard lock up. There are no hardware events logged. It may happen again in which case we will need to look at swapping it out, which will come with some downtime for those hosted on it. Apologies for the disruption and thanks for your patience. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 3 months

Backup usage alerts haven't been happening

by Andy Smith

Hi, It's come to my attention that customers haven't been receiving backups usage alerts as described here: https://tools.bitfolk.com/wiki/Backups#Usage I have been receiving copies of them, but customers haven't. This was first reported today and seems to have been the case since we switched from Nagios to Icinga2 in December 2018. Sorry about that. I expect to have it fixed shortly. There are currently 7 customers who are over their backup quota so when it's fixed you'll start receiving those alerts again. Backups not happening leads to "backups age" alerts, and they have always been working. Those 7 customers have been receiving those ones. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 4 months

Backup age alerting blip

by Andy Smith

Hi, For those who have backups hosted on backup4, you will have today received some alerts like this: [PROBLEM] backup age ruminant on backup4.bitfolk.com is CRITICAL! (or it may have said WARNING instead of CRITICAL) This was caused by me doing some work on that host that reset timestamps on the backup directories leading to false positive alerts, until the next backup run. The work is finished now so by now you should have received a corresponding "OK" recovery notification. If you did not then there is an actual problem. You can check at: https://mon.bitfolk.com/ as all customers with backups do have monitoring set up. Apologies for the confusion; I will try to remember to set such monitoring into downtime mode in future. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 4 months

Billing process delayed for a couple of days, now catching up

by Andy Smith

Hi, Due to a problem with one of our payments providers our billing processes were stuck for a couple of days. As a result we haven't been sending invoices out as much ahead of time as usual. Things are caught up again now but as a side effect some invoices have just been created that are due today or in a day or two, instead of giving you the usual week of notice. We still allow 30 days for payment, but it does mean that some reminders will go out from tomorrow. Those paying by automated means are not going to be affected. In the unlikely event that anyone reaches the 30 days of non-payment we will of course give a couple more days of leniency in light of this hiccup. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 5 months

Do not try to upgrade a BitFolk VM to Ubuntu 20.04 yet

by Andy Smith

Hi, Ubuntu 20.04 is scheduled to be released at some point today. As things stand its kernel won't boot at BitFolk. I think it's due to some bugs in Xen's LZ4 decompression. These problems have since been fixed but it would mean a forced reboot to deploy them and I'm not going to do that, for this. I will let you know when there is a host available that will boot these LZ4-compressed kernels, so if you are desperate to upgrade to 20.04 you can be moved there. If you already did try to upgrade to 20.04 and now it doesn't boot, your previous (18.04) kernel will boot, which clearly isn't ideal but does at least regain access. Possibly you could build a custom kernel that doesn't use LZ4 or find a way to unpack (and optionally repack) the existing vmlinuz. I'll follow up with more info as I experiment with that. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 6 months

Minimum Direct Debit charge will be £5.00 from 27 April 2020

by Andy Smith

Hi, If you do not currently pay us by Direct Debit and have no interest in doing so then you can ignore this email. == Short version == From 27 April 2020 we're introducing a minimum Direct Debit charge of £5.00. Any time we would charge you less than £5.00, we are going to charge you £5.00 and credit your account with the difference. As there is no BitFolk plan that costs less than £6.49+VAT per month THIS WILL AFFECT ALMOST NO ONE, unless you somehow get a lot of credit on your account, or order something small (like 5GiB of storage) and generate a one-off bill. If you do not like this, I'm afraid we will have to ask you to cancel your Direct Debit mandate and pay us by some other method. You can do that from: https://panel.bitfolk.com/account/billing/ If you are feeling nice you can also set a higher minimum charge, which saves us giving fees to our payment provider. You can do that from the link above. Or you could switch to quarterly or yearly payments. Please contact support@ to do that. That's everything you need to know and you can stop reading now if you like. Thanks! == Long version == === What's happening === From 27 April 2020 we'll be introducing a minimum charge of £5.00 on Direct Debit payments. === Background === Almost half of you currently pay us by Direct Debit. I previously mentioned that as of March 2020 our Direct Debit payment provider (GoCardless) was introducing a £0.20 loading on every transaction, in addition to their usual 1% transaction fee. The majority of our customers are on the smallest plan, at £7.79 inc VAT per month. As an example, pre-March the fees for that charge were £0.08 (1%). Now they are £0.28. Overall, the £0.20 loading has resulted in an 83% increase in our transaction fees. For that reason I asked customers to please consider either switching to quarterly or yearly payments, or else consenting to a higher minimum charge, with the remainder being credited to the account. A fair few of you did switch and one customer increased their minimum charge. As only one person had switched I did not actually implement anything except the ability to choose the charge, and we carried on charging the exact amount. Today I have noticed that a Direct Debit payment failed to go through, because we were trying to charge only £0.83. GoCardless have obviously altered their API to not support such a small charge, which I suppose makes sense and gives me the motivation needed to do the same on BitFolk's side, because otherwise this customer's service can never be marked as paid for. === What we're doing === If I am going to implement a minimum charge then I don't see the point of trying to make it as low as it possibly can be, given that I still have to do all the work and write this email just to tell you about whatever value it is. It appears that the minimum charge is £1.00. I've made the decision that our minimum charge will be £5.00, which is still well below the regular cost of any plan we sell, so will affect almost no one. So, we have no choice but to implement a minimum charge. We could handle it by batching up your invoices until they are >= £5.00 but that is a lot of work for a situation that is really rare, and I also don't want to be billing people in arrears. I'm choosing to set the minimum at £5.00, not £1.00. === What you can do === If this upsets you and you never want to be charged potentially £4.99 extra that sits on your account as credit then I'm afraid you will have to cancel your Direct Debit mandate before 27 April 2020 and start paying us by some other means. You can do that from: https://panel.bitfolk.com/account/billing/ Otherwise, thank you for your understanding! Additionally if you want to be nice you could: - Set a minimum charge that's higher than £5.00. You can do that from the link above. The excess will sit on your account as credit, you can have it back at any time, and it will reduce what we pay out as fees. - Change to a quarterly or yearly contract. Please contact support@ if you'd like to do that. === We still like GoCardless / Direct Debit === Obviously I'm not thrilled that fees for anything have gone up by 83%, but I just want to state factually that even so, the fees for GoCardless are still less than PayPal and Stripe (credit cards). We are contractually obliged by various merchant contracts to not place one payment method above another in terms of favour or prominence, but hopefully just repeating a fact is okay! === What about Direct Debit payments from outside the UK? === At the moment in order to pay us by Direct Debit you need a bank account in GBP and a UK postal address¹. GoCardless recently opened up their service to those in the Single European Payments Area, USA, Canada, and probably Australia/NZ too. This would be charged in your local currency and I am still grappling with the changes needed for that. It will happen at Some Point. === Further questions? === If you have any questions please reply; your reply will go to the users mailing list and I will answer there, or else if you want to ask in private please send to support@. Thanks! Andy ¹ Some financial providers like TransferWise do provide GBP accounts with a Direct Debit feature and provide a UK address you can use to set up Direct Debit mandates no matter where in the world you live. That works. Also if you have a GBP account with a UK bank but happen to live outside UK, it works right now to provide your bank's address. -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 7 months

Base data transfer doubled

by Andy Smith

Hi, Effective immediately we have doubled the base plan's data transfer quota from 1TB/month outbound to 2TB/month for paying customers. Those on 95th percentile billing have had the base plan doubled from 4Mbit/s to 8Mbit/s. If you currently pay for extra data transfer quota (or 95th percentile commitment) and don't need it anymore then you should check that out and contact support to remove it if necessary. If you pay by PayPal subscription then your subscription in PayPal won't have updated its description as these are set at the time they are created; please check with the Panel² to know what your specs are and compare them with your running VM. We've also gone through and had a look at the specs of the sponsored hosting accounts¹ and standardised most of them to 1.5G memory, 2TB data transfer as some of these were quite old and were on the lower defaults of old. Not all of those have had their data transfer quota doubled, because some were already higher than the base, but everyone got something! I absolutely am not going to do a "keep calm" line, so I will just say take care of yourselves and each other and please use more bandwidth. Preferably for good, not evil. Cheers, Andy ¹ https://tools.bitfolk.com/wiki/Sponsored_hosting ² https://panel.bitfolk.com/account/#toc-vps-plan and https://panel.bitfolk.com/account/config/ -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 8 months

Clarifying what you may consider to be a data leak issue with our Grafana

by Andy Smith

Hi, Apologies up front that the subject line reads like some sort of passive-aggressive dodge ("I Am Sorry If You Feel You Have Been Abused By Us", or "I Could Tell You But Then You Would Have To Be Destroyed By Me"¹), but I didn't want to use language like THERE HAS BEEN A COMPROMISE OF YOUR DATA …because at the heart of this is a miscommunication on my part, of a situation that has existed forever. Essentially nothing has changed but it is perhaps surprising to some that things were/are the way they are. == The issue == If you are logged in to BitFolk's Grafana² to look at the graphs of your service, then you can look at the queries that the Javascript is sending, edit some of the placeholders, and look at graphs for any other BitFolk customer. This was known by me when I set up our Grafana last October/November. The possibility of logged-in users sending arbitrary queries was raised to me at the time by a couple of people, and my response to that was to set some ACLs such that only the specific queries that correspond to defined dashboards are allowed. That is, the things that you can see can also be seen by any logged in user if they make some trivial effort to do so. I thought this was acceptable because with our previous solution (Cacti), all users' graphs were visible by anyone on the Internet, and that was the case from some time in 2007 up until late last year. So in fact the current Grafana is more restrictive than Cacti was. I thought that this had been adequately communicated to you, the customer base, including to the people who had raised concerns about the Prometheus/Grafana security model. I was wrong. Conrad Wood was one of the people who kindly advised me when I was setting up Grafana/Prometheus; Conrad did warn me about this issue as did a couple of other people, and I thought that I had communicated what my solution was going to be (and that it was actually stricter than Cacti was), but I might not have. That suggests that there will be other customers who are unaware of this, and unhappy about it. == What we will do about it == I don't want customers to be unhappy, so what I will do is work on tightening up the ACLs such that logged-in users can only use the label/placeholder values that pertain to them. I will work on this as a priority but I think it will still take a couple of weeks to do. In the mean time, if you are not comfortable with the situation that any other user can craft a query to look at your CPU / bandwidth / block IO stats, please drop an email to support(a)bitfolk.com and I will block all non-admin access to your stats. That will include your own access. == Don't shoot the messenger == I appreciate that many of you probably will not care that other customers could with some minor effort look at your graphs. The fact that Conrad doesn't think it's acceptable, and repeatedly tried to tell me that (but I failed to communicate and our wires initially got crossed), probably means that there are some other customers for whom this is news and who would be unhappy about it. So I thank Conrad for bringing it to my attention again; I have adjusted my viewpoint; I will restrict it; it would not be useful for anyone to comment that they personally don't care. There will be others who care, and this is for them. (I did check with Conrad that they wanted to be named here and they indicated that would be fine. If they'd said no then I'd just have said, "a customer has brought to my attention…") Thanks for reading, and I apologise for not making the situation about visibility of stats clear throughout this long period of time. Cheers, Andy ¹ https://www.amazon.com/Could-Tell-Then-Would-Destroyed/dp/193555414X ² https://tools.bitfolk.com/grafana/ -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 9 months

← Newer
1
...
4
5
6
7
8
9
10
...
27
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BitFolk Announcements