19 Apr
2024
19 Apr
'24
5:35 p.m.
I am responsible for several VPSes, here and elsewhere. Five of them are
running Ubuntu 22.04, three Debians.
The script I use to update them checks, at the end, for the existence of
/var/run/reboot-required
If it finds it, it offers to reboot the VPS.
It does this happily on all but one VPS, one of the Ubuntu ones here. The
Ubuntu version of apt-get on all of the Ubuntu ones recognises that a
reboot is required after a kernel update etc and will popup a message
saying so, but it looks like only on this single machine, that file doesn't
exist afterwards.
I have no idea why not. Anyone got any ideas?
Ian
19 Apr
19 Apr
8:06 p.m.
Hi,
On Fri, Apr 19, 2024 at 06:35:40PM +0100, Ian via BitFolk Users wrote:
The Ubuntu version of apt-get on all of the Ubuntu
ones recognises
that a reboot is required after a kernel update etc and will popup
a message saying so, but it looks like only on this single
machine, that file doesn't exist afterwards.
I have no idea why not. Anyone got any ideas?
I don't know what is different about that one Ubuntu VM, but I think
I might reach for auditd either to tell me what creates the file on
the working ones, or what deletes the file on the non-working one,
or both.
# Install auditd
$ sudo apt install auditd
# Add a rule that watches /root/reboot-required (/var/run is
# symlink so not sure if that will be acceptable). The thing after
# the -k is the rule key to search for afterwards in the results.
$ sudo auditctl -a always,exit -F arch=b64 -F path=/run/reboot-required -S
open,openat,rename,rmdir,unlink,unlinkat,renameat -k reboot-required
# Test it
$ sudo touch /run/reboot-required
$ sudo rm /run/reboot-required
$ sudo ausearch -k reboot-required
[…]
----
time->Fri Apr 19 20:58:54 2024
type=PROCTITLE msg=audit(1713556734.896:5070):
proctitle=746F756368002F72756E2F7265626F6F742D7265717569726564
type=PATH msg=audit(1713556734.896:5070): item=1 name="/run/reboot-required"
inode=7586 dev=00:17 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0
cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1713556734.896:5070): item=0 name="/run/" inode=1 dev=00:17
mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0
cap_frootid=0
type=CWD msg=audit(1713556734.896:5070): cwd="/home/andy/src/ncmpcpp-setup"
type=SYSCALL msg=audit(1713556734.896:5070): arch=c000003e syscall=257 success=yes exit=3
a0=ffffff9c a1=7ffd7680a74a a2=941 a3=1b6 items=2 ppid=3314410 pid=3314411 auid=1000 uid=0
gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts43 ses=3 comm="touch"
exe="/usr/bin/touch" subj=unconfined key="reboot-required"
----
time->Fri Apr 19 20:58:59 2024
type=PROCTITLE msg=audit(1713556739.308:5077):
proctitle=726D002F72756E2F7265626F6F742D7265717569726564
type=PATH msg=audit(1713556739.308:5077): item=1 name="/run/reboot-required"
inode=7586 dev=00:17 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=DELETE cap_fp=0
cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1713556739.308:5077): item=0 name="/run/" inode=1 dev=00:17
mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0
cap_frootid=0
type=CWD msg=audit(1713556739.308:5077): cwd="/home/andy/src/ncmpcpp-setup"
type=SYSCALL msg=audit(1713556739.308:5077): arch=c000003e syscall=263 success=yes exit=0
a0=ffffff9c a1=5610f7a7c4a0 a2=0 a3=7f8863f59f60 items=2 ppid=3314463 pid=3314464
auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts43 ses=3
comm="rm" exe="/usr/bin/rm" subj=unconfined
key="reboot-required"
Note from the "type=SYSCALL" results you can see:
----
pid=3314411
uid=0
tty=pts43
comm="touch"
exe="/usr/bin/touch"
----
pid=331446400
uid=0
tty=pts43
comm="rm"
exe="/usr/bin/rm"
amongst other things…
Delete all rules again with:
$ sudo auditctl -D
With those rules installed you should get logs for every time a
process creates or removes that file path.
A simpler thing is inotifywait but as far as I know you don't get
access to what did the change there, only that a change has
happened.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
8:52 p.m.
I look after lots of Ubuntu servers. I see reboot-required often enough -
I'm pretty sure apt (via unattended-updates in my case) makes it when it
wants a reboot; I have nagios detect the file and say that a reboot is
required (to save manually looking at each machine).
Only thing I've personally seen remove the file is a reboot, in all the
years I've looked after Ubuntu has anything different happened. I like
Andy's idea of using auditd to keep an eye on reboot-required.
Kind regards
Murray Crane
On Fri, 19 Apr 2024 at 21:07, Andy Smith via BitFolk Users <
users(a)mailman.bitfolk.com> wrote:
Hi,
On Fri, Apr 19, 2024 at 06:35:40PM +0100, Ian via BitFolk Users wrote:
or send an email to
<users-leave(a)mailman.bitfolk.com>
The Ubuntu version of apt-get on all of the
Ubuntu ones recognises
that a reboot is required after a kernel update etc and will popup
a message saying so, but it looks like only on this single
machine, that file doesn't exist afterwards.
I have no idea why not. Anyone got any ideas?
I don't know what is different about that one Ubuntu VM, but I think
I might reach for auditd either to tell me what creates the file on
the working ones, or what deletes the file on the non-working one,
or both.
# Install auditd
$ sudo apt install auditd
# Add a rule that watches /root/reboot-required (/var/run is
# symlink so not sure if that will be acceptable). The thing after
# the -k is the rule key to search for afterwards in the results.
$ sudo auditctl -a always,exit -F arch=b64 -F path=/run/reboot-required -S
open,openat,rename,rmdir,unlink,unlinkat,renameat -k reboot-required
# Test it
$ sudo touch /run/reboot-required
$ sudo rm /run/reboot-required
$ sudo ausearch -k reboot-required
[…]
----
time->Fri Apr 19 20:58:54 2024
type=PROCTITLE msg=audit(1713556734.896:5070):
proctitle=746F756368002F72756E2F7265626F6F742D7265717569726564
type=PATH msg=audit(1713556734.896:5070): item=1
name="/run/reboot-required" inode=7586 dev=00:17 mode=0100644 ouid=0 ogid=0
rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0
cap_frootid=0
type=PATH msg=audit(1713556734.896:5070): item=0 name="/run/" inode=1
dev=00:17 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0
cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1713556734.896:5070): cwd="/home/andy/src/ncmpcpp-setup"
type=SYSCALL msg=audit(1713556734.896:5070): arch=c000003e syscall=257
success=yes exit=3 a0=ffffff9c a1=7ffd7680a74a a2=941 a3=1b6 items=2
ppid=3314410 pid=3314411 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
sgid=0 fsgid=0 tty=pts43 ses=3 comm="touch" exe="/usr/bin/touch"
subj=unconfined key="reboot-required"
----
time->Fri Apr 19 20:58:59 2024
type=PROCTITLE msg=audit(1713556739.308:5077):
proctitle=726D002F72756E2F7265626F6F742D7265717569726564
type=PATH msg=audit(1713556739.308:5077): item=1
name="/run/reboot-required" inode=7586 dev=00:17 mode=0100644 ouid=0 ogid=0
rdev=00:00 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0
cap_frootid=0
type=PATH msg=audit(1713556739.308:5077): item=0 name="/run/" inode=1
dev=00:17 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0
cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1713556739.308:5077): cwd="/home/andy/src/ncmpcpp-setup"
type=SYSCALL msg=audit(1713556739.308:5077): arch=c000003e syscall=263
success=yes exit=0 a0=ffffff9c a1=5610f7a7c4a0 a2=0 a3=7f8863f59f60 items=2
ppid=3314463 pid=3314464 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
sgid=0 fsgid=0 tty=pts43 ses=3 comm="rm" exe="/usr/bin/rm"
subj=unconfined
key="reboot-required"
Note from the "type=SYSCALL" results you can see:
----
pid=3314411
uid=0
tty=pts43
comm="touch"
exe="/usr/bin/touch"
----
pid=331446400
uid=0
tty=pts43
comm="rm"
exe="/usr/bin/rm"
amongst other things…
Delete all rules again with:
$ sudo auditctl -D
With those rules installed you should get logs for every time a
process creates or removes that file path.
A simpler thing is inotifywait but as far as I know you don't get
access to what did the change there, only that a change has
happened.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
BitFolk Users mailing list <users(a)mailman.bitfolk.com>
You're subscribed as <murray.crane(a)gmail.com>
Unsubscribe: <
https://mailman.bitfolk.com/mailman/postorius/lists/users.mailman.bitfolk.c…
8:59 p.m.
Hi,
On Fri, Apr 19, 2024 at 09:52:52PM +0100, Murray Crane via BitFolk Users wrote:
Only thing I've personally seen remove the file is
a reboot, in all the
years I've looked after Ubuntu has anything different happened.
I do think it's more likely to be something missing on the broken
one that results in the file never being created, as opposed to
something deleting the file.
auditd should still help, in that perhaps whatever creates it will not
show up.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
11 May
11 May
1:54 p.m.
Andy said:
I do think it's more likely to be something missing on the broken
one that results in the file never being created, as
opposed to
something deleting the file.
A couple of recent-ish updates needed restarts. Looking at the logs from
auditd says, yep, the file is never created for some reason.
Ian
14 May
14 May
8:30 p.m.
Doing some more looking, this is the only one with "Ubuntu Advantage" set
up on it - its 'livepatch' feature to do (some) kernel updates without
needing a restart is being too aggressive in stopping the file being
created.
Sigh. There is a script to detect that this is the case, but sigh...
Ian
On Sat, 11 May 2024, 14:54 Ian, <ian(a)lovingboth.com> wrote:
Andy said:
I do think it's more likely to be something missing on the broken
one that results in the file never being created,
as opposed to
something deleting the file.
A couple of recent-ish updates needed restarts. Looking at the logs from
auditd says, yep, the file is never created for some reason.
Ian
>
19 Apr
19 Apr
8:59 p.m.
unattended-upgrades, even...
This is the problem with keeping all my deployment documents on a wiki; I
just copy/paste the commands and run them now, I don't have to pay a lot of
attention to what they say...
Kind regards
Murray Crane
On Fri, 19 Apr 2024 at 21:52, Murray Crane <murray.crane(a)gmail.com> wrote:
I look after lots of Ubuntu servers. I see
reboot-required often enough -
I'm pretty sure apt (via unattended-updates in my case) makes it when it
wants a reboot; I have nagios detect the file and say that a reboot is
required (to save manually looking at each machine).
Only thing I've personally seen remove the file is a reboot, in all the
years I've looked after Ubuntu has anything different happened. I like
Andy's idea of using auditd to keep an eye on reboot-required.
Kind regards
Murray Crane
On Fri, 19 Apr 2024 at 21:07, Andy Smith via BitFolk Users <
users(a)mailman.bitfolk.com> wrote:
Hi,
On Fri, Apr 19, 2024 at 06:35:40PM +0100, Ian via BitFolk Users wrote:
or send an email to
<users-leave(a)mailman.bitfolk.com>
The Ubuntu version of apt-get on all of the
Ubuntu ones recognises
that a reboot is required after a kernel update etc and will popup
a message saying so, but it looks like only on this single
machine, that file doesn't exist afterwards.
I have no idea why not. Anyone got any ideas?
I don't know what is different about that one Ubuntu VM, but I think
I might reach for auditd either to tell me what creates the file on
the working ones, or what deletes the file on the non-working one,
or both.
# Install auditd
$ sudo apt install auditd
# Add a rule that watches /root/reboot-required (/var/run is
# symlink so not sure if that will be acceptable). The thing after
# the -k is the rule key to search for afterwards in the results.
$ sudo auditctl -a always,exit -F arch=b64 -F path=/run/reboot-required
-S open,openat,rename,rmdir,unlink,unlinkat,renameat -k reboot-required
# Test it
$ sudo touch /run/reboot-required
$ sudo rm /run/reboot-required
$ sudo ausearch -k reboot-required
[…]
----
time->Fri Apr 19 20:58:54 2024
type=PROCTITLE msg=audit(1713556734.896:5070):
proctitle=746F756368002F72756E2F7265626F6F742D7265717569726564
type=PATH msg=audit(1713556734.896:5070): item=1
name="/run/reboot-required" inode=7586 dev=00:17 mode=0100644 ouid=0 ogid=0
rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0
cap_frootid=0
type=PATH msg=audit(1713556734.896:5070): item=0 name="/run/" inode=1
dev=00:17 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0
cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1713556734.896:5070):
cwd="/home/andy/src/ncmpcpp-setup"
type=SYSCALL msg=audit(1713556734.896:5070): arch=c000003e syscall=257
success=yes exit=3 a0=ffffff9c a1=7ffd7680a74a a2=941 a3=1b6 items=2
ppid=3314410 pid=3314411 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
sgid=0 fsgid=0 tty=pts43 ses=3 comm="touch" exe="/usr/bin/touch"
subj=unconfined key="reboot-required"
----
time->Fri Apr 19 20:58:59 2024
type=PROCTITLE msg=audit(1713556739.308:5077):
proctitle=726D002F72756E2F7265626F6F742D7265717569726564
type=PATH msg=audit(1713556739.308:5077): item=1
name="/run/reboot-required" inode=7586 dev=00:17 mode=0100644 ouid=0 ogid=0
rdev=00:00 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0
cap_frootid=0
type=PATH msg=audit(1713556739.308:5077): item=0 name="/run/" inode=1
dev=00:17 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0
cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1713556739.308:5077):
cwd="/home/andy/src/ncmpcpp-setup"
type=SYSCALL msg=audit(1713556739.308:5077): arch=c000003e syscall=263
success=yes exit=0 a0=ffffff9c a1=5610f7a7c4a0 a2=0 a3=7f8863f59f60 items=2
ppid=3314463 pid=3314464 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
sgid=0 fsgid=0 tty=pts43 ses=3 comm="rm" exe="/usr/bin/rm"
subj=unconfined
key="reboot-required"
Note from the "type=SYSCALL" results you can see:
----
pid=3314411
uid=0
tty=pts43
comm="touch"
exe="/usr/bin/touch"
----
pid=331446400
uid=0
tty=pts43
comm="rm"
exe="/usr/bin/rm"
amongst other things…
Delete all rules again with:
$ sudo auditctl -D
With those rules installed you should get logs for every time a
process creates or removes that file path.
A simpler thing is inotifywait but as far as I know you don't get
access to what did the change there, only that a change has
happened.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
BitFolk Users mailing list <users(a)mailman.bitfolk.com>
You're subscribed as <murray.crane(a)gmail.com>
Unsubscribe: <
https://mailman.bitfolk.com/mailman/postorius/lists/users.mailman.bitfolk.c…
221
days inactive
246
days old
6 comments
3 participants
participants (3)
-
Andy Smith -
Ian -
Murray Crane