/dev/null: Permission denied on Alpine Linux

I upgraded two virtual machines to Alpine Linux 3.24 last night. One worked perfectly. The other came back up after reboot, but the services it runs didn’t start up (reminder to self: this is why you don’t run upgrades late at night), and when I logged in (fortunately SSH was still working!) I saw that some of the scripts were throwing errors like this:

 -ash: can't create /dev/null: Permission denied

Wait, what?

Sure enough:

ls -l /dev/null
crw-rw----    1 root     root        1,   3 Jun 10 10:52 /dev/null

The obvious solution was to fix the permissions on /dev/null:

doas chmod a+rw /dev/null

But that didn’t persist when I rebooted, because devtmpfs isn’t on disk, it’s in memory. Besides, I needed to check for other devices that might have broken permissions. I re-ran the upgrade, followed by repair…

doas apk upgrade --available
doas apk fix --directory-permissions
doas apk fix -x

But that didn’t work either. Because /dev isn’t built from packages, it’s built by a device manager. I found a bunch of forum and Stack Exchange posts on how to fix the problem in udev on distros like Arch or Gentoo, but Alpine doesn’t use udev, so none of those suggestions were even relevant.

The Fix

What did fix the problem was just resetting Alpine’s device manager, in this case mdev:

doas setup-devd mdev

Permissions stayed fixed when I rebooted it again, and everything started up correctly!

I assume doing the same with mdevd or eudev would have worked if a system using one of those broke the same way.

Possible Cause

I’m not sure why one of them broke and the other was fine.

It’s an older VM, and I started it on an older version of Alpine, so it may have been something left over that didn’t break on 3.23, but broke on 3.24.

I also forgot to add --available to the apk upgrade command the first time through, and I don’t remember if I re-ran it before rebooting the first time or after. It may have broken during the reboot with the partial upgrade.

It could be either of those, or something else entirely.