Back to overview
Resolved

server went down, and we know

Jun 24, 2026 at 11:20am UTC
Affected services
Jellyfin
Git Forge
Immich
Object Store
Object Store Health

Resolved
Jun 25, 2026 at 4:48pm UTC

back online

Updated
Jun 25, 2026 at 4:45pm UTC

the fix has been applied, verified and the nixos system flake no longer hard-depends on md0 coming up; this should no longer happen with disastrous recovery setiup needed in the future

server will be replaced in a cupboard and come back online soon

Updated
Jun 25, 2026 at 4:13pm UTC

a reverted uuid change in the nixos hardware configuration has only shown itself to be erroneous due to the rebooting of the system during a power surge

this uuid will be updated, and the system will be rebuilt from another live environment

Updated
Jun 24, 2026 at 4:12pm UTC

nixos, the server's linux distro, is unable to find the bulk data raid array at boot

however, it is visible and r/w-able in fedora thanks to livecd auto-assembling

even if a disk has died, the 3x8TB hdds are set up in a RAID 10 mdadm array, meaning one drive failure will result in data survival

Created
Jun 24, 2026 at 11:20am UTC

🚨 server is down, and i know, and am fixing it

affected services: all
forgejo @ forge.joshwel.co
affected users: n/a
objectstore @ datashower.joshwel.co
affected users: me and ryan
immich @ i.scorchthinks.dev
affected users: ryan-side + cmm album r/w
jellyfin @ watch.scorchthinks.dev
affected users: me and ryan

hope to be back up by 1940 earliest, no upper bound on latest