server went down, and we know
Resolved
Jun 25, 2026 at 4:48pm UTC
back online
Affected services
Updated
Jun 25, 2026 at 4:45pm UTC
the fix has been applied, verified and the nixos system flake no longer hard-depends on md0 coming up; this should no longer happen with disastrous recovery setiup needed in the future
server will be replaced in a cupboard and come back online soon
Affected services
Updated
Jun 25, 2026 at 4:13pm UTC
a reverted uuid change in the nixos hardware configuration has only shown itself to be erroneous due to the rebooting of the system during a power surge
this uuid will be updated, and the system will be rebuilt from another live environment
Affected services
Updated
Jun 24, 2026 at 4:12pm UTC
nixos, the server's linux distro, is unable to find the bulk data raid array at boot
however, it is visible and r/w-able in fedora thanks to livecd auto-assembling
even if a disk has died, the 3x8TB hdds are set up in a RAID 10 mdadm array, meaning one drive failure will result in data survival
Affected services
Created
Jun 24, 2026 at 11:20am UTC
🚨 server is down, and i know, and am fixing it
affected services: all
forgejo @ forge.joshwel.co
affected users: n/a
objectstore @ datashower.joshwel.co
affected users: me and ryan
immich @ i.scorchthinks.dev
affected users: ryan-side + cmm album r/w
jellyfin @ watch.scorchthinks.dev
affected users: me and ryan
hope to be back up by 1940 earliest, no upper bound on latest
Affected services