Status updates | sota staircase

Maintenance: maintenance

Fri, 02 Oct 2026 08:50:04 -0000

Maintenance completed

Maintenance: maintenance

Fri, 02 Oct 2026 08:37:04 -0000

culling the server into 12c + powersave; may reboot

Maintenance: this is the end

Thu, 01 Oct 2026 18:02:07 -0000

and we up

Maintenance: this is the end

Thu, 01 Oct 2026 18:02:07 -0000

Maintenance completed

Maintenance: this is the end

Mon, 28 Sep 2026 01:30:00 -0000

ryan is renovating

Maintenance: WE ARE GETTING A KVM!

Fri, 03 Jul 2026 09:53:00 -0000

so the server has to go DOWN to plug in mobo power control

server went down, and we know

Thu, 25 Jun 2026 16:48:00 -0000

back online

server went down, and we know

Thu, 25 Jun 2026 16:45:00 -0000

the fix has been applied, verified and the nixos system flake no longer hard-depends on md0 coming up; this should no longer happen with disastrous recovery setiup needed in the future server will be replaced in a cupboard and come back online soon

server went down, and we know

Thu, 25 Jun 2026 16:13:00 -0000

a reverted uuid change in the nixos hardware configuration has only shown itself to be erroneous due to the rebooting of the system during a power surge this uuid will be updated, and the system will be rebuilt from another live environment

server went down, and we know

Wed, 24 Jun 2026 16:12:00 -0000

nixos, the server's linux distro, is unable to find the bulk data raid array at boot however, it is visible and r/w-able in fedora thanks to livecd auto-assembling even if a disk has died, the 3x8TB hdds are set up in a RAID 10 mdadm array, meaning one drive failure will result in data survival

server went down, and we know

Wed, 24 Jun 2026 11:20:00 -0000

🚨 server is down, and i know, and am fixing it affected services: all forgejo @ forge.joshwel.co affected users: n/a objectstore @ datashower.joshwel.co affected users: me and ryan immich @ i.scorchthinks.dev affected users: ryan-side + cmm album r/w jellyfin @ watch.scorchthinks.dev affected users: me and ryan hope to be back up by 1940 earliest, no upper bound on latest

separating hot and bulk data

Tue, 23 Jun 2026 09:19:00 -0000

services are back online

separating hot and bulk data

Tue, 23 Jun 2026 05:20:00 -0000

moving container dbs and configs into nvme to prevent service slowdowns; letting bulk data like repos, media, and objects take its time stretch goal is also docker root hopefully will drastically improve latency

disks are saturated

Tue, 23 Jun 2026 05:19:00 -0000

we good but not for long

disks are saturated

Tue, 23 Jun 2026 03:19:00 -0000

seeding a little too much linux ISOs for poor SMR hdds trying out things; bringing media stack down, etc

minecraft killed the ssd

Sun, 17 May 2026 16:35:00 -0000

disk saturation from emergency recovery of /mnt/ssd is done, we up

minecraft killed the ssd

Sun, 17 May 2026 12:46:00 -0000

disks are saturated; server performance will be slow till est noon tomorrow, 1200 18/5/2026 UTC+8 (SGT)

minecraft killed the ssd

Sun, 17 May 2026 10:12:00 -0000

should be up (except mc); monitoring stuff

minecraft killed the ssd

Fri, 15 May 2026 18:42:00 -0000

due to ryan and me having zero time, the restoration will be done on sunday

minecraft killed the ssd

Sun, 10 May 2026 15:00:00 -0000

server boots off an nvme, core services like the forge, immich, and object store are on a hdd raid array this only affects a personal minecraft instance but we're working on data recovery

lets slime on mama

Sat, 10 Jan 2026 18:32:00 -0000

we up

lets slime on mama

Tue, 06 Jan 2026 11:45:00 -0000

ocis is GONE buh bye there will be planned migration of data this is the reckoning thanks bye

Maintenance: everything is DOWN

Tue, 06 Jan 2026 11:30:52 -0000

we are going to migrate our main raid array from RAID5 to RAID10 (1+0) because ryan forgot the drives were SMR instead of CMR retort by ryan: "i did not know there was this distinction" retort-retort by mark: i forgot tbh

Maintenance: system upgrade

Sat, 03 Jan 2026 11:30:07 -0000

moving from NixOS 25.05 to NixOS 25.11, and will reboot afterwards

lets slime on mama

Sat, 03 Jan 2026 09:40:00 -0000

disks are saturated

the server is down, and now we know

Tue, 02 Dec 2025 15:59:00 -0000

resolved

the server is down, and now we know

Sat, 29 Nov 2025 08:05:00 -0000

all bare metal services are currently unresponsive and time out; we are aware and will resolve it in the following hours. any affected users if at all may consider reaching out to mark at joshwel dot co for urgent data retrieval or likewise

Maintenance: object store is being redeployed!!

Fri, 05 Sep 2025 11:27:25 -0000

was previously a monitored preprod/test instance but i like it!!! so i'm redeploying it with slightly different configs!!!

Maintenance: IMMICH IS DOWN, MIGRATING DATA

Tue, 02 Sep 2025 16:00:22 -0000

data is currently being migrated, disks are probably saturated

Maintenance: EVERYTHING IS DOWN

Thu, 28 Aug 2025 03:00:00 -0000

we are fully upgrading the server because it will explode soon blah blah to serve you better yall dont have school bro but like if needed just tele us or email with (some) of our hearts, Mark Joshwel Ryan Lin

the server is down, and we knew

Tue, 24 Jun 2025 15:17:00 -0000

an unexpected router restart had messed with port forwarding, after a twiddle around router settings (turning port forwarding on and off), we're back up.

the server is down, and we know

Fri, 13 Jun 2025 07:48:00 -0000

it was a bootloop, server restored around 1230h SGT

the server is down, and we know

Thu, 12 Jun 2025 22:20:00 -0000

server is down but ryan is sleeping; it could be a: - heater-related power trip (in which downtime up to six hours may be expected) - or, a router failure (in which downtime up to an hour is expected)

the server is down, and we know

Wed, 04 Jun 2025 01:15:00 -0000

aight we up should be good

the server is down, and we know

Wed, 04 Jun 2025 01:09:00 -0000

a power trip has caused shit to fly, and services are expected to resume around noon for any qualms contact ryan (at) scorchthinks (dot) dev

la poisson du forge is down, and we know

Tue, 18 Mar 2025 12:31:00 -0000

the server restoration post-power fault did not properly start the backend before the frontend. resulting in service disruption although the status monitor remained up. the monitored url has been updated and the container stack was fully restarted in proper. no data was lost.

la poisson du forge is down, and we know

Sun, 16 Mar 2025 05:42:00 -0000

due to a server power fault, the forge's data is currently inaccessible. an investigation will be launched in the following days. any requests for data can be sent to ryan (at) scorchthinks (dot) dev