Incidents | sota staircase

Jellyfin recovered

Fri, 17 Jul 2026 07:48:40 +0000

Jellyfin recovered

Immich recovered

Fri, 17 Jul 2026 07:48:15 +0000

Immich recovered

Git Forge recovered

Fri, 17 Jul 2026 07:46:55 +0000

Git Forge recovered

Object Store recovered

Fri, 17 Jul 2026 07:46:06 +0000

Object Store recovered

Object Store Health recovered

Fri, 17 Jul 2026 07:46:01 +0000

Object Store Health recovered

Object Store went down

Thu, 16 Jul 2026 22:34:41 +0000

Object Store went down

Immich went down

Thu, 16 Jul 2026 22:33:52 +0000

Immich went down

Jellyfin went down

Thu, 16 Jul 2026 22:33:26 +0000

Jellyfin went down

Git Forge went down

Thu, 16 Jul 2026 22:32:47 +0000

Git Forge went down

Object Store Health went down

Thu, 16 Jul 2026 22:32:07 +0000

Object Store Health went down

WE ARE GETTING A KVM!

Fri, 03 Jul 2026 11:23:47 -0000

Maintenance completed

Git Forge recovered

Fri, 03 Jul 2026 11:10:51 +0000

Git Forge recovered

Git Forge went down

Fri, 03 Jul 2026 11:07:54 +0000

Git Forge went down

Git Forge recovered

Fri, 03 Jul 2026 10:58:53 +0000

Git Forge recovered

Immich recovered

Fri, 03 Jul 2026 10:57:12 +0000

Immich recovered

Immich went down

Fri, 03 Jul 2026 10:54:14 +0000

Immich went down

Git Forge went down

Fri, 03 Jul 2026 10:53:18 +0000

Git Forge went down

Git Forge recovered

Fri, 03 Jul 2026 10:37:51 +0000

Git Forge recovered

Object Store Health recovered

Fri, 03 Jul 2026 10:37:07 +0000

Object Store Health recovered

Object Store recovered

Fri, 03 Jul 2026 10:36:39 +0000

Object Store recovered

Immich recovered

Fri, 03 Jul 2026 10:36:15 +0000

Immich recovered

Jellyfin recovered

Fri, 03 Jul 2026 10:35:54 +0000

Jellyfin recovered

Object Store went down

Fri, 03 Jul 2026 09:57:52 +0000

Object Store went down

Immich went down

Fri, 03 Jul 2026 09:57:19 +0000

Immich went down

Jellyfin went down

Fri, 03 Jul 2026 09:57:17 +0000

Jellyfin went down

Git Forge went down

Fri, 03 Jul 2026 09:55:59 +0000

Git Forge went down

Object Store Health went down

Fri, 03 Jul 2026 09:55:28 +0000

Object Store Health went down

WE ARE GETTING A KVM!

Fri, 03 Jul 2026 09:53:00 -0000

so the server has to go DOWN to plug in mobo power control

Git Forge recovered

Sun, 28 Jun 2026 10:16:54 +0000

Git Forge recovered

Immich recovered

Sun, 28 Jun 2026 10:15:25 +0000

Immich recovered

Immich went down

Sun, 28 Jun 2026 10:12:56 +0000

Immich went down

Git Forge went down

Sun, 28 Jun 2026 10:11:45 +0000

Git Forge went down

server went down, and we know

Thu, 25 Jun 2026 16:48:00 -0000

back online

Immich recovered

Thu, 25 Jun 2026 16:45:07 +0000

Immich recovered

server went down, and we know

Thu, 25 Jun 2026 16:45:00 -0000

the fix has been applied, verified and the nixos system flake no longer hard-depends on md0 coming up; this should no longer happen with disastrous recovery setiup needed in the future server will be replaced in a cupboard and come back online soon

Jellyfin recovered

Thu, 25 Jun 2026 16:44:49 +0000

Jellyfin recovered

Object Store Health recovered

Thu, 25 Jun 2026 16:42:58 +0000

Object Store Health recovered

Object Store recovered

Thu, 25 Jun 2026 16:42:35 +0000

Object Store recovered

Git Forge recovered

Thu, 25 Jun 2026 16:37:53 +0000

Git Forge recovered

server went down, and we know

Thu, 25 Jun 2026 16:13:00 -0000

a reverted uuid change in the nixos hardware configuration has only shown itself to be erroneous due to the rebooting of the system during a power surge this uuid will be updated, and the system will be rebuilt from another live environment

server went down, and we know

Wed, 24 Jun 2026 16:12:00 -0000

nixos, the server's linux distro, is unable to find the bulk data raid array at boot however, it is visible and r/w-able in fedora thanks to livecd auto-assembling even if a disk has died, the 3x8TB hdds are set up in a RAID 10 mdadm array, meaning one drive failure will result in data survival

Jellyfin went down

Wed, 24 Jun 2026 11:33:15 +0000

Jellyfin went down

server went down, and we know

Wed, 24 Jun 2026 11:20:00 -0000

🚨 server is down, and i know, and am fixing it affected services: all forgejo @ forge.joshwel.co affected users: n/a objectstore @ datashower.joshwel.co affected users: me and ryan immich @ i.scorchthinks.dev affected users: ryan-side + cmm album r/w jellyfin @ watch.scorchthinks.dev affected users: me and ryan hope to be back up by 1940 earliest, no upper bound on latest

Object Store Health went down

Wed, 24 Jun 2026 10:55:11 +0000

Object Store Health went down

Git Forge went down

Wed, 24 Jun 2026 10:53:24 +0000

Git Forge went down

Object Store went down

Wed, 24 Jun 2026 10:51:58 +0000

Object Store went down

Immich went down

Wed, 24 Jun 2026 10:49:14 +0000

Immich went down

separating hot and bulk data

Tue, 23 Jun 2026 09:19:00 -0000

services are back online

Git Forge recovered

Tue, 23 Jun 2026 09:13:51 +0000

Git Forge recovered

Object Store Health recovered

Tue, 23 Jun 2026 09:13:01 +0000

Object Store Health recovered

Object Store recovered

Tue, 23 Jun 2026 09:12:32 +0000

Object Store recovered

Immich recovered

Tue, 23 Jun 2026 09:12:13 +0000

Immich recovered

Git Forge went down

Tue, 23 Jun 2026 08:47:50 +0000

Git Forge went down

Object Store Health went down

Tue, 23 Jun 2026 08:46:53 +0000

Object Store Health went down

Object Store went down

Tue, 23 Jun 2026 08:46:33 +0000

Object Store went down

Immich went down

Tue, 23 Jun 2026 08:45:50 +0000

Immich went down

Immich recovered

Tue, 23 Jun 2026 07:39:19 +0000

Immich recovered

Immich went down

Tue, 23 Jun 2026 07:15:22 +0000

Immich went down

Immich recovered

Tue, 23 Jun 2026 06:45:16 +0000

Immich recovered

Git Forge recovered

Tue, 23 Jun 2026 06:38:31 +0000

Git Forge recovered

Object Store Health recovered

Tue, 23 Jun 2026 06:37:15 +0000

Object Store Health recovered

Object Store recovered

Tue, 23 Jun 2026 06:36:45 +0000

Object Store recovered

Object Store went down

Tue, 23 Jun 2026 05:40:29 +0000

Object Store went down

Object Store Health went down

Tue, 23 Jun 2026 05:37:22 +0000

Object Store Health went down

Immich went down

Tue, 23 Jun 2026 05:27:21 +0000

Immich went down

Git Forge went down

Tue, 23 Jun 2026 05:25:58 +0000

Git Forge went down

separating hot and bulk data

Tue, 23 Jun 2026 05:20:00 -0000

moving container dbs and configs into nvme to prevent service slowdowns; letting bulk data like repos, media, and objects take its time stretch goal is also docker root hopefully will drastically improve latency

disks are saturated

Tue, 23 Jun 2026 05:19:00 -0000

we good but not for long

Object Store recovered

Tue, 23 Jun 2026 05:10:18 +0000

Object Store recovered

Object Store Health recovered

Tue, 23 Jun 2026 05:10:03 +0000

Object Store Health recovered

Git Forge recovered

Tue, 23 Jun 2026 05:07:52 +0000

Git Forge recovered

Git Forge went down

Tue, 23 Jun 2026 03:55:51 +0000

Git Forge went down

Object Store Health went down

Tue, 23 Jun 2026 03:55:06 +0000

Object Store Health went down

Object Store went down

Tue, 23 Jun 2026 03:54:40 +0000

Object Store went down

disks are saturated

Tue, 23 Jun 2026 03:19:00 -0000

seeding a little too much linux ISOs for poor SMR hdds trying out things; bringing media stack down, etc

Git Forge recovered

Sat, 20 Jun 2026 00:43:50 +0000

Git Forge recovered

Git Forge went down

Sat, 20 Jun 2026 00:41:09 +0000

Git Forge went down

Git Forge recovered

Tue, 16 Jun 2026 01:16:55 +0000

Git Forge recovered

Git Forge went down

Tue, 16 Jun 2026 01:14:53 +0000

Git Forge went down

Immich recovered

Wed, 03 Jun 2026 21:18:17 +0000

Immich recovered

Immich went down

Wed, 03 Jun 2026 21:15:54 +0000

Immich went down

Immich recovered

Wed, 03 Jun 2026 21:03:24 +0000

Immich recovered

Immich went down

Wed, 03 Jun 2026 21:00:49 +0000

Immich went down

Immich recovered

Wed, 03 Jun 2026 01:36:09 +0000

Immich recovered

Immich went down

Wed, 03 Jun 2026 01:33:52 +0000

Immich went down

Object Store recovered

Thu, 28 May 2026 11:24:39 +0000

Object Store recovered

Immich recovered

Thu, 28 May 2026 11:24:15 +0000

Immich recovered

Immich went down

Thu, 28 May 2026 11:21:43 +0000

Immich went down

Object Store went down

Thu, 28 May 2026 11:19:29 +0000

Object Store went down

Immich recovered

Mon, 18 May 2026 19:33:11 +0000

Immich recovered

Git Forge recovered

Mon, 18 May 2026 19:31:52 +0000

Git Forge recovered

Object Store Health recovered

Mon, 18 May 2026 19:31:06 +0000

Object Store Health recovered

Object Store recovered

Mon, 18 May 2026 19:30:49 +0000

Object Store recovered

Git Forge went down

Mon, 18 May 2026 17:31:54 +0000

Git Forge went down

Object Store Health went down

Mon, 18 May 2026 17:31:05 +0000

Object Store Health went down

Object Store went down

Mon, 18 May 2026 17:30:39 +0000

Object Store went down

Immich went down

Mon, 18 May 2026 17:30:20 +0000

Immich went down

minecraft killed the ssd

Sun, 17 May 2026 16:35:00 -0000

disk saturation from emergency recovery of /mnt/ssd is done, we up

minecraft killed the ssd

Sun, 17 May 2026 12:46:00 -0000

disks are saturated; server performance will be slow till est noon tomorrow, 1200 18/5/2026 UTC+8 (SGT)

Object Store recovered

Sun, 17 May 2026 12:18:33 +0000

Object Store recovered

Object Store went down

Sun, 17 May 2026 12:12:37 +0000

Object Store went down

Object Store Health recovered

Sun, 17 May 2026 10:13:13 +0000

Object Store Health recovered

Object Store recovered

Sun, 17 May 2026 10:12:40 +0000

Object Store recovered

Immich recovered

Sun, 17 May 2026 10:12:15 +0000

Immich recovered

minecraft killed the ssd

Sun, 17 May 2026 10:12:00 -0000

should be up (except mc); monitoring stuff

Git Forge recovered

Sun, 17 May 2026 10:10:52 +0000

Git Forge recovered

minecraft killed the ssd

Fri, 15 May 2026 18:42:00 -0000

due to ryan and me having zero time, the restoration will be done on sunday

Object Store Health went down

Sun, 10 May 2026 15:07:03 +0000

Object Store Health went down

Object Store went down

Sun, 10 May 2026 15:06:29 +0000

Object Store went down

Immich went down

Sun, 10 May 2026 15:06:19 +0000

Immich went down

Git Forge went down

Sun, 10 May 2026 15:05:22 +0000

Git Forge went down

minecraft killed the ssd

Sun, 10 May 2026 15:00:00 -0000

server boots off an nvme, core services like the forge, immich, and object store are on a hdd raid array this only affects a personal minecraft instance but we're working on data recovery

lets slime on mama

Sat, 10 Jan 2026 18:32:00 -0000

we up

lets slime on mama

Tue, 06 Jan 2026 11:45:00 -0000

ocis is GONE buh bye there will be planned migration of data this is the reckoning thanks bye

everything is DOWN

Tue, 06 Jan 2026 11:30:52 -0000

we are going to migrate our main raid array from RAID5 to RAID10 (1+0) because ryan forgot the drives were SMR instead of CMR retort by ryan: "i did not know there was this distinction" retort-retort by mark: i forgot tbh

system upgrade

Sat, 03 Jan 2026 11:30:07 -0000

moving from NixOS 25.05 to NixOS 25.11, and will reboot afterwards

lets slime on mama

Sat, 03 Jan 2026 09:40:00 -0000

disks are saturated

the server is down, and now we know

Tue, 02 Dec 2025 15:59:00 -0000

resolved

the server is down, and now we know

Sat, 29 Nov 2025 08:05:00 -0000

all bare metal services are currently unresponsive and time out; we are aware and will resolve it in the following hours. any affected users if at all may consider reaching out to mark at joshwel dot co for urgent data retrieval or likewise

object store is being redeployed!!

Fri, 05 Sep 2025 11:27:25 -0000

was previously a monitored preprod/test instance but i like it!!! so i'm redeploying it with slightly different configs!!!

IMMICH IS DOWN, MIGRATING DATA

Tue, 02 Sep 2025 16:00:22 -0000

data is currently being migrated, disks are probably saturated

EVERYTHING IS DOWN

Thu, 28 Aug 2025 03:00:00 -0000

we are fully upgrading the server because it will explode soon blah blah to serve you better yall dont have school bro but like if needed just tele us or email with (some) of our hearts, Mark Joshwel Ryan Lin

the server is down, and we knew

Tue, 24 Jun 2025 15:17:00 -0000

an unexpected router restart had messed with port forwarding, after a twiddle around router settings (turning port forwarding on and off), we're back up.

the server is down, and we know

Fri, 13 Jun 2025 07:48:00 -0000

it was a bootloop, server restored around 1230h SGT

the server is down, and we know

Thu, 12 Jun 2025 22:20:00 -0000

server is down but ryan is sleeping; it could be a: - heater-related power trip (in which downtime up to six hours may be expected) - or, a router failure (in which downtime up to an hour is expected)

the server is down, and we know

Wed, 04 Jun 2025 01:15:00 -0000

aight we up should be good

the server is down, and we know

Wed, 04 Jun 2025 01:09:00 -0000

a power trip has caused shit to fly, and services are expected to resume around noon for any qualms contact ryan (at) scorchthinks (dot) dev

la poisson du forge is down, and we know

Tue, 18 Mar 2025 12:31:00 -0000

the server restoration post-power fault did not properly start the backend before the frontend. resulting in service disruption although the status monitor remained up. the monitored url has been updated and the container stack was fully restarted in proper. no data was lost.

la poisson du forge is down, and we know

Sun, 16 Mar 2025 05:42:00 -0000

due to a server power fault, the forge's data is currently inaccessible. an investigation will be launched in the following days. any requests for data can be sent to ryan (at) scorchthinks (dot) dev