Incidents | sota staircase

Git Forge recovered

Sun, 29 Mar 2026 22:31:51 +0000

Git Forge recovered

Object Store Health recovered

Sun, 29 Mar 2026 22:30:59 +0000

Object Store Health recovered

Object Store recovered

Sun, 29 Mar 2026 22:30:33 +0000

Object Store recovered

Immich recovered

Sun, 29 Mar 2026 22:30:16 +0000

Immich recovered

Object Store Health went down

Sun, 29 Mar 2026 22:16:57 +0000

Object Store Health went down

Object Store went down

Sun, 29 Mar 2026 22:16:31 +0000

Object Store went down

Immich went down

Sun, 29 Mar 2026 22:15:55 +0000

Immich went down

Git Forge went down

Sun, 29 Mar 2026 22:14:50 +0000

Git Forge went down

Git Forge recovered

Sun, 29 Mar 2026 14:01:57 +0000

Git Forge recovered

Git Forge went down

Sun, 29 Mar 2026 13:59:45 +0000

Git Forge went down

Git Forge recovered

Mon, 09 Mar 2026 21:04:51 +0000

Git Forge recovered

Object Store Health recovered

Mon, 09 Mar 2026 21:04:08 +0000

Object Store Health recovered

Object Store recovered

Mon, 09 Mar 2026 21:03:33 +0000

Object Store recovered

Git Forge went down

Mon, 09 Mar 2026 20:17:50 +0000

Git Forge went down

Object Store went down

Mon, 09 Mar 2026 20:16:33 +0000

Object Store went down

Object Store Health went down

Mon, 09 Mar 2026 20:13:59 +0000

Object Store Health went down

Immich recovered

Fri, 27 Feb 2026 14:39:24 +0000

Immich recovered

Immich went down

Fri, 27 Feb 2026 14:36:55 +0000

Immich went down

Object Store Health recovered

Mon, 09 Feb 2026 23:25:13 +0000

Object Store Health recovered

Object Store recovered

Mon, 09 Feb 2026 23:24:37 +0000

Object Store recovered

Immich recovered

Mon, 09 Feb 2026 23:24:24 +0000

Immich recovered

Git Forge recovered

Mon, 09 Feb 2026 23:22:57 +0000

Git Forge recovered

Git Forge went down

Mon, 09 Feb 2026 14:20:35 +0000

Git Forge went down

Object Store Health went down

Mon, 09 Feb 2026 14:19:55 +0000

Object Store Health went down

Object Store went down

Mon, 09 Feb 2026 14:19:29 +0000

Object Store went down

Immich went down

Mon, 09 Feb 2026 14:18:43 +0000

Immich went down

Object Store Health recovered

Mon, 09 Feb 2026 14:01:08 +0000

Object Store Health recovered

Immich recovered

Mon, 09 Feb 2026 14:00:22 +0000

Immich recovered

Git Forge recovered

Mon, 09 Feb 2026 13:58:52 +0000

Git Forge recovered

Immich went down

Mon, 09 Feb 2026 13:57:14 +0000

Immich went down

Git Forge went down

Mon, 09 Feb 2026 13:56:35 +0000

Git Forge went down

Object Store Health went down

Mon, 09 Feb 2026 13:55:50 +0000

Object Store Health went down

Immich recovered

Mon, 09 Feb 2026 11:57:34 +0000

Immich recovered

Git Forge recovered

Mon, 09 Feb 2026 11:55:55 +0000

Git Forge recovered

Git Forge went down

Mon, 09 Feb 2026 11:53:22 +0000

Git Forge went down

Immich went down

Mon, 09 Feb 2026 11:51:58 +0000

Immich went down

Object Store Health recovered

Sun, 18 Jan 2026 23:07:14 +0000

Object Store Health recovered

Object Store Health went down

Sun, 18 Jan 2026 23:04:02 +0000

Object Store Health went down

Immich recovered

Tue, 13 Jan 2026 14:42:36 +0000

Immich recovered

Immich went down

Tue, 13 Jan 2026 06:09:23 +0000

Immich went down

lets slime on mama

Sat, 10 Jan 2026 18:32:00 -0000

we up

lets slime on mama

Tue, 06 Jan 2026 11:45:00 -0000

ocis is GONE buh bye there will be planned migration of data this is the reckoning thanks bye

everything is DOWN

Tue, 06 Jan 2026 11:30:52 -0000

we are going to migrate our main raid array from RAID5 to RAID10 (1+0) because ryan forgot the drives were SMR instead of CMR retort by ryan: "i did not know there was this distinction" retort-retort by mark: i forgot tbh

system upgrade

Sat, 03 Jan 2026 11:30:07 -0000

moving from NixOS 25.05 to NixOS 25.11, and will reboot afterwards

lets slime on mama

Sat, 03 Jan 2026 09:40:00 -0000

disks are saturated

the server is down, and now we know

Tue, 02 Dec 2025 15:59:00 -0000

resolved

the server is down, and now we know

Sat, 29 Nov 2025 08:05:00 -0000

all bare metal services are currently unresponsive and time out; we are aware and will resolve it in the following hours. any affected users if at all may consider reaching out to mark at joshwel dot co for urgent data retrieval or likewise

object store is being redeployed!!

Fri, 05 Sep 2025 11:27:25 -0000

was previously a monitored preprod/test instance but i like it!!! so i'm redeploying it with slightly different configs!!!

IMMICH IS DOWN, MIGRATING DATA

Tue, 02 Sep 2025 16:00:22 -0000

data is currently being migrated, disks are probably saturated

EVERYTHING IS DOWN

Tue, 02 Sep 2025 16:00:00 -0000

Maintenance completed

EVERYTHING IS DOWN

Thu, 28 Aug 2025 03:00:00 -0000

we are fully upgrading the server because it will explode soon blah blah to serve you better yall dont have school bro but like if needed just tele us or email with (some) of our hearts, Mark Joshwel Ryan Lin

the server is down, and we knew

Tue, 24 Jun 2025 15:17:00 -0000

an unexpected router restart had messed with port forwarding, after a twiddle around router settings (turning port forwarding on and off), we're back up.

the server is down, and we know

Fri, 13 Jun 2025 07:48:00 -0000

it was a bootloop, server restored around 1230h SGT

the server is down, and we know

Thu, 12 Jun 2025 22:20:00 -0000

server is down but ryan is sleeping; it could be a: - heater-related power trip (in which downtime up to six hours may be expected) - or, a router failure (in which downtime up to an hour is expected)

the server is down, and we know

Wed, 04 Jun 2025 01:15:00 -0000

aight we up should be good

the server is down, and we know

Wed, 04 Jun 2025 01:09:00 -0000

a power trip has caused shit to fly, and services are expected to resume around noon for any qualms contact ryan (at) scorchthinks (dot) dev

la poisson du forge is down, and we know

Tue, 18 Mar 2025 12:31:00 -0000

the server restoration post-power fault did not properly start the backend before the frontend. resulting in service disruption although the status monitor remained up. the monitored url has been updated and the container stack was fully restarted in proper. no data was lost.

la poisson du forge is down, and we know

Sun, 16 Mar 2025 05:42:00 -0000

due to a server power fault, the forge's data is currently inaccessible. an investigation will be launched in the following days. any requests for data can be sent to ryan (at) scorchthinks (dot) dev