Incidents | sota staircase

Immich recovered

Thu, 11 Dec 2025 06:21:16 +0000

Immich recovered

Git Forge recovered

Thu, 11 Dec 2025 06:19:52 +0000

Git Forge recovered

Object Store Health recovered

Thu, 11 Dec 2025 06:19:04 +0000

Object Store Health recovered

Object Store recovered

Thu, 11 Dec 2025 06:18:37 +0000

Object Store recovered

ownCloud Infinite Scale recovered

Thu, 11 Dec 2025 06:18:31 +0000

ownCloud Infinite Scale recovered

ownCloud Infinite Scale went down

Thu, 11 Dec 2025 06:15:58 +0000

ownCloud Infinite Scale went down

Object Store Health went down

Thu, 11 Dec 2025 06:13:58 +0000

Object Store Health went down

Object Store went down

Thu, 11 Dec 2025 06:13:30 +0000

Object Store went down

Immich went down

Thu, 11 Dec 2025 06:12:52 +0000

Immich went down

Git Forge went down

Thu, 11 Dec 2025 06:11:49 +0000

Git Forge went down

Git Forge recovered

Mon, 08 Dec 2025 22:08:05 +0000

Git Forge recovered

Object Store Health recovered

Mon, 08 Dec 2025 22:07:13 +0000

Object Store Health recovered

Object Store recovered

Mon, 08 Dec 2025 22:06:44 +0000

Object Store recovered

Immich recovered

Mon, 08 Dec 2025 22:06:24 +0000

Immich recovered

ownCloud Infinite Scale recovered

Mon, 08 Dec 2025 22:05:39 +0000

ownCloud Infinite Scale recovered

Immich went down

Mon, 08 Dec 2025 08:05:43 +0000

Immich went down

Git Forge went down

Mon, 08 Dec 2025 08:04:38 +0000

Git Forge went down

ownCloud Infinite Scale went down

Mon, 08 Dec 2025 08:04:27 +0000

ownCloud Infinite Scale went down

Object Store Health went down

Mon, 08 Dec 2025 08:03:50 +0000

Object Store Health went down

Object Store went down

Mon, 08 Dec 2025 08:03:22 +0000

Object Store went down

the server is down, and now we know

Tue, 02 Dec 2025 15:59:00 -0000

resolved

the server is down, and now we know

Tue, 02 Dec 2025 15:59:00 -0000

resolved

the server is down, and now we know

Tue, 02 Dec 2025 15:59:00 -0000

resolved

the server is down, and now we know

Tue, 02 Dec 2025 15:59:00 -0000

resolved

the server is down, and now we know

Tue, 02 Dec 2025 15:59:00 -0000

resolved

Immich recovered

Tue, 02 Dec 2025 14:52:19 +0000

Immich recovered

Git Forge recovered

Tue, 02 Dec 2025 14:50:52 +0000

Git Forge recovered

Object Store Health recovered

Tue, 02 Dec 2025 14:50:05 +0000

Object Store Health recovered

Object Store recovered

Tue, 02 Dec 2025 14:49:35 +0000

Object Store recovered

ownCloud Infinite Scale recovered

Tue, 02 Dec 2025 14:47:56 +0000

ownCloud Infinite Scale recovered

Object Store went down

Sat, 29 Nov 2025 13:28:11 +0000

Object Store went down

Immich went down

Sat, 29 Nov 2025 13:27:51 +0000

Immich went down

Git Forge went down

Sat, 29 Nov 2025 13:26:34 +0000

Git Forge went down

ownCloud Infinite Scale went down

Sat, 29 Nov 2025 13:26:34 +0000

ownCloud Infinite Scale went down

Object Store Health went down

Sat, 29 Nov 2025 13:26:07 +0000

Object Store Health went down

Object Store recovered

Sat, 29 Nov 2025 12:16:12 +0000

Object Store recovered

Immich recovered

Sat, 29 Nov 2025 12:15:50 +0000

Immich recovered

ownCloud Infinite Scale recovered

Sat, 29 Nov 2025 12:14:30 +0000

ownCloud Infinite Scale recovered

Git Forge recovered

Sat, 29 Nov 2025 12:14:29 +0000

Git Forge recovered

Object Store Health recovered

Sat, 29 Nov 2025 12:13:40 +0000

Object Store Health recovered

the server is down, and now we know

Sat, 29 Nov 2025 08:05:00 -0000

all bare metal services are currently unresponsive and time out; we are aware and will resolve it in the following hours. any affected users if at all may consider reaching out to mark at joshwel dot co for urgent data retrieval or likewise

the server is down, and now we know

Sat, 29 Nov 2025 08:05:00 -0000

the server is down, and now we know

Sat, 29 Nov 2025 08:05:00 -0000

the server is down, and now we know

Sat, 29 Nov 2025 08:05:00 -0000

the server is down, and now we know

Sat, 29 Nov 2025 08:05:00 -0000

Object Store Health went down

Sat, 29 Nov 2025 07:50:32 +0000

Object Store Health went down

Object Store went down

Sat, 29 Nov 2025 07:49:58 +0000

Object Store went down

Immich went down

Sat, 29 Nov 2025 07:49:24 +0000

Immich went down

Git Forge went down

Sat, 29 Nov 2025 07:48:20 +0000

Git Forge went down

ownCloud Infinite Scale went down

Sat, 29 Nov 2025 07:48:09 +0000

ownCloud Infinite Scale went down

Immich recovered

Thu, 20 Nov 2025 02:00:45 +0000

Immich recovered

Immich went down

Thu, 20 Nov 2025 01:57:48 +0000

Immich went down

surplus documentation recovered

Tue, 18 Nov 2025 12:45:11 +0000

surplus documentation recovered

surplus documentation went down

Tue, 18 Nov 2025 11:42:19 +0000

surplus documentation went down

Immich recovered

Tue, 18 Nov 2025 08:58:06 +0000

Immich recovered

Immich went down

Tue, 18 Nov 2025 08:55:39 +0000

Immich went down

Immich recovered

Mon, 17 Nov 2025 01:26:37 +0000

Immich recovered

Immich went down

Mon, 17 Nov 2025 01:17:35 +0000

Immich went down

Immich recovered

Mon, 10 Nov 2025 13:33:57 +0000

Immich recovered

Immich went down

Mon, 10 Nov 2025 13:31:37 +0000

Immich went down

ownCloud Infinite Scale recovered

Wed, 15 Oct 2025 14:18:17 +0000

ownCloud Infinite Scale recovered

Git Forge recovered

Wed, 15 Oct 2025 14:18:16 +0000

Git Forge recovered

Immich recovered

Wed, 15 Oct 2025 14:16:55 +0000

Immich recovered

Git Forge went down

Wed, 15 Oct 2025 14:16:11 +0000

Git Forge went down

ownCloud Infinite Scale went down

Wed, 15 Oct 2025 14:15:54 +0000

ownCloud Infinite Scale went down

Immich went down

Wed, 15 Oct 2025 14:14:14 +0000

Immich went down

ownCloud Infinite Scale recovered

Wed, 15 Oct 2025 13:24:16 +0000

ownCloud Infinite Scale recovered

ownCloud Infinite Scale went down

Wed, 15 Oct 2025 13:22:01 +0000

ownCloud Infinite Scale went down

Object Store Health recovered

Tue, 07 Oct 2025 12:54:44 +0000

Object Store Health recovered

Object Store recovered

Tue, 07 Oct 2025 12:54:07 +0000

Object Store recovered

Immich recovered

Tue, 07 Oct 2025 12:53:47 +0000

Immich recovered

ownCloud Infinite Scale recovered

Tue, 07 Oct 2025 12:52:29 +0000

ownCloud Infinite Scale recovered

Git Forge recovered

Tue, 07 Oct 2025 12:52:29 +0000

Git Forge recovered

Git Forge went down

Tue, 07 Oct 2025 12:35:01 +0000

Git Forge went down

ownCloud Infinite Scale went down

Tue, 07 Oct 2025 12:31:49 +0000

ownCloud Infinite Scale went down

Object Store Health went down

Tue, 07 Oct 2025 12:31:34 +0000

Object Store Health went down

Object Store went down

Tue, 07 Oct 2025 12:31:02 +0000

Object Store went down

Immich went down

Tue, 07 Oct 2025 12:30:28 +0000

Immich went down

Object Store recovered

Sun, 21 Sep 2025 06:48:00 +0000

Object Store recovered

ownCloud Infinite Scale recovered

Sun, 21 Sep 2025 06:46:20 +0000

ownCloud Infinite Scale recovered

Git Forge recovered

Sun, 21 Sep 2025 06:46:17 +0000

Git Forge recovered

Immich recovered

Sun, 21 Sep 2025 06:44:37 +0000

Immich recovered

Git Forge went down

Sun, 21 Sep 2025 06:44:17 +0000

Git Forge went down

ownCloud Infinite Scale went down

Sun, 21 Sep 2025 06:44:08 +0000

ownCloud Infinite Scale went down

Object Store went down

Sun, 21 Sep 2025 06:42:53 +0000

Object Store went down

Immich went down

Sun, 21 Sep 2025 06:42:19 +0000

Immich went down

IMMICH IS DOWN, MIGRATING DATA

Sun, 07 Sep 2025 16:00:22 -0000

Maintenance completed

object store is being redeployed!!

Fri, 05 Sep 2025 11:35:25 -0000

Maintenance completed

object store is being redeployed!!

Fri, 05 Sep 2025 11:35:25 -0000

Maintenance completed

object store is being redeployed!!

Fri, 05 Sep 2025 11:27:25 -0000

was previously a monitored preprod/test instance but i like it!!! so i'm redeploying it with slightly different configs!!!

object store is being redeployed!!

Fri, 05 Sep 2025 11:27:25 -0000

was previously a monitored preprod/test instance but i like it!!! so i'm redeploying it with slightly different configs!!!

DISKS ARE SATURATED

Wed, 03 Sep 2025 15:45:00 -0000

internal oCIS queues have finished processing orphaned/missing file nodes and immich-side backup is done, things should work normally again

DISKS ARE SATURATED

Tue, 02 Sep 2025 18:37:00 -0000

the main raid-5 cluster of hard drives are being assaulted by small file copy operations, and internal oCIS queues are also messed up thanks to dead parent nodes looking into this, expect deathly slow uploads please use desktop clients/rclone where you can or face processing hell

IMMICH IS DOWN, MIGRATING DATA

Tue, 02 Sep 2025 16:00:22 -0000

data is currently being migrated, disks are probably saturated

EVERYTHING IS DOWN

Tue, 02 Sep 2025 16:00:00 -0000

Maintenance completed

EVERYTHING IS DOWN

Tue, 02 Sep 2025 16:00:00 -0000

Maintenance completed

EVERYTHING IS DOWN

Thu, 28 Aug 2025 03:00:00 -0000

we are fully upgrading the server because it will explode soon blah blah to serve you better yall dont have school bro but like if needed just tele us or email with (some) of our hearts, Mark Joshwel Ryan Lin

EVERYTHING IS DOWN

Thu, 28 Aug 2025 03:00:00 -0000

EVERYTHING IS DOWN

Thu, 28 Aug 2025 03:00:00 -0000

EVERYTHING IS DOWN

Thu, 28 Aug 2025 03:00:00 -0000

EVERYTHING IS DOWN

Thu, 28 Aug 2025 03:00:00 -0000

EVERYTHING IS DOWN

Thu, 28 Aug 2025 03:00:00 -0000

the server is down, and we knew

Tue, 24 Jun 2025 15:17:00 -0000

an unexpected router restart had messed with port forwarding, after a twiddle around router settings (turning port forwarding on and off), we're back up.

the server is down, and we knew

Tue, 24 Jun 2025 15:17:00 -0000

an unexpected router restart had messed with port forwarding, after a twiddle around router settings (turning port forwarding on and off), we're back up.

the server is down, and we know

Fri, 13 Jun 2025 07:48:00 -0000

it was a bootloop, server restored around 1230h SGT

the server is down, and we know

Fri, 13 Jun 2025 07:48:00 -0000

it was a bootloop, server restored around 1230h SGT

the server is down, and we know

Thu, 12 Jun 2025 22:20:00 -0000

server is down but ryan is sleeping; it could be a: - heater-related power trip (in which downtime up to six hours may be expected) - or, a router failure (in which downtime up to an hour is expected)

the server is down, and we know

Thu, 12 Jun 2025 22:20:00 -0000

the server is down, and we know

Wed, 04 Jun 2025 01:15:00 -0000

aight we up should be good

the server is down, and we know

Wed, 04 Jun 2025 01:15:00 -0000

aight we up should be good

the server is down, and we know

Wed, 04 Jun 2025 01:09:00 -0000

a power trip has caused shit to fly, and services are expected to resume around noon for any qualms contact ryan (at) scorchthinks (dot) dev

the server is down, and we know

Wed, 04 Jun 2025 01:09:00 -0000

a power trip has caused shit to fly, and services are expected to resume around noon for any qualms contact ryan (at) scorchthinks (dot) dev

la poisson du forge is down, and we know

Tue, 18 Mar 2025 12:31:00 -0000

the server restoration post-power fault did not properly start the backend before the frontend. resulting in service disruption although the status monitor remained up. the monitored url has been updated and the container stack was fully restarted in proper. no data was lost.

la poisson du forge is down, and we know

Sun, 16 Mar 2025 05:42:00 -0000

due to a server power fault, the forge's data is currently inaccessible. an investigation will be launched in the following days. any requests for data can be sent to ryan (at) scorchthinks (dot) dev