Back to overview
Downtime

Skills service down

Jan 2, 2024 at 10:29am UTC
Affected services
skills

Resolved
Jan 2, 2024 at 3:27pm UTC

To prevent further occurrences we:
- have limited the number of retries
- have reduced queue depth for long running tasks
- will expand our liveliness checks to kill instances that are blocked by too long running requests or deadlocks.

Updated
Jan 2, 2024 at 12:10pm UTC

After purging queues and terminating long running tasks our skills service is up and running again.

Updated
Jan 2, 2024 at 11:30am UTC

We have identified the root cause. Due to long running requests not being terminated automatically our infrastructure was not scaling properly leading to capacity issues.

Updated
Jan 2, 2024 at 10:45am UTC

Our entire skills service is overloaded at the moment, we are investigating the cause. Please bear with us while you receive timeouts.

Created
Jan 2, 2024 at 10:29am UTC

Our skills service is currently unavailable.