Skills service down
Resolved
Jan 2, 2024 at 3:27pm UTC
To prevent further occurrences we:
- have limited the number of retries
- have reduced queue depth for long running tasks
- will expand our liveliness checks to kill instances that are blocked by too long running requests or deadlocks.
Affected services
Updated
Jan 2, 2024 at 12:10pm UTC
After purging queues and terminating long running tasks our skills service is up and running again.
Affected services
Updated
Jan 2, 2024 at 11:30am UTC
We have identified the root cause. Due to long running requests not being terminated automatically our infrastructure was not scaling properly leading to capacity issues.
Affected services
Updated
Jan 2, 2024 at 10:45am UTC
Our entire skills service is overloaded at the moment, we are investigating the cause. Please bear with us while you receive timeouts.
Affected services
Created
Jan 2, 2024 at 10:29am UTC
Our skills service is currently unavailable.
Affected services