Elevated Errors

Incident Report for CallTrackingMetrics

Resolved

This incident has been resolved.

Posted Oct 10, 2016 - 13:21 EDT

Monitoring

The network appears to have stabilized and cluster health has returned to green. We continue to actively monitor our Job queue as tasks finish being processed. We expect the service to be 100% in the next 5 to 10 minutes.

Posted Oct 10, 2016 - 13:14 EDT

Identified

Actively troubleshooting an unstable network between 2 regions in AWS causing a temporary out of sync error between our search cluster and database cluster. The automated recovery process is handling the failures automatically but the result is degraded throughput of our backend servers resulting in elevated 502 errors. We are working on identifying the root cause for the multi-region connection issues within our primary VPC.

Posted Oct 10, 2016 - 12:52 EDT

Investigating

Investigating reports of elevated errors in service.

Posted Oct 10, 2016 - 12:42 EDT