?unique=0abe218)
Updates!
(The following updates have been posted as information is obtained from AWS and Avanan, the original article may be found in the section below)
October 22: Avanan confirms that messages that were in queue are being re-processed and released, though significantly delayed.
October 21: Avanan has confirmed that not all messages were released from processing and is working in collaboration with AWS to reprocess the queued messages and restore them to inboxes.
7:00PM EDT - Amazon is reporting that all services have been restored to normal, though some specific services are processing backlog, we believe that in the case of Avanan message delivery should catch up overnight for any undelivered messages.
5:48PM EDT - Instance launches have been restored to “pre-event” levels, meaning performance should be restored to normal. However, there will be backlog work that is processing as the services are now running normally. It is anticipated that the backlog will be processed within the next two hours.
More information is expected at approximately 6:30PM
4:11PM EDT - From AWS Support: Service recovery continues to improve across all AWS services. As a result, throttles implemented for new EC2 launches designed to mitigate impact are being reduced consistently. Errors within the Lambda service have fully recover and conditions there continue to improve. As a result, they are scaling up more quickly to restore services to pre-event levels. Another update is expected by 4:45 PM EDT.
This confirms our subjective evidence that message flow specifically for Avanan continues to improve with message delays ranging from 3-15 minutes on an an approximate average.
3:53PM EDT - While we have not received any confirmation from AWS or Avanan, we are internally seeing mail flow increase, including backlogged messages. It was also noted that some messages sent were received within a much more acceptable time frame of their original sent time (within 2-4 minutes). While completely subjective, it does provide some promise that restoration is going much better. An official update from AWS is expected at approximately 4:00 PM.
If your inbox seems calmer than normal - it's not just you.
Unfortunately, Amazon Web Services, which is a public cloud provider for many organizations is having major issues with its US-EAST-1 datacenter and our Message Security Platform provider, Avanan, is affected (among many other companies.)
In a nutshell, things aren't working smoothly. And more frustrating for all of us, it's very intermittent, so as it gets fixed and the backlog starts to clear, the whole thing seems to crash again.
Here's what we know with timestamps:
0300 - Amazon Web Services (AWS) began investigating large error rates in the US-EAST-1 region.
0351 - AWS Confirmed that there is an issue
0426 - It got worse. So badly that even creating a ticket was not possible
0501 - A potential fix was discovered and recovery/implementation was being performed
0635 - The issue had been believed to be resolved, but the large backlog of work meant that requests were being throttled, so error rates were anticipated to continue to be high.
Throttling and Delays
By this point, things looked to be trending well and by 9am, even AWS looked happy. However, by 10:00a EDT, more errors started to show up in different services.
In response, AWS believes the root cause to this issue has been identified, however, it is currently throttling traffic - creating increased delays.
Avanan, this means delays in processing emails.
Since email is unreliable at this time, stay tuned to this post for details, or learn more directly from AWS here: