This page contains all current information on the status of this incident.
| Incident Details | |
|---|---|
| Incident Status | closed |
| Services Impacted |
Network Connectivity
Office 365 mail (bristol-schools.uk) Web Filtering |
| Start Time | 2017-02-17 10:00 GMT |
| End Time | 2017-02-23 14:00 GMT |
| Description | slow internet being reported by various sites. Issue is now with our Networks team for investigation. |
| Incident Updates | |
|---|---|
| Time | Detail |
| 2017-02-17 13:55:06 GMT | Possible routing issue. Evidence currently being gathered to supply to our ISP for further investigation. |
| 2017-02-20 08:14:36 GMT | Virgin Media are still investigating this routing issue. We will update this page when we have more information. |
| 2017-02-20 12:24:14 GMT | At this time Virgin Media are still investigating the issue. As and when we have further information we will update this page. |
| 2017-02-20 15:14:25 GMT | Apologies for the on-going difficulties accessing certain websites. Virgin Media are still trying to diagnose exactly where the issue lies as the problem is quite complex and may be due to an error by an Internet Provider elsewhere which is difficult to pin down. |
| 2017-02-21 08:41:13 GMT | The issue is still ongoing this morning. Unfortunately we still have no ETA of a fix. Virgin Media have identified that some of our internet bound traffic stops at an ISP in Sweden. They are currently investigating why this is. |
| 2017-02-21 13:57:19 GMT | Our Internet providers have determined that a content delivery network used by many larger websites is currently blocking network traffic from schools. They are now making contact with the network (Akamai Techniologies) to try to resolve this and to identify the reasons behind this. We note that BBC and Office 365 are examples of websites that have part of their site hosted by Akamai. |
| 2017-02-22 09:00:33 GMT | Improvement seen to BBC plus some Office 365 access restored. Microsoft have also been engaged to help resolve issues with Akamai Technologies. |
| 2017-02-22 09:42:34 GMT | Overnight we have seen some improvements, however there are still issues with Office 365. We are currently working directly with Microsoft to resolve this issue. We should have a further update shortly. |
| 2017-02-22 11:57:59 GMT | Microsoft are now contacting Akamai to remove the blacklisting. We have also worked with Microsoft to create a trust relationship between our mail servers and theirs in an attempt to resolve the issue with undelivered mail. We will be speaking to Microsoft within the hour for an update. |
| 2017-02-22 14:17:17 GMT | Microsoft are still investigating this with Akamai. We are regularly chasing Microsoft for updates. Unfortunately we have no further update at the moment. |
| 2017-02-22 20:43:34 GMT | As part of our continued investigations over the last few days working with Virgin Media and JANET, we initially discovered that Akamai were blocking requests from JANET’s Swedish ISP (This is how JANET connect to the internet). Various attempts to contact Akamai were unsuccessful. However, with the troubleshooting that followed Virgin identified a routing misconfiguration, which both Virgin Media and JANET agreed should be rectified. This was changed on Monday afternoon while the Akamai issue still lay unresolved (the hope was that it would fix it). As of Tuesday afternoon we could see the Akamai symptoms were seemingly improving due to this change, however issues were still being experienced hence the reason we started working with Microsoft to resolve the specific issue with Office 365. (See next update for further details...) |
| 2017-02-22 20:44:41 GMT | (From previous update) Further troubleshooting has taken place today by our Networks team. They have identified issues created by Mondays change and rolled back Monday’s routing change as a temporary work-around. This has resolved various mail relay issues , inbound access problems and brought some improvement to Office 365 access. The cause of these specific problems is now known and the work-around will remain in place until they are resolved. Please note that if you cannot access Office 365 via the web interface you may have more success using the Outlook client. Please contact us if you would like further details on this. Further evidence is now being supplied to Microsoft so they can continue their investigations with Akamai. |
| 2017-02-23 11:00:22 GMT | Virgin Media and Corporate IT will be implementing a potential fix at 12.30pm. Please be aware that there may be further disruption at this time. We will be monitoring closely and rolling back within 5 minutes if this does not resolve ths issue. |
| 2017-02-23 14:04:33 GMT | A workaround has now been implemented which means internet access is now vastly improved. Please be aware that this workaround involves sending requests for Akamai sites temporarily via the Corporate Council internet feed. As a result we will be closely monitoring traffic on this connection. Following the tests at 12.30pm, Virgin and Corporate IT will be doing further work at 2.40pm which may disrupt internet access for 5 minutes or so. |
| 2017-02-23 17:03:47 GMT | We can confirm that testing at 2.40pm was successful. Routing changes have been successfully implemented, however traffic to Akamai is still routed through the council internet feed and is being monitored. Internet access for schools has effectively been fully restored. Further work will be undertaken behind the scenes to fully resolve this issue. We would like to apologise for the delay in restoring access - further details on this outage will be forthcoming shortly from Trading with Schools. |