Customers of Virgin Media experienced a loss of broadband service nationwide last night, with the problem continuing through into this morning for some, according to reports.
This reporter's cable modem lost its link at around 9pm, for about 20 minutes, while others reported similar symptoms at the same time in places such as Glasgow, Belfast and the South, confirming the scale of the problem.
There have also been reports of outages earlier in the day, although it appears as though the problems grew more widespread into Monday evening. Both cable and ADSL users appear to have been affected.
Some users report being affected for longer than others. Virgin Media's broadband status page acknowledges the fault as a "Critical Service Outage" - the highest fault level, which comes with a target resolution time of 4 hours. The fault ticket was published at 10:09pm Monday 17th and has been updated several times since then, but remains an open ticket some 12 hours after problems first started occurring.
Customers affected by the out but now reconnected are reporting varying speeds - some stating that everything is back to normal whilst others are complaining of very slow rates.
The national outage comes at a time when the company - which has come under fire for introducing a confusing premium rate broadband technical support line and poorly delivered 10->20Mbit speed upgrades - is piloting a 50Mbps service. Some 20Mbps "XL" customers have yet to see a download run at full speed (or close enough to full speed to be satisfied).
HEXUS has attempted to contact Virgin Media for a statement regarding the outage, but at the time of publication it is still outside of office hours and we've had no response.
We will, of course, keep an eye on the status page and eagerly await a response from Virgin Media. Check back here for more info as we get it.
Update: 18/12/07 10:48
Virgin Media's Mark Crombleholme has posted on the support ticket for this fault that the engineers have now resolved the issue. Do let us know if your service is back to normal (that is, if you are able).
We've still not received an official response from Virgin Media on the problem, what caused it, and why so many people have had broadband trouble for up to 12 hours and beyond.
Update: 13:43
A spokesperson for Virgin Media contacted us regarding the issue, apologising for the problems.
We were told that the problem related to a routine maintenance process involving the company's routers. This process failed - how so is under investigation - leading to a mass IP lease expiration and a lot of requests for new leases.
The DHCP servers couldn't respond to the sheer volume of requests, leaving modems without valid IPs for varying lengths of time.
A small number of routers affected by the maintenance issue are still not back online. Virgin Media engineers are working on fixing them. The company advised customers still experiencing problems to power cycle their modems/STBs.
According to the spokesperson, not everyone lost their IP, but the number of people that did lead to saturation of the support lines. Automated messages were put in place, but there weren't enough available lines for people to actually connect to hear the message, hence the confusion.
Virgin Media is still investigating the precise cause of the failure, in an effort to ensure it won't happen again. When we asked about compensation, we were told that in the general case, the outage wasn't extended enough for it to be considered under the terms and conditions of VM's residential broadband services.
