TATA Stratford Power Issue - RESOLVED

Jon

Hi,

We have just lost all power in the TATA Stratford datacenter, currently we are investigating but can see that other providers in that site are also down.

As such we believe its a entire building power issue.

We are working on TATA to get as much information as possible and will update you when we have more.

Regards
Jon

chrisbyrd · OpenHosting Joined: 30 Nov 2003 Posts: 1124

This appears to be affecting a number of large providers, including ServerCity and Coreix amongst others

chrisbyrd · OpenHosting Joined: 30 Nov 2003 Posts: 1124

update - this appears to be a widespread problem affecting the larger Stratford area in East London and not just the datacentre

(Yes, there is full UPS and generator backup at the generator but this appears to have failed)

more to follow as we get it

chrisbyrd · OpenHosting Joined: 30 Nov 2003 Posts: 1124

Latest update from the datacentre operator is estimating 30-60 minutes for restoration

Jon · Posted: Thu Nov 26, 2009 7:01 pm Post subject: Staff

Hi Guys,

We have got extra staff on call ready for when the power is back.

Hopefully looking at full restore in under 1 hour.

Jon

Jon · Posted: Thu Nov 26, 2009 7:10 pm Post subject: twitter

Hi,

For more information please follow us on twitter @m247

Regards
Jon

chrisbyrd · OpenHosting Joined: 30 Nov 2003 Posts: 1124

November 26th, Power Outage at TATA Stratford Facility

As of 4:48pm grid power has been lost to the facility, the UPS systems took over as expected however the generators did not come online, a power engineer has been on site and resolved this issue, we are currently working to bring all racks and servers back online.

POWER RETURNED, RACKS BEING POWERED UP IN PHASES

The above estimate has come from the power engineer on site, please note that once power has been restored to the site we will begin a phased start up of all servers starting with core infrastructure, we expect to complete the initial power up within 30 minutes of the above times however support will be slow to react for several hours afterwards.

Incident History
4:48pm - Power to site lost - running on UPS.
5:15pm - UPS systems depleted, generators failed to start.
5:30pm - Generators failovers will not funciton, power engineer dispatched.
6:32pm - Power Engineer arrived on site.
6:54pm - Power Engineer estimates 30-45 minutes to return power to site.
7:16pm - Power returned to site, process of booting up each rack commenced

Jon · Posted: Thu Nov 26, 2009 7:48 pm Post subject: BGP is back

Hi,

We now have BGP back online!

89.238.139.1 4 33970 909265 77872620 0 0 0 00:00:16 0
89.238.139.7 4 33970 898589 76936706 0 0 161 00:00:30 0

Regards
Jon

Jon · Posted: Thu Nov 26, 2009 8:40 pm Post subject: All Back

Hi,

99% of servers are now back online, if you are still having issues please create a helpdesk ticket with as much information as possible.

http://helpdesk.openhosting.co.uk/

Regards
Jon

Jon · Posted: Thu Nov 26, 2009 9:40 pm Post subject: At Risk

Hi,

We have just been told the DC is AT RISK for the next 8 hours while the UPS re-charges.

Fingers crossed everyone.

Regards
Jon

Jon · Posted: Thu Nov 26, 2009 10:07 pm Post subject: Last few boxes

Hi,

Our on site guys are just working on the last few servers and then we will be all back online.

Thank you everyone who helped out.

Regards
Jon