It’s one thing to experience downtime, but quite another when the downtime is caused by a completely preventable human error. See how British Airways learned this lesson the hard way in its recent outage.
Estimates: Outage Costs Airline More Than Half a Billion Dollars
What’s the second worst thing that can happen to an airline? Bad PR, of course. And that’s exactly what happened to British Airways on May 27 when a contractor reportedly shut down a power supply unit in one of the airline’s data center. The result? Flights were canceled at London’s Heathrow and Gatwick airports, stranding 75,000 passengers on a busy holiday weekend.
Initial reports indicate that the power supply unit was working properly when it was accidentally shut off by a contractor who was visiting the property. BA’s inquiry will likely conclude the very expensive error to be a human - and very preventable - one.
More than 52 percent of IT data center outages are caused by human error, which means that despite the best intentions, best practices are still ignored wholesale. And it’s getting costlier every time, The BA outage shaved nearly half a billion dollars off the top of the airline’s main investor and that’s before the nearly $150,000,0000 compensation bill BA is expected to pony up to stranded flyers.
So, how do you avoid such a costly error in your data center? While far from comprehensive, this list will get you started on the path to more uptime and fewer mistakes.
Emergency Power Off buttons are often located near doorways in data centers. Too often, these buttons are not covered or marked and are mistakenly shut off during an emergency alert, which can shut down power to the entire data center. Labeling and covering Emergency Power Off buttons can go a long way toward preventing someone from accidentally pushing buttons they have no business pushing.
Label everything. To properly operate a power system, all switching devices must be labeled precisely. Once everything is labeled, double check that it’s labeled properly.
Anyone with access to the data center, including emergency, IT, security and other personnel should have basic knowledge of your equipment so that it is not shut down by mistake.
Having a sign-in policy that requires an attendant overseeing visiting vendors will help data center administrators know:
- Who is entering or exiting the area,
- Where anyone with food and drink is, and,
- How long individuals are in the facility.
Ask any truck driver: Working long shifts can cause fatigue, which leads to mistakes. Likewise, having food or drink near critical IT equipment is generally a bad idea.
It’s always a good idea to review your safety and security practices with data center personnel regularly to ensure you don’t become complacent. Otherwise, next thing you know, an outside vendor is pressing power buttons and you’re the one on the hook.
Let Instor’s team help with your data security design and installation or other whitespace project today.