ProRail: power outage was in British Telecom data center – update

The original cause of the major signal and switch failure in and around Utrecht was in a British Telecom computer center in Nieuwegein. An independent party will now investigate this, ProRail announced on Tuesday.

ProRail’s computer systems and networks for train control around Utrecht are located at British Telecom, says ProRail. The data center meets all kinds of security requirements. For example, facilities such as power supply and air conditioning are duplicated and meet the so-called Tier 3, the highest possible international standard for data centers.

Last weekend, ProRail wanted to expand the power supply for its systems and took a close look at the supply of the data center in consultation with British Telecom. The company, which is the sole user of the power supply, has taken precautionary measures in its own words. Apart from a short malfunction, this would not have caused any consequences.

Everything seemed to be going well until Monday morning ‘unexpectedly’ one of the computer systems of the Utrecht traffic control post failed. According to ProRail, it concerned a computer that was connected to the system that briefly failed in Nieuwegein last weekend. The system then switched to a backup system, which took over, but the data server couldn’t handle the load.

ProRail says that the malfunction caused defects in the hardware of both the data server and the redundant computers. The company does not know more yet. ProRail therefore says that it has commissioned an independent party to investigate the disruption in order to ‘learn from this disruption and be able to prevent it in the future’.

“The first conclusion is that the entire system, including the redundancy, received a ‘tick’ during the weekend due to the short-term outage, which most likely was caused by a disruption in the power supply. It is also clear that the ProRail systems have facilities that have the impact of power changes must occur, but it seems that this functioned insufficiently. Research has yet to confirm this,” explains ProRail. All damage has now been repaired.

Despite a substantial investment, ProRail says it can never completely prevent ICT failures. Since 2007, the company has invested more than 100 million euros in improvements to the traffic control systems. This includes upgrading computing centers and dual computer systems and networks. According to the company, this would have resulted in at least eighty percent fewer malfunctions. “These disruptions range from a broken computer screen at the post with one delayed train as a result, to a major malfunction in the systems with a lot of inconvenience for travelers.”

Update, Wednesday 12:00: ProRail emphasizes that the power outage is not British Telecom’s fault. That guilt was initially aroused in this piece.