Post Incident Report - Sydney DC Power Failure
The following is the PIR regarding the loss of power at our Sydney data centre (owned and operated by PIPE Networks in Cromer, NSW) on June 6, 2012.
At approximately 8:09PM, one of the UPS units suffered an electrical fault which caused damage to one of the three UPS units in the “A” power feed group. This caused the entire “A” side UPS group to drop load and ultimately power to all services using the “A” feed was lost.
On-site generators, which are fully tested on regular intervals throughout the year, were automatically activated and took the “A” power load as planned. However, since the “A” side UPS group had gone offline, power was unable to be supplied from the generator which resulted in an extended power degradation.
Within minutes both our own technical staff and our vendor’s Facilities Management team, as well as their electrical contractor, were all on-site to assess the situation.
At approximately 8:45PM, the faulty UPS group was placed in to manual bypass which allowed the power feed to flow directly from mains power through to equipment in the data centre. While this restored power to our equipment, it was without power protection while the fault was investigated.
Electrical contractors continued working to restore the two remaining operational UPS units to full service, before seamlessly restoring UPS protection to the circuit just after 11:45PM.
Whilst only two of the three UPS units are currently online, we have been assured that the load on the “A” side is low enough to continue to provide N+1 power protection using the current configuration, allowing the vendor to isolate the defective unit and complete repair works before bringing it back online.
It is important to note that prior to moving in to this facility, our Management team conducted thorough due diligence and were satisfied that the facility was of high quality. As many people have been able to deduce after this power degradation, this facility is also home to several large national Internet Service Providers and content providers who would have all conducted the same level of due diligence before moving in.
Does the data centre have two separate power feeds?
Yes. It’s important to note that during this incident the power supply to the “B” feed remained uninterrupted and with full UPS protection. However, our equipment only takes power from the “A” feed as the majority of our equipment does not have dual power supplies. If we were to have equipment that had dual power supplies, it would cost us a lot more to deploy per device, and our data centre costs would double. All of these costs would then need to be passed on to our customers, which we know they would not be willing to pay.
All of our core networking infrastructure and our Dell Enterprise Cloud hardware do have dual power supplies. It is not uncommon for companies, big or small, to have devices that only use one power source. This is easily proven by the calibre of the casualties that were affected during this incident.
Will this ever happen again?
It would be foolish to say that this incident, or one like it, will never happen again. We have a lot of trust and confidence in our vendor to investigate this incident, conduct the repair works, and do everything possible to ensure that it does not happen again. Our vendor has a rich history of providing quality data centre services to thousands of customers who house their equipment in their facilities, and at this stage we see no reason to doubt that pedigree.
VentraIP Australia does everything else, why not build your own data centre?
On several occasions over the past two years, we have discussed internally whether or not we could build a business case for our own small-scale data centre, which could act as an off-site facility to many companies who need backup equipment outside of the CBD where the revenue could be used to negate our retail cost base. While we have always believed that the business case to do it is sound, finding the right location proved very difficult when taking into account power requirements, fibre runs and council zoning restrictions (among the many other considerations).
In the end we decided that we should focus more on what we do best, hosting and domains, and continue to partner with companies who are able to deliver these services as part of their own core focus.
On behalf of the entire VentraIP team, we would like to sincerely apologise to customers who were adversely affected by this incident. Your custom and support are greatly appreciated.