[Thehostingnews] Data Center Web Hosting Provider, 365 Main, Releases Root Power Report
San Francisco, California - (The Hosting News) - August 7, 2007 - Data center developer and operator, 365 Main, has released the results of its investigation that detail the root cause behind a back-up power generator failure, during a PG and E power outage last week.
365 Main's San Francisco data center was impacted by a power surge caused when transformer breakers at a local PG and E power station unexpectedly opened. PG and E has still not determined what caused the breakers to open, resulting in approximately 40 percent of customers in the facility losing power to their equipment for up to 45 minutes.
365 Main is sharing the discoveries of its investigation with other Hitec customers. In addition, Hitec has expanded its preventative maintenance procedures as a direct result of discoveries made during the 365 Main investigation.
Chris Dolan, President and CEO of 365 Main commented, ''365 Main has a track record of providing customers with data centers that are considered to be among the world's finest. We extend our sincere apologies to customers who were impacted by this incident. Addressing customer concerns is our top priority. In the days since the incident occurred, we have identified and corrected the root source of the problem and are taking steps to prevent this type of problem from happening again. We are also making our comprehensive findings available to other data centers to try to prevent the same problem from recurring elsewhere.''
Typically when a power outage occurs, the outage triggers 365 Main's rigorously maintained and tested back-up diesel generators to start-up and take over providing power supply to customers. 365 Main's San Francisco facility has ten 2.1 megawatt back-up generators to be used in the event of a loss of utility power. Eight primary generators can successfully power the building, with two generators available on stand-by in case there are any failures with the primary eight.
However, following the power outage last week, three of 365 Main's 10 back-up power generators, manufactured by Hitec, failed to complete their start sequence. A complete investigation of the incident began immediately.
Within hours of the incident, an international team of specialists was deployed to 365 Main's San Francisco data center facility to join on-site technicians and begin systematically testing the generators in search of a root cause. After days of thorough testing around the clock, the team discovered a weakness in an essential component of the back-up generator system known as a DDEC (Detroit Diesel Electronic Controller).
The team discovered a setting in the DDEC that was not allowing the component to correctly reset its memory. Erroneous data left in the DDEC's memory subsequently caused misfiring or engine start failures when the generators were called on to start during the power outage on July 24.
The investigation team discovered DDEC issues on each of the failed Hitec units and were able to successfully simulate failure. A fix was introduced by altering the timing of a command to the DDEC component, allowing more time between the engine shut-down command and the DDEC reset command. Once this fix was introduced, the Hitec generators successfully passed more than 50 consecutive start-up sequence tests without incident.
The testing methodology was performed by Hitec specialists along with 365 Main's chief technician and staff. Specialists from Cupertino Electric were present during all testing, and EYP Mission Critical Facilities will provide independent verification of the findings the week of 8/6/07.
365 Main has implemented the DDEC fix in its San Francisco and El Segundo facilities. Of the five data centers in 365 Main's portfolio, the San Francisco and El Segundo facilities are the only ones with Hitec generators containing DDECs. All other facilities feature other brands of generators or have different models of Hitecs.
In the wake of the outage, 365 Main published an apology to customers and daily updates directly from the investigation team meeting minutes, allowing customers and the public at large to track progress. A complete archive of these updates and more details about today's update are available at: www.365main.com/status_update.html.
Glenn Ellis, president and CEO of Hitec USA, also commented: Our top priority is taking steps to prevent this type of unforeseen incident from occurring again. We sincerely apologize to 365 Main and its customers that our generators failed to deliver the continuous power as designed.
Since its inception over five years ago, 365 Main has delivered 99.9967 percent power uptime to customers across its five-data-center portfolio. This includes the outage experienced in San Francisco last week. 365 Main's San Francisco facility has delivered 99.9942 percent uptime to customers during the last five years, inclusive of last week's outage.
As part of their service level agreements with 365 Main, 365 Main customers receive rent abatements (refunds) in the event that electrical power is dropped in the section(s) of the data center where their servers are located. 365 Main is honoring all service level agreements with affected customers.
Recently, 365 Main participated in The Green Grid's global nonprofit consortium of energy ecosystem technology, joining other companies and professionals, dedicated to advancing energy efficiency. Formed earlier this year, The Green Grid is committed to driving new user-centric metrics, technology standards and best practices for use by data center managers worldwide. Membership is open to any company developing products and technologies aimed at the data center market, as well as to information technology professionals tasked with data center operations. Member companies currently sitting on the organization's board of directors include HP, IBM, Intel, Microsoft, AMD, APC, Dell, Rackable Systems, SprayCool, Sun Microsystems and VMware.
The Green Grid's key areas of focus include:
Defining meaningful, user-centric models and metrics
Developing standards, measurement methods, processes and new technologies to improve performance against the defined metrics
Promoting the adoption of energy-efficient standards, processes, measurements and technologies.
365 Main Inc. develops and operates data centers designed, managed, developed and certified according to industry standards and compliance requirements, including SAS70, that feature 24/7/365 power, cooling, security and connectivity and capabilities to ensure mission-critical operations and business continuity for its tenants.
365 Main began with a single data center in San Francisco in 2003 and earlier this year expanded to Los Angeles, Phoenix and Northern Virginia. Customers include Charles Schwab, Sun Microsystems and Esurance. The company now has nearly one million square feet under management nationwide. Flexible cost structures are available for SCALE 365 customers. The company is privately held and based in San Francisco, Calif. and also offers core hosting services to large enterprises such as Charles Schwab, the Oakland Raiders and Sun Microsystems.
For more information about 365 Main Inc., please visit: www.365main.com.
http://www.thehostingnews.com/news-data-center-web-hosting-provider-365-main-releases-root-power-report-3531.html
San Francisco, California - (The Hosting News) - August 7, 2007 - Data center developer and operator, 365 Main, has released the results of its investigation that detail the root cause behind a back-up power generator failure, during a PG and E power outage last week.
365 Main's San Francisco data center was impacted by a power surge caused when transformer breakers at a local PG and E power station unexpectedly opened. PG and E has still not determined what caused the breakers to open, resulting in approximately 40 percent of customers in the facility losing power to their equipment for up to 45 minutes.
365 Main is sharing the discoveries of its investigation with other Hitec customers. In addition, Hitec has expanded its preventative maintenance procedures as a direct result of discoveries made during the 365 Main investigation.
Chris Dolan, President and CEO of 365 Main commented, ''365 Main has a track record of providing customers with data centers that are considered to be among the world's finest. We extend our sincere apologies to customers who were impacted by this incident. Addressing customer concerns is our top priority. In the days since the incident occurred, we have identified and corrected the root source of the problem and are taking steps to prevent this type of problem from happening again. We are also making our comprehensive findings available to other data centers to try to prevent the same problem from recurring elsewhere.''
Typically when a power outage occurs, the outage triggers 365 Main's rigorously maintained and tested back-up diesel generators to start-up and take over providing power supply to customers. 365 Main's San Francisco facility has ten 2.1 megawatt back-up generators to be used in the event of a loss of utility power. Eight primary generators can successfully power the building, with two generators available on stand-by in case there are any failures with the primary eight.
However, following the power outage last week, three of 365 Main's 10 back-up power generators, manufactured by Hitec, failed to complete their start sequence. A complete investigation of the incident began immediately.
Within hours of the incident, an international team of specialists was deployed to 365 Main's San Francisco data center facility to join on-site technicians and begin systematically testing the generators in search of a root cause. After days of thorough testing around the clock, the team discovered a weakness in an essential component of the back-up generator system known as a DDEC (Detroit Diesel Electronic Controller).
The team discovered a setting in the DDEC that was not allowing the component to correctly reset its memory. Erroneous data left in the DDEC's memory subsequently caused misfiring or engine start failures when the generators were called on to start during the power outage on July 24.
The investigation team discovered DDEC issues on each of the failed Hitec units and were able to successfully simulate failure. A fix was introduced by altering the timing of a command to the DDEC component, allowing more time between the engine shut-down command and the DDEC reset command. Once this fix was introduced, the Hitec generators successfully passed more than 50 consecutive start-up sequence tests without incident.
The testing methodology was performed by Hitec specialists along with 365 Main's chief technician and staff. Specialists from Cupertino Electric were present during all testing, and EYP Mission Critical Facilities will provide independent verification of the findings the week of 8/6/07.
365 Main has implemented the DDEC fix in its San Francisco and El Segundo facilities. Of the five data centers in 365 Main's portfolio, the San Francisco and El Segundo facilities are the only ones with Hitec generators containing DDECs. All other facilities feature other brands of generators or have different models of Hitecs.
In the wake of the outage, 365 Main published an apology to customers and daily updates directly from the investigation team meeting minutes, allowing customers and the public at large to track progress. A complete archive of these updates and more details about today's update are available at: www.365main.com/status_update.html.
Glenn Ellis, president and CEO of Hitec USA, also commented: Our top priority is taking steps to prevent this type of unforeseen incident from occurring again. We sincerely apologize to 365 Main and its customers that our generators failed to deliver the continuous power as designed.
Since its inception over five years ago, 365 Main has delivered 99.9967 percent power uptime to customers across its five-data-center portfolio. This includes the outage experienced in San Francisco last week. 365 Main's San Francisco facility has delivered 99.9942 percent uptime to customers during the last five years, inclusive of last week's outage.
As part of their service level agreements with 365 Main, 365 Main customers receive rent abatements (refunds) in the event that electrical power is dropped in the section(s) of the data center where their servers are located. 365 Main is honoring all service level agreements with affected customers.
Recently, 365 Main participated in The Green Grid's global nonprofit consortium of energy ecosystem technology, joining other companies and professionals, dedicated to advancing energy efficiency. Formed earlier this year, The Green Grid is committed to driving new user-centric metrics, technology standards and best practices for use by data center managers worldwide. Membership is open to any company developing products and technologies aimed at the data center market, as well as to information technology professionals tasked with data center operations. Member companies currently sitting on the organization's board of directors include HP, IBM, Intel, Microsoft, AMD, APC, Dell, Rackable Systems, SprayCool, Sun Microsystems and VMware.
The Green Grid's key areas of focus include:
Defining meaningful, user-centric models and metrics
Developing standards, measurement methods, processes and new technologies to improve performance against the defined metrics
Promoting the adoption of energy-efficient standards, processes, measurements and technologies.
365 Main Inc. develops and operates data centers designed, managed, developed and certified according to industry standards and compliance requirements, including SAS70, that feature 24/7/365 power, cooling, security and connectivity and capabilities to ensure mission-critical operations and business continuity for its tenants.
365 Main began with a single data center in San Francisco in 2003 and earlier this year expanded to Los Angeles, Phoenix and Northern Virginia. Customers include Charles Schwab, Sun Microsystems and Esurance. The company now has nearly one million square feet under management nationwide. Flexible cost structures are available for SCALE 365 customers. The company is privately held and based in San Francisco, Calif. and also offers core hosting services to large enterprises such as Charles Schwab, the Oakland Raiders and Sun Microsystems.
For more information about 365 Main Inc., please visit: www.365main.com.
http://www.thehostingnews.com/news-data-center-web-hosting-provider-365-main-releases-root-power-report-3531.html
