Power of planning key to IT survival

Crews kept lines open despite leaky roof, pounding wind

BY MIKE WILLIAMS
Rice News staff

With the winds howling and the roof pulling loose, a crew of Information Technology staffers had its hands full keeping systems online at the Mudd Computer Science Laboratory early Saturday morning.

“It was wild and hairy,” said Barry Ribbeck, the director of systems architecture and infrastructure and one of the ride-out crew. “We needed to stay in the building, because if the generators for the phones didn’t kick off, we’d have to do it manually.”

The building’s generators were, indeed, needed to power the systems that keep Rice online. “As soon as the first power generators came on, we were there to verify them,” he said. “The generators maintained the core services – the phone and Internet. Those had to be up and running.”

It was never a given, he said. Ribbeck, Reggie Clarkson, John Thompson, Ruben Tamez and Ryan Moore were charged with keeping the systems dry, which became a challenge when wind-driven rain started finding its way into the building.

“The main thing was maintaining power and hoping the structural integrity would hold,” he said. “The building is not shelter. Like the police department, we have a wood roof, and it didn’t come off, but it certainly was loose.”

The Mudd crew was also responsible for keeping cable TV signals supplied to the campus, a particularly important link for crisis managers installed in the Founders Room at Lovett Hall who were monitoring the storm’s progress. “At some point, it did cut off because the carrier lost it,” said Ribbeck. “There was nothing we could do about that.”

Relief didn’t come until 6 a.m., when it became clear the brunt of the storm had passed.

“We didn’t lose Internet connectivity, and thankfully the campus didn’t lose power,” said Ribbeck, whose own home in Houston’s Southgate community was still without power Tuesday afternoon.

At the off-campus Primary Data Center, Shelby Sims, Ed Jazinski, Hubert Daugherty and William Deigaard held down the fort. The storm knocked out power from CenterPoint  and the two emergency generators worked for several days before they also failed.  At that point, critical services (Web and e-mail) were switched over to the Mudd facility.

“We had tested this several times prior to this event, and it paid off big time,” said Kamran Khan, vice provost for information technology. “Without such dedicated staff at Rice, we would not have such exceptional services.”

“The power infrastructure out there isn’t the best you can buy,” said Sims, the Data Center operations manager, who got about four hours sleep between Thursday and Monday night. “In the last year, during brief 5 mph gusts of wind, we’ve lost power about 30 times.

“We did have some equipment failures down there, mostly heavy equipment,” she said. “A transfer switch blew up, and that knocked out the generators one at a time, but we were able to work around some things.

She said the building itself passed the test. “In the office area, there’s a lobby that has floor-to-ceiling glass. I was out there almost all night long, and it was terrifying, but I wanted to know if I was going to have to run. It was like a high-pressure water hose was being pointed at the windows.”