Oh Crap! What Was That Beeping?

About 2:30 this afternoon I was working in the server room in the San Francisco office.  I was hooking up a laptop to the UPS (uninterruptable power source - backup battery system) at the request of one of the engineers in the DC office.  I had just hit the send button on an e-mail to him to tell him I had completed the setup when I heard it:  beeeep...beeeeep....beeeeeeeep.....

I said it out loud:  Oh crap!  What was that beeping?  I looked at the servers that are mounted in the racks and they all looked okay - I could see all the lights and displays were still on.  That's when it got crazy.

My blackberry started buzzing.  The first 2 messages were voice mails - one from my boss and one from my counterpart in Los Angeles.  The next message was an e-mail from another engineer in DC - the system was reporting a power outage.  A power outage?  Seriously?  I'm standing there in a room of fully powered equipment!  Surely that must be a mistake.

I called Smitty, the engineer in DC who sent the e-mail.  I explained to him that I was seeing power on everything; then we started talking about the work I was doing and when I looked down, I saw it:  the UPS unit was completely dark.

I killed one of the UPS units!  My heart started racing as Smitty got another couple of the engineers on the line and they started questioning me about the condtions in the room and what was going on just before I heard the beeping.  In the meantime, by blackberry keeps buzzing with more and more e-mails coming in; the receptionist came to the server room door to say that everyone was calling her looking for me because they're having computer problems (and what, you think I'm having a party in here?); then the office manager came in excitedly exclaiming "computers are crashing everywhere!"

OMG!  What have I done?  I tried to remain calm and after about 10 minutes of consultation with the DC guys, we were able to revive the UPS.  I did a little happy dance!  When the DC guys confirmed that everything had come back online, I did a big happy dance.

And then it hit me:  I took the system down.  I TOOK the system down.  I TOOK THE FREAKING SYSTEM DOWN!!!  I thought I was having a heart attack.  By the time I got to my office, I couldn't breathe, my heart was racing and all I could do was cry. 

My boss called to talk about it.  I thought for sure I would get fired.  But, he was totally cool and said that "shit happens" and not to worry so much.  As long as we were back up and running and nobody lost any work, it was fine.

So, it only took me three months to achieve my first big whoopsie.  I'm glad today is Friday and I can go home and hibernate for the weekend.

Until Next Time,

Kel