Another Cluster

Ugh. So, I had a production install that I had to do on Thursday night. This is always a bit stressful as there are a lot of moving parts and steps in the process and there are a lot of people invested in this application. Thus it is a very visible project, which adds to my stress and worry when updating production. Anyway, I like to do the migration from the office because it’s a little bit easier from there and I worry about losing connections and whatever from home. It makes for a pretty long day (since the work has to happen after 5 PM) but my worry ends up being slightly less, so it’s worth the little bit of inconvenience.

Anyway, so I did all that I had to do after 5 PM as scheduled and it seemed to go relatively smoothly. Finally I was ready to start the services again by about 6 PM. I did so and then tried to get into the app online through the admin tool (to perform my very last step). Only the tool hung and I couldn’t get in. Repeated the process a couple of times with the same result. Tried to recycle the services but that didn’t work. Finally I did a hard kill and re-start and even THAT didn’t work. In a panic I noticed that one of my team members was online, so I pinged him and told him that I’m having some trouble. He said, oh no worries that he’s doing some work with the data warehouse with an engineer (Jim) on a partner team. Okay fine, but since he KNEW that I was going to be doing an installation during this time period, wouldn’t it have been nice if he had thought to contact me and say something like, “hey, I’m going to be working on some stuff, so don’t try to start services until I shoot you an email”. I mean, would that have been too much to ask?

So anyway, in the meantime the other guy (Jim) pings me and asks me how the migration went. I tell him that it was okay and that I’m just waiting on Team Member. Jim says that Team Member has been driving him crazy (they’ve been on IM and the phone for a while, I gather). Jim starts a chat with the three of us and then we’re just waiting, waiting, waiting. It’s going on 7 PM at this stage so finally I ask if anyone has an idea how long it’s going to take because I want to know whether I should run home (at this point I’m going on 11 hours in this cube). Team Member says that he doesn’t like to throw out time predictions as these things can be unpredictable (now THAT is helpful). And then he says, “If you have to go home, then GO”. Well, GEEZ, that’s easy for him to say, he's already sitting at home! (And has been all day).

So I run home, boot back up, log back in and ping both Team Member and Jim to let them know that I’m back online. Apparently nothing has happened yet so I’m pretty glad I made the decision to go home. About this time Jim starts telling me more about how Team Member is making him crazy, talking to him on speaker phone with all hell breaking loose and complete chaos going on in the background at his house so much so that Jim literally has to yell and neither can hear each other. He also says that Team Member was really weird, like, “loopy”. I ask him, “In what way? Do you mean like having cocktails or something?” and he says, yes seriously, it does seem like that, or something like that. So. Freaking. Weird.

Finally Team Member tells me that he’s done and that the services are running. I try the admin tool again and Jim the front-end, but neither of us can get in (still). I ping Team Member back, but by now he’s just completely MIA, must have just walked away without waiting to see that everything was okay. He was so cryptic and vague the whole time that this doesn’t surprise me – it makes me crazy when people don’t communicate clearly! Anyway, luckily once I do a hard kill and restart, everything seems to sort itself out, uh about three hours after I started this whole thing. What a nightmare. But at least I was able to close the loop and Jim ended up happy in the end.

Oh and the last part was that the customer told me that they wouldn’t be validating the install until after 10:30 PM that night. I had involved the off-shore team just in case but, worried, I monitored email until midnight and then woke up throughout the night to check, finally getting up around 5:30 AM to send an email out looking for status. Well. I never got the final validation from them until 3 PM or so on Friday! By then I was a wreck.

Was I ever happy to see the end to this work week. It must have been the longest short week ever.