I was terribly proud of what we were doing and had done. A lot of the problems of the past had been solved. Things didn't take so long. Monitoring was easy. One of the simplest things I did was to purchase and set up monitoring workstations, one with large dual monitors for Syslogs, and firewall logs, IP monitoring, everything. I put it front and center. Everyone checked all the time and we knew when things were going wrong. The quad monitor turned into another monitoring station as well as something for programming. Everything got used.
The server room expanded from one rack to two, from two racks to six. I redid the power, the cooling, the networking.
But there was a plateau to this growth that I was going to meet. Two in fact:
Our Datacenter
Most departments hadn't the foresight when they introduced servers and client systems to put their equipment in dedicated places. Any old room would do, better still if it is crowded and misshapen and unsuitable to any other academic or research task. Closets would be fine. Our network equipment escaped some problems going into dedicated telecom room though I did need to upgrade the cooling or the big C6500s and add high amp power to accommodate the powerful UPSs.
On the other hand, our server room was an old storage room. Adding a dedicated HVAC condenser with its own thermostat didn't solve our mounting heat problems -- there wasn't enough room for something powerful in the ceiling. Adding a backup (large) portable chilled water AC still didn't do it. Much of the problem was the way the air flowed due to the room layout. They couldn't move the vents and the servers couldn't be flipped back to front. So cold air pushed down from the cooling vents onto the hot server exhaust migrating up. The return vents weren't anywhere near the sources to pull in the hot air. The portable AC could only sit in one spot where there was room, so it generally blew somewhat at the sides of the servers, but probably only created more pressure to keep the warm air from circulating. We had warm and warmer pockets in the room and couldn't get cold air to the server intakes.
So the temperatures in the server room continued to rise as more devices were added. Leaving the doors open in a public school was a bad idea. Virtualization would help, but not entirely.
We discussed this with the COO. We needed a room we could properly outfit as a server room. We would design it from the beginning, rather than jerry-rig solutions. Picking a room on-site was problematic since all other rooms big enough for the task were classrooms that were in use.
But I did detailed site surveys, analyzed every room, analyzed what could go in, approximating costs/benefits. No stone unturned, every stone turned over ad nauseum. The price tag would be approximately $60k. One of the best works I'd written, a solid 30 pages, heavily diagrammed, plus appendices.
But my documentation didn't work with this new Dean, so after months of my efforts and my team's, we weren't going to get a new server room. My senior colleague opened the doors.
It was my first big letdown.
I laid out a short book of plans, considerations, how we could provide a centralized datacenter model while keeping some management distributed thereby appeasing various schools. How we could provide varying service models that, when done correctly, would likely incentivize trust and collaboration without alienating the engineers. My idea would have been to produce a model that would eventually pool the engineers from a lot of different schools together in a manner that could pave the way for an actual consolidation of engineer roles (a centralized cohesive unit serving the schools together) without the sense of one school simply swallowing everything else in. Even if the Medical School remained inclined to swallow everyone, this would make it easier rather than harder for them since now people depended on the collaboration working. The key to getting the infrastructure in one place was getting getting the engineers in one place to perform and develop working trust.
I don't know if I was over-simplifying or over-complicating the reality of the situation, or if I'd in fact captured the concerns of each department correctly with solid mitigation strategies. I wondered if all these hopes and plans were in my head. Delusions of grandeur for all of the effort I put in.
My remote hope was that if I could think through enough and get this to Medicine that I might have some part in at least putting it together. I wanted to do something bigger.
I was also beginning to wonder if my next career move wouldn't in fact be to a larger enterprise like Medicine so I tried my best to stand out and be useful.
But nothing ever came of it during my tenure. As far as I know, it was all talk. The CIO of the School of Medicine, who I knew and liked, took my emails, maybe some comments, but little else.
Admittedly, there was slight desperation on my part. The writing was on the wall. There was less and less for me to do. Smashing success had come from glaring problems that created the collective will I needed to back up my efforts. Some problems were outstanding for IT, but no longer apparent to the rest of the school.
The new Dean hadn't experienced the pains of failing IT so we were not much on his radar. It was a Nursing School after all, with all attention focused on a top 10 academic and research program.
I applied to a number of departments. When Engineering needed a high-level engineer to re-architect their unbelievably old, mixed vendor, 300-device switched, flat vlan topology network, running off of one aging C6500 with Sup II (a cold dead spare chassis sat nearby), I jumped at it. They experienced constant and sometimes catastrophic outages and didnt even know where all the equipment lay. This seemed exactly like the sort of chaos I could turn around. I went into the interview. I was perhaps overly contemplative about the problem, or overly eager. I wanted to impress. I didn't get the job. It was a network heavy job. They gave it to a Linux girl out of grad school.
When Engineering needed a lower-level engineer to (again) manage and upgrade the same network, I applied again, and interviewed again, perhaps more muted. I didn't get it. They gave it to a helpdesk guy with no Cisco experience who was freshly graduated.
Just in the mere hopes of being a part of their upgrade I mentored him for the next year. He was a very nice guy. He knew nothing about networking when he started.
Any other positions went nowhere.
I applied outside of UCLA, finally. And in this job market I got back responses from companies offering less money, less challenging work, less authority, and in most cases less of all of the above. Some of my less pleasant interview stories come from this period too. Interviews are unsurprisingly better when companies are competing for engineers.
Once, a year before my departure, I got a solid offer with a consulting company. But I wasn't ready to leave. I had more I planned to do. I felt already effective. When I showed my boss the offer, now he had the justification to request a raise that got me nearly up to that point. If that hadn't happened I might have gone.
(When I did leave, my replacement was a proper Manager, title, pay and all. He was apparently a bit of a campus apparatchik, attending meetings constantly, engineering minimally if at all. He got so little done and was so difficult to get rid of that they simply appointed a director above him under the COO. A good friend in that department told me they missed me. Things got done while I was there. Then nothing got done.)
Months before my departure I had another offer, this time the money was terrific. It would make a huge difference. Ok, I wasn't thrilled about the particular online retail heavy business, but it was a proper datacenter, lots of web hosting, and they needed a lead network engineer. The manager was very excited, his boss ok'd everything, and we waited a short time for the next budget cycle to flip. When it did, their numbers didn't look as good as they were previously. My position was put on indefinite hold. That was it. (In retrospect, knowing how shady the company has turned out, probably a good thing I didn't join even if the IT work would have been complex).
I was flying high my first years in the spot. It was all my vision. I accomplished a lot. And now... I was having a hard time getting call backs for positions I might have done five years ago.
My predecessor had, for the last year, been moping more around the school, grumbling under his breath in a way that brought Eeyore to mind. Now I hung my head like him too. This despondency wasn't so strange. His departure fostered the crisis i thrived in, but the crisis was over. There was nothing to look forward to. It could be maintained but there was nothing left to build.
I thought up a number of new, smaller projects. Things I could do to improve things in a smaller way. But it wasn't boredom. It was longing, missing the feelings of being part of something big, of plugging holes in a ship so it could get to its destination. And doing something that affected many people in profound and often obvious ways.
Even little things. It took up to 2 minutes to load Windows. And I got that down to a fraction of the time. Users had the sense of IT moving quickly, rare in academia, even if little else did. We responded quickly and provided solutions. We could all do things we couldn't before.
I missed being the person who brought that. It was time to go and I didn't know how that would happen. The least worst trouble was that I was grossly underpaid compared to every one of my peers on campus in bigger and smaller departments.
I couldn't even see my next step.
So the temperatures in the server room continued to rise as more devices were added. Leaving the doors open in a public school was a bad idea. Virtualization would help, but not entirely.
We discussed this with the COO. We needed a room we could properly outfit as a server room. We would design it from the beginning, rather than jerry-rig solutions. Picking a room on-site was problematic since all other rooms big enough for the task were classrooms that were in use.
But I did detailed site surveys, analyzed every room, analyzed what could go in, approximating costs/benefits. No stone unturned, every stone turned over ad nauseum. The price tag would be approximately $60k. One of the best works I'd written, a solid 30 pages, heavily diagrammed, plus appendices.
But my documentation didn't work with this new Dean, so after months of my efforts and my team's, we weren't going to get a new server room. My senior colleague opened the doors.
It was my first big letdown.
All of Ours Datacenter
I was only temporarily deterred. There was another option. There was a project being talked about of an intramural datacenter consolidation between School of Medicine, Dentistry, Nursing, Public Health and other schools and research centers. It was regional and Medical Sciences specific. Medicine, as the biggest player, was running the show. It was in the very initial stages of consideration. I met with the IT heads of the other schools, talking to them about their problems, needs and concerns, running through ideas of how this could work. There was enormous suspicion of a consolidated effort. So I was trying to think through the governance of the project in addition to the technical details.I laid out a short book of plans, considerations, how we could provide a centralized datacenter model while keeping some management distributed thereby appeasing various schools. How we could provide varying service models that, when done correctly, would likely incentivize trust and collaboration without alienating the engineers. My idea would have been to produce a model that would eventually pool the engineers from a lot of different schools together in a manner that could pave the way for an actual consolidation of engineer roles (a centralized cohesive unit serving the schools together) without the sense of one school simply swallowing everything else in. Even if the Medical School remained inclined to swallow everyone, this would make it easier rather than harder for them since now people depended on the collaboration working. The key to getting the infrastructure in one place was getting getting the engineers in one place to perform and develop working trust.
I don't know if I was over-simplifying or over-complicating the reality of the situation, or if I'd in fact captured the concerns of each department correctly with solid mitigation strategies. I wondered if all these hopes and plans were in my head. Delusions of grandeur for all of the effort I put in.
My remote hope was that if I could think through enough and get this to Medicine that I might have some part in at least putting it together. I wanted to do something bigger.
I was also beginning to wonder if my next career move wouldn't in fact be to a larger enterprise like Medicine so I tried my best to stand out and be useful.
But nothing ever came of it during my tenure. As far as I know, it was all talk. The CIO of the School of Medicine, who I knew and liked, took my emails, maybe some comments, but little else.
Admittedly, there was slight desperation on my part. The writing was on the wall. There was less and less for me to do. Smashing success had come from glaring problems that created the collective will I needed to back up my efforts. Some problems were outstanding for IT, but no longer apparent to the rest of the school.
The new Dean hadn't experienced the pains of failing IT so we were not much on his radar. It was a Nursing School after all, with all attention focused on a top 10 academic and research program.
Jumping Ship
As the more successful projects were concluding I began applying to various campus entities. This was the most difficult part for me. I was on campus, I could easily meet with anyone, I was even known by a lot of people now. I was a Programmer Analyst II. My peers were P/A IIIs, some IVs and a number of Computing Resource Managers (the ones officially allowed to manage). My promotion, initially with the blessing of the COO, stalled in the paperwork. At first it was approved for P/A III signed and submitted. Then the COO let me know the Dean had stopped the request and really wanted my slot to be a manager now and I needed to rework this as a CRM. I was elated and I reworked the papers and sent it up to the COO who sent it on. I don't know what happened to it. I don't think it was cancelled.I applied to a number of departments. When Engineering needed a high-level engineer to re-architect their unbelievably old, mixed vendor, 300-device switched, flat vlan topology network, running off of one aging C6500 with Sup II (a cold dead spare chassis sat nearby), I jumped at it. They experienced constant and sometimes catastrophic outages and didnt even know where all the equipment lay. This seemed exactly like the sort of chaos I could turn around. I went into the interview. I was perhaps overly contemplative about the problem, or overly eager. I wanted to impress. I didn't get the job. It was a network heavy job. They gave it to a Linux girl out of grad school.
When Engineering needed a lower-level engineer to (again) manage and upgrade the same network, I applied again, and interviewed again, perhaps more muted. I didn't get it. They gave it to a helpdesk guy with no Cisco experience who was freshly graduated.
Just in the mere hopes of being a part of their upgrade I mentored him for the next year. He was a very nice guy. He knew nothing about networking when he started.
Any other positions went nowhere.
I applied outside of UCLA, finally. And in this job market I got back responses from companies offering less money, less challenging work, less authority, and in most cases less of all of the above. Some of my less pleasant interview stories come from this period too. Interviews are unsurprisingly better when companies are competing for engineers.
Once, a year before my departure, I got a solid offer with a consulting company. But I wasn't ready to leave. I had more I planned to do. I felt already effective. When I showed my boss the offer, now he had the justification to request a raise that got me nearly up to that point. If that hadn't happened I might have gone.
(When I did leave, my replacement was a proper Manager, title, pay and all. He was apparently a bit of a campus apparatchik, attending meetings constantly, engineering minimally if at all. He got so little done and was so difficult to get rid of that they simply appointed a director above him under the COO. A good friend in that department told me they missed me. Things got done while I was there. Then nothing got done.)
Months before my departure I had another offer, this time the money was terrific. It would make a huge difference. Ok, I wasn't thrilled about the particular online retail heavy business, but it was a proper datacenter, lots of web hosting, and they needed a lead network engineer. The manager was very excited, his boss ok'd everything, and we waited a short time for the next budget cycle to flip. When it did, their numbers didn't look as good as they were previously. My position was put on indefinite hold. That was it. (In retrospect, knowing how shady the company has turned out, probably a good thing I didn't join even if the IT work would have been complex).
I was flying high my first years in the spot. It was all my vision. I accomplished a lot. And now... I was having a hard time getting call backs for positions I might have done five years ago.
My predecessor had, for the last year, been moping more around the school, grumbling under his breath in a way that brought Eeyore to mind. Now I hung my head like him too. This despondency wasn't so strange. His departure fostered the crisis i thrived in, but the crisis was over. There was nothing to look forward to. It could be maintained but there was nothing left to build.
I thought up a number of new, smaller projects. Things I could do to improve things in a smaller way. But it wasn't boredom. It was longing, missing the feelings of being part of something big, of plugging holes in a ship so it could get to its destination. And doing something that affected many people in profound and often obvious ways.
Even little things. It took up to 2 minutes to load Windows. And I got that down to a fraction of the time. Users had the sense of IT moving quickly, rare in academia, even if little else did. We responded quickly and provided solutions. We could all do things we couldn't before.
I missed being the person who brought that. It was time to go and I didn't know how that would happen. The least worst trouble was that I was grossly underpaid compared to every one of my peers on campus in bigger and smaller departments.
I couldn't even see my next step.
No comments:
Post a Comment