We had my boss, professorial in his mannerisms, and we had my senior engineer, who was jovial, down-to-earth, quieter, absorbing information like a sponge to be released at the appropriate times in measured doses. I occupied a middle ground, having a lot still to prove to myself. We used to have really interesting conversations that took up more time than they should.
When he left, the dynamic reset. Our workload increased and the first few months were triage. My boss's workload was divided between us. As I grabbed parts, my colleague seemed happy to release any claim to them. As a result, I got budgeting, core infrastructure planning, servers and network. He got the tasks I didn't want: the day to day provisioning of accounts, management of user concerns, the teacher electronic bulletin boards. If there was an established procedure for it, especially where he had established the procedure, he took it. And oh yes, he volunteered to work on the monumental piece of digital garbage that was our student records database: an atrocious beast based on MS Access, customized superficially, replicated for multiple customers by its one lone programmer who alone understood how it worked and whether we could get rid of him. But the extortionate payments didn't come from my budget, so not my problem.
I appreciated but never understood how content he seemed in doing much of the same tasks over and over again, getting better each time he did them. We know he got better because he timed himself (our first foray into departmental metrics). Part of the misery of our inheritance is that I remember it took 2 days properly to provision one user with everything software and hardware they would need.
In all fairness, the process was cumbersome and multi-faceted. There was a process for creating the user account on Novell. Then the shared and home drives were a separate one. Then email using Mercury with the Pegasus client. And then you had to make 100 little changes creating a local user profile on the Windows machine to make sure all of the customizations were there first login. And it went on. It was grueling. And with my boss gone, we also saw an influx of additions to the school.
The Dean of Research was our boss. She was (with rare exception) a dependably cheerful technophile. She was your favorite aunt if she liked you and the lion who jumped between you and any threat. We heard stories of those unfortunate to be on her bad side. But she liked us. The stress got to the senior engineer so much that when she popped in to ask why a certain account was taking too long she left in tears at his response. Most days we were solidly in her camp and she in ours.
These next years marked a number of transitions and change to the group dynamic. The Dean (CEO) was replaced with an Acting Dean and then a permanent one with a completely different personality. Our Dean of Research decided to step down (she hated the politics and administration) to go back to her very lucrative research projects. Our previous boisterous and nonchalant COO was replaced by an older gentleman, quieter, well-mannered and reserved who began to prove his worth immediately and became a stabilizing and efficiency-driving influence thereafter. IT fell under the COO rather than the next Dean of Research.
As for IT, I was effectively the CIO of a department of two (myself included) with no direct reports. But we were not short on problems and I had a relatively clear road ahead of me to come up with the solutions.
The senior engineer and I couldn't have been more different in our personalities. He was very conservative and liked proven methods. I took risks, was terribly impatient, aggressive with making production changes and saw potential. He spoke daily about the lack of security we had, how inefficient the processes were, how far behind we were -- and he wasn't wrong. I had the never-ending optimism of someone who hadn't done much and with few corresponding frustrations to curb me.
If we were very ill before, and if my changes often resulted in additional growing pains, he had a necessary moderating influence on my approach. I've found having someone like that to work with to be immensely useful, even as my approach has matured itself over the years. The existence of a conservative influence yields a certain freedom to be aggressive without worrying that it will go unchecked. So I produced.
It wouldn't be a perfect balance, and there would be rare, minor frustrations between us, but it worked. And our team would expand to five FTE and numerous part time workers.
We began our search for a replacement for my boss shortly after his departure. As with most everything, it proceeded slowly. My colleague and I saw a number of applicants. One of the last was a gentleman whose resume was filled with "responsible for... responsible for..." lines which I had come to see meant people gave you tasks and you were giving us no indication as to how well you did them, or even if. They were all nice people and terribly unimpressive.
After four months I mentioned seriously to my boss that if we didn't find someone in the next couple months, he wouldn't need to and I will be doing the job fully. Less than two months later our COO closed the search for an IT head and opened another to backfill my previous duties.
The Network Rebuild
First things first. I was throwing away my budget on Cisco consultants. Our C5500s kept killing their supervisors. And it became apparent that either a GBIC or the fiber link between two floors was bad. There were no redundant links. We still had a 100+Mbps ATM link to campus for the internet. We had the "BOGON" list which was the ONLY (and stateless) ACL between us and campus (thus, between us and the internet).I learned the hybrid CatOS/IOS system and replaced my first supervisor shortly after. I now could back up configurations as well. I became accustomed enough to the CatOS syntax that IOS felt foreign in the beginning. I had campus come out and replace the fiber link between floors (after much repetitive diagnostics after which most times they assured us the fiber was good and it still failed), and add a redundant link to complete the triangle between core switches.
We had C5500s so it seemed reasonable to replace them with C6500s (I wouldn't do it that way now). I researched the topologies, backbone link speeds, security controls and worked with Cisco on the quotes. Finally, it was solid and I went to the Dean's meeting requesting about $100,000. The presentation went splendidly and when they asked what more could be done, I discussed security, remote access, and wireless access. I was sent out with a request to see if we could add all those, and as they had extra monies, what would $300,000 get us?
The project moved forward with more research and a new wrinkle was added. The Campus CIO's office had created this TIER program (Technology Infrastructure for Education and Research) which might pay for our upgrade. It aroused suspicion as well. Campus went through cycles. A centralized architecture and management model would woefully underperform, forcing individual colleges and units to operate their own networks which would do the job. Campus would realize how much is being wasted through duplication of effort and so would convince the campus to move to an improved centralized model. This would realize cost savings and in short order would become a mess where departments felt under-served which would compel them to go it alone again.
We wanted the money but recognized yet another cycling of the pattern. The Dean called me into her office to explain that if campus would pay, let's use their money, but many departments would love to assume our network and functions so at all costs I needed to resist any threat to our independent operation.
So knowing nothing about Cisco engineering, I found myself attending TIER meetings with the CIO of the School of Medicine as our region's lead, pretending that, if I wasn't entirely competent, I was at the least confident and only mildly incompetent (which put me ahead of some others). Outside the meetings I would research like crazy to understand what had been said. I spoke to other campus engineers, I made good use of our Cisco sales engineers and in short order I understood.
I added security devices and other gear to the quote. C6500s were overkill but campus didn't want us putting in stacks of C3750s, so while they were paying, C6500s it was. We were to pay half, campus half. I pushed to the point of great annoyance of others to get things done quickly. Campus used its funds to buy the equipment and would request reimbursement later. Until my departure, they never did. Within a year the Cisco equipment arrived and I got to play with my first toy that cost more than many luxury cars.
Nursing was first. The School of Medicine actually followed in the years to come. Dentistry wasn't even started by the time I left. In that time, looking over Medicine's documents, I found an error that saved them more than $100k. Icing on the cake.
We stayed fiercely independent, despite a few well-intentioned efforts of Medicine to pitch us what they could do. I actually liked those people but I had my instructions so I pushed back at any hint of consolidating equipment and management. And they knew we could manage it.
Almost immediately after, I added Sonicwall enterprise firewalls (they had DPI while Cisco did not), Cisco ASAs to function as S2S and SSL VPN concentrators, dedicated multilayer switches for the server room which I was also expanding quickly.
My colleague had complained often about outside players scanning or attempting intrusion into our unguarded Novell servers. I clicked about and his eyes nearly popped out of his head when he saw the logs immediately stop incrementing.
Our new Dean planned for a satellite site in Qatar so I spent my final months researching everything to do with those big telepresence systems, the ones with triple giant monitors and two rows of desks. I left before that happened.
Throughout my time I continued expanding the environment, introducing connections and services for external research units and remote sites. When I got bored, I hiked to other departments to see how I could help them.
When I left and Nursing reached out to Medicine to help with the network (no one else wanted to do it), the engineer who came over commented to a friend that our topology was more suited to an ISP. It wasn't a compliment I think. It pointed to overkill in his mind. But I took it very happily. Our network, unlike so many on campus, was very stable, very well engineered. It could survive multiple planned and unplanned device and link failures without the end users so much as noticing a blip. Far bigger schools struggled to go a week without a hiccup.
Years later it was confirmed that what I left in place had stayed very much in place. Problems came rarely when someone needed to make a change and no one could operate it.
No comments:
Post a Comment