Vista - The Boundless Horizon

The Risk in Every Step

When I left the Service Provider, I felt fully capable to comprehend and configure a global service provider Layer 3/Transport network. Many technologies I had configured to the point of comfort: L2VPNs, VPNv4, VPNv6, pseudowire, circuit emulation, and the Provider Core functions. If I had forgotten syntax, it should be trivial to look it up. I had never configured Carrier Supporting Carrier (MPLS over another MPLS network) but I'd done enough reading to the point. VPLS I could relearn. More importantly, I knew when and where these technologies should be used. There was a lot I didn't know, like optical transport rings, WDM, etc., but these were Rumsfield's "known unknowns".

I had gotten enough from the Service Provider. I would be leaving engineers still more advanced, but my depth was sufficient to be useful. The standard is always can I use my current depth in a way that is more effective? More depth comes when needed. It was time to stop testing and starting building.

I got the call from Vista's recruiter about some brand new (greenfield!) ISP they were looking to create. From my resume, it seemed they could use someone like me. The gentleman talked up the incredible benefits and that I'd be working on a "rock star, A-team" of engineers. I didn't believe a word, of course. I passed his screening and went to the on-site interview where I met several engineers with a final more detailed phone call with the VP of Operations who would be my boss. Ultimately, I left a solid impression and accepted the offer.

First, the bad, because there was so little of it. If you ask me what my perfect job was, I would tell you for a brief time I had it.

I was told Vista's CEO came up with a lot of ambitious visions, and then he passed them on to his engineers to make it happen. And they did. To my surprise, the engineers I met were very capable. But the downside I found with my project was that I was unmanaged and I needed direction since I didn't have the high-level vision which I needed to support.

I took a flawed understanding from the pre-hire conversations of what we were doing and had to run with it. And I would talk to one engineer and find I was way off base, then I would plan more, and communicate my current work to my VP and find out I had missed something fundamental. Most of my information came by trial and error.

For example, I was given a picture by the VP of what they were doing, with little detail. Researching online I found our parent company built the balloons for Google's Project Loon. Good gracious, that's what this was? For weeks I proceeded under the flawed assumption that I was looking at Project Loon. I had been asked to do research prior to hire, and I naturally thought through the problems of the model trying to glean lessons from the published tests thus far. Had I known, ours was similar, but very different from Loon, my planning would have taken a different path from the outset. No one told me what this was nor what this wasn't. I had to ask questions and parse the answers for clues.

As time went on I became better at figuring out which engineers to speak to, but I never abandoned  the feeling of wishing management would be proactive. If I complete high level picture was there, I never got the briefing I needed. In fact, my first official meeting with my boss was my last, when I got the news that the project was cancelled and the engineers laid off.

Things were happening at the company which I wouldn't know about until it was too late. On the flip side, not knowing had the advantage that, up until the end, I was still entirely devoted to the project and some of my best planning and research of my career came during this short time. Three years before my hire, our private company had been acquired by a large corporation. We promised profits and they promised greater capital and for a time, that we would be left unchanged. One month into my tenure, the CEO was sacked by the parent company and their COO (I believe) brought on to act as the new CEO. All the employees came into the conference room to here a sad explanation for the news and the new CEO assured us that we didn't have to worry and that this had represented a difference in vision between their company and our CEO only. A lawyer was brought in and an HR person and they finally did their audit of the projects (reasonable in my book). Finally, two months into my project, my boss came in with the parent company HR representative and let me know the news. Weeks later my boss told me he left as well.

I've wondered without answer if this explained the lack of communication earlier. Maybe. Maybe not.

I got two weeks pseudo-severance agreeing not to disparage the company, and I was unceremoniously packed up and escorted from the building within the hour. It was very abrupt and admittedly disheartening. My boss hadn't been permitted to give us any heads up and I don't envy his role in this. I suspect that this was the parent company's exit process here, rather than our own.

I had a comparable offer within the week. We were moving houses and the (effective) paid time off while I packed and job-searched was rather comfortable. All in all, I think we only had to swallow two weeks of pain.

Angry? No. Discouraged? Yes. I had willingly taken the risk. I would maintain for months after that Vista was well worth it.

Restoring the Horizon

(As a disclaimer, I'm going to generalize much of the following. The actual solutions, even the ones I did, belong to the company so, whether used or not, that's theirs. But I'm still left some latitude as to what can be said just on the basis of knowing we were not the only ones trying to solve these problems and there is a lot openly published on these things.)

First of all, the company was different. They had real engineers, that was my feeling. For us network engineers, yes what we do is complex, but by and large it's about feeding our toolbox and solving problems using the tools we have.

I was completely outclassed when an Electrical Engineer went to his whiteboard and drew out equations that I hadn't seen since university to describe the RF behavior that affected my end. I simply don't work at that level. I would come to him a number of times with a problem that lacked the necessary technology on the market and he would tell me that we could design and build a custom solution. This was the frontier. We were actually innovating, and not simply implementing any number of permutations of the same solution for which products already existed. If I had a problem we could potentially design and build our way out of it. I felt no longer bound by so many limits. My optimism soared. It was thrilling.

Whether or not it indicated a problem, I felt enormous freedom compared to what I had before. I could step out of my office whenever I wanted and grab lunch. I often took printouts of my work or research and read through them making notes over a sandwich. I took it on the train. I took it home. If a family situation happened at home, I left and took care of it. I felt naturally guilty all the time, and yet was never made to feel guilty. I felt few restrictions on my home life, so I felt more free to pour myself into the work part of that balance. And the organic conversations with other engineers gave me a sense of all sorts of very different people with very different skills, all working together as a team, each providing the necessary solutions the other needed, each working at full steam. I felt that this company made it supremely easy for us to just do what we were best geared to do. They took away our basic concerns and turned us loose on theirs.

I've always jumped at the chance to wear larger hats. There is a thrill that comes with the wondering if you will succeed in this new, greater endeavor and the reality that this time you might well finally fall on your face.

I want ambition in my work. I want the steeling myself for something I don't yet know how to handle. There are substance-abusing junkies, looking for their next high. There are adrenaline junkies looking for their next death-defying fix. I identify with these pictures. I have put myself in uncomfortable situations many times before, and always to my improvement, nearly always to smashing success. How do you let that go?

The task at hand seemed monumental, the more I looked at it. Africa is one of those places that you can still build new things. Where resources are not abundant, where qualified people are not abundant. If you want to build a new ISP, for example, this is where you look. And that was the task. Wired infrastructure for the most part can't be relied upon, particularly for last-mile access. Cell towers are expensive and need active guarding many times to prevent constant theft. Countries are often too poor for the necessary (and high) capital expenses of traditional offerings. So, there is a market which is as yet, unsaturated and even relatively untapped. Everyone uses cell phones, but business to business or high-bandwidth access are still unreliable and elusive. We target this segment of the market and we're not in competition with existing cell carriers.

Vista was looking into ways to turn balloons into the WiFi towers others find too expensive to operate. This is not a new idea. Google has Project (Bal)-Loon, using free floating high-altitude balloons to beam wireless Internet to remote villages and schools. Facebook proposes to use drones at lower altitude for the same. We considered a variant and, replacing one access technology with LTE, now looked at adding a specialized cell phone coverage and many applications to our service offering.

Alongside the company's core competencies which covered some of their solution, they needed an LTE engineer and a network engineer (myself) since at some point you would need to link the radios to the Internet.

I was told a figure for the potential value of the market throughout Africa, and told which countries we were looking at for the earlier deployments. I was told that we were looking at small, self-contained single balloon solutions as well as multiple-balloon solutions with backhaul supporting remote picocells.

This was a different animal, than the Service Provider. Yes, I can configure a global ISP service with the equipment largely in place. Given a similar proposed topology and fiber links in place, I can recreate it. Here, we start from nothing. Here, the most ambitious cases required plans for building an ISP from scratch.

Even from the interview, I got the sense that this felt more complex for me than it did for them. Maybe I got it wrong. Or maybe this is what happens when you put a RF-heavy company in the front-seat of such a project. The LTE engineer knows the equipment needed, the RF people can get the balloon antennae working, but no is used to thinking about backhaul for significant amounts of data or the network switching and LTE-supporting environment needed.

I admit a certain guilty pleasure believing that the company didn't have a clear picture for the network side of things, and that they had hired me for a task they assumed would be simpler. If I was right, I was under no illusions and adrenaline was racing through me trying to keep up with the number of questions and problems that multiplied in my mind, which I needed to solve.

If there were deficiencies in their high level view, this gave me more latitude to fill them with solutions myself, before they became minefields for the project.

You had the technology issues:

Backhaul was going to be problematic. You needed to support cell and data subscribers (individuals and high bandwidth aggregates via antenna) from your aerial eNodeB radios. But they would also need to provision remote picocells (villages, buildings) as well as backhaul to other balloons. Microwave relay was reasonable. But you have range limitations, line-of-site, dramatically lower bandwidth than wired-solutions and now the worries of vertically moving microwave relays (the company's engineers gave me the solution for this). Great, so a balloon can pass on all of its traffic, but can it do that and relay all the traffic coming from another? I worked on an architecture that would mitigate these issues and settled on a number of products worth testing.

I was right in my suspicions that MPLS could and should be extended through the backhaul network right up to the eNodeB's. That's apparently industry standard for LTE deployments. I needed to set the boundaries of the backhaul/access network as well as the intersection with the packet data network which carries native IP data once it is spit out of the EPC (Evolved Provider Core).

There were some wrinkles that needed to be solved on the LTE side whereby, in order to offer the same sort of IP Transport services I was used to and which I think would greatly enrich our target market, I needed several features to be implemented. My LTE colleague wasn't aware of these, if they even existed, so we investigated.

Balloons have payload weight limitations in addition to increased environmental factors (temperature and humidity at higher altitudes) which require ruggedizing. I would find equipment I needed only to realize it was too heavy, or it was light but wouldn't survive the climate or temperature. For interconnections to the ground, there was very limited fiber by reason of weight and other issues due to implementation. I learned about CWDM and DWDM to make the most of what I had. I would rely on a number of less common solutions to get this working. I learned about the exact capacity of the eNodeB cell radios to make sure I could accommodate the traffic I was given.

I didn't know anything about 4G and LTE. I would have to come up to speed quickly as to how eNodeB radios, SVGWs (Serving Gateways), MMEs (Mobility Management Entities) and PDNGW (Packet Data Network Gateways) interacted. These would both feed the network I was hired to build and (in the LTE model) depend on the IP access network to join them together.

Then the less immediate technology issues:

It's somewhat simpler to put everything in the same chassis (LTE-in-a-box) and then link into some Internet connection, whether wired or a satellite link. You could use this for emergency situations. The higher up you went, the more outstanding the coverage, so long as signal strength was acceptable. But if the model was successful you would be looking at potentially separating the EPC from the eNodeB radios, backhauling the latter to the former. This would require infrastructure, identifying sites, power, physical security.

If the distance was great, then the backhaul grew more complex. Would you use microwave still? Would you need transport using another ISP?

What about situations where a balloon served as an eNodeB installation for the subscribers as well as a relay from another eNodeB or picocell setup? Given a worst case, complex multiple eNodeB installation, the backhaul solution would likely involve a mix of microwave relay and ISP-rented links presenting capacity and failover issues.

I had to research what service providers existed in various countries. What was the avialable infrastructure? Some places everything was antiquated, in others, it surprisingly modern. Would we use transit providers for Internet connectivity? Would be peer directly with an Internet Exchange Point? Some countries had one or more IXPs for various regions, so you would need transport circuits to get there. Other countries, the solution was more complex.

And if we were to be an ISP of any kind, we'd need addresses. IPv6 addresses would be easy and luckily this part of the world was the only part no to yet have to exhausted its IPv4 allocation.

How would I acquire addresses? I would need contacts at many ISPs in the region as well at at the Regional IP Registry. This, at a minimum.

For plants of scale, you would be looking at redundancy in eNodeBs as well as in the EPC, possibly with the spin-off of multiple physical datacenters which would have to be supported as well as connected with the larger network.

We would finally have to factor coverage density and strength over a variety of environments, urban and rural densities. One balloon is trivial. Multiple over a large population center adds new complexity. Splicing population centers together into one consistent entity would make it even more interesting.

Still less immediately:

I wanted a full offering for the customers. Old technologies (if even used) like circuit emulation for T1 or ATM, we wouldn't support. But I wanted to enable L2VPN, L3VPN (VPNv4 and VPNv6), VPLS. If I could work in Carrier Supporting Carrier (CSC, whereby one MPLS network is transported over ours) that could potentially help capture government work. I had to envision various use cases which we could present to potential customers.

Too, on the datacenter side, I looked at what we could do. Starting from the way we sold an existing technology, we had the potential with even one datacenter, to offer simple colocation of equipment, or perhaps some managed services up to full hosting services where we rented them virtual space on physical VM hosts or even configured and ran the service entirely as a subscription. This model opened such potential offerings.

At this point, I'm sure this was well beyond what my VP had envisioned. This isn't what the company did. But then, offering cell phone service or even data transport to the Internet wasn't what they did either. They were trying to take their core competency, and use it in a secondary capacity, while building an entirely new core competency from scratch (as I saw it). So, if that's what they were doing, the sky is the limit. Think big. And when the day came, give the C-suite an idea of just how far this endeavor could take them. From so many varied positions I've had in the past, I knew what we needed.

At the least, I knew we would need more engineers to do this. I can configure a network but I'm not well versed in the security and tools side, capabilities I had early learned were crucial from my days at Time Warner Cable. You have to secure the network and monitor it to make sure everything is running correctly. You're inviting disaster for a service provider if you don't do this. Compared to what we needed here, all of my previous work on these lines was amateurish. So I would need help. I would need a team. But that was for the long-term, after this effort showed proof-of-concept technical success.

Which brings me to the next issue: you need an organization. You will need support and provisioning models. How will you handle and assign the various tasks. How many datacenters? How many POP sites? What would the security and data architectures look like? At least at a high level I needed to be able to point to something to show upper management and they would take to their customers.

I fully expected, after I got all my thoughts ordered on paper, that I was going to have to go back to old friends, Cisco and Juniper engineers and architects, and people I knew from Time Warner Cable, and vet this. I liked the feeling of knowing what was necessary, looking to the next steps, and having the freedom to adjust my time and get in the car and make contacts if that's what it took.

Everything you do will need a roadmap between here and the next level of expansion, should this be successful. So I was thinking in terms of scalability as well. We might stop with a very limited product offering. But if my boss was right and we found it possible to capture more of the market's potential...

I was thinking big. I loved it. I felt in my element. I was taking care of the smaller problems, making them more manageable, but I had the big picture in mind. Where could we take this? In five years, what would this be worth?

And you had political and out-sourcing considerations. Some countries are not stable. Other countries are. The market may have potential in both, but the approaches are different as well as the possibility (or not) of on-ground staff. Political considerations and bureaucracy would affect everything. I did as much superficial research as I could just to prepare myself for issues that (possibly) the execs at our company already knew about. I needed to know.

Lastly, the big one: the viability of the market depends on people seeing value in your offering and being willing to pay for it. At no time did anyone talk to me about what people paid for cell phone service let alone existing data and data transport services for business. We weren't looking to compete with phone carriers, of course, but what people pay for existing service will reflect in what they are willing to pay for something better. If we offered, even small scale, the most wonderful solution in the market, but individuals couldn't use it, businesses didn't want it, nor did governments, it was all for nothing.

If I was over thinking even half of this, I was energized to be thinking about it. Admittedly, I find it much harder to go back to anything that is normal, regular technical-only engineering work. It just doesn't have the same appeal.

I like thinking big and having to solve all the problems underneath to get me there.

So, if it expanded my horizon, it felt worth it.

It would become a target to which I now aspire.