How I Upgraded 50 Facilities to Cloud-Based VoiceHow I Upgraded 50 Facilities to Cloud-Based Voice

Last year, my director asked me to select and upgrade 50 of our facilities to a cloud-based VoIP solution. We’d had some experience with cloud-based voice solutions, having had trialed various vendors and having acquired some through mergers, and were ready for this next step.

But before I launch into my deployment story, let me tell you a bit about myself.

Who I Am

Fairly fresh out of the military in the late ‘90s, I found myself back in school and exploring career options in IT. An internship I landed turned into a permanent job as an IT tech supporting computers. I earned my Microsoft Certified Systems Engineer certification -- back when we were running Windows NT -- and then took a bunch of Avaya classes and became the “phone guru.”

Twenty years later, I’m with this same company, a Fortune 500 enterprise, as senior voice architect (actually, I’m the only voice architect but shhh, that’s just between us). I have much to be thankful for in my career here. I’ve had the opportunity to travel to dozens of offices, from coast to coast, building and supporting our voice systems, rightsizing services, reducing costs, and training personnel. I’ve managed mergers and divestitures; migrated multiple intercontinental private lines across the country; built an enterprise fax solution, then outsourced it 12 years later; and installed our first VoIP call center.

Fast-forward to 2018, and I found myself facing the cloud-based VoIP goal with all eyes on me. I say “me” because my department was downsized several years ago. We went from a voice department of nine, down to two, then one. Over time, the company merged my department with our network engineering department, and outsourced telecom expense management (TEM) and first-level help desk functions to a third party.

Cloud VoIP Project: Three Goals

After a few years of dabbling in cloud-based VoIP, my company selected a go-forward solution and, as I noted in the opening, directed me to upgrade 50 of our facilities (roughly 3,000 handsets) this year. The goals were to reduce operating cost, provide greater reliability, and improve support.

From small sales offices to large manufacturing plants, our facilities vary in size from two up to about 500 employees. Infrastructure varies from modern Cat 6 cabling with perfectly-labeled patch panels and shiny Power over Ethernet switches to faded and brittle 66 blocks under layers of warehouse dust and grime and serial ties to the telco demarc nowhere near the data center.

Needless to say, I knew that I had my work cut out for me.

Question: How do you eat an elephant?

Answer: One bite at a time.

Assembling My Team

Successful projects don’t happen in a vacuum. While I might be the only employee in my cost center, I still had a team to lean on. Each member listed below is an important stakeholder, whether they realize it or not -- and I stress this fact early in my kickoff calls.

In no particular order, here are my partners for this VoIP migration project:

TEM Vendor -- I used our TEM vendor to gather general operating costs at the launch of the project. Then I returned to the vendor on a regular basis for bill copies and customer service records. Finally, the TEM vendor reconciled the initial bill for the new cloud-based VoIP service against the quote of each site.

Field IT -- Field IT participation is critical. We have more than 100 IT personnel in the field -- our eyes and ears and boots on the ground. They’re also our liaison with the business -- our customer. The good ones see the big picture. They understand the value of the project and help point out potential hurdles in advance. The great ones read the material l provide and work proactively.

Network Engineer -- A great network engineer is critical to a successful VoIP deployment. Just like a concrete foundation is essential when building a house, a solid network is critical to supporting VoIP. Our network architect sets the standards, then our network engineers assembled a process we could repeat for each site. That said, not every site was a simple “wash, rinse, and repeat.” The network engineer and I had to adapt to and overcome various challenging environments several times.

Voice Vendor Project Manager -- We were able to negotiate a project manager into the deal with our cloud-based provider. Already having had experience rolling out cloud-based solutions with a few different providers, I can say that having a seasoned project manager on the vendor side was definitely helpful. As my telco point of contact, she was able to head off potential issues before they evolved into problems. She was my second set of eyes to ensure thoroughness of my cut sheets. She was the liaison with the local number portability group, the engineers, the designers, and so on. She was invaluable!

The Business -- Getting a business representative from each facility to participate in the project was a challenge. Some held the notion that these VoIP migrations were strictly IT projects and they shouldn’t be bothered with the details. This is where many IT folks need to come out of their shells, and shine. It’s time to exercise those communication skills; bridge the gap between the business and IT; and make a compelling argument that input from the business is crucial to ensure that we shape their new communications system to help them drive efficiencies. Once I made my pitch, some quickly jumped on board and helped out. They were excited to come into the 21st century with their communications system and equally happy that I was going to pay for it. Others required more guidance, and sometimes I needed to hold their hands through the process and drag them along kicking and screaming. And if stakeholders didn’t come around quickly enough, I let them know that I’d take their locations off the migration list and move on -- and I had a formal process for how this went down, as discussed below.

Voice Vendor Technicians -- Per terms of our contract, vendor-provided techs were responsible for unboxing, testing, and installing the phones. This was an extremely valuable contract term, especially in remote locations that lacked field IT personnel.

Me -- I’m a blend of a telecom director, architect, and project manager all wrapped into one.

My Methodology

“Respect the process!” We’ve all heard those three little words for many years. It was recently popularized by Marcus Lemonis from the reality TV show, “The Profit.” That simple phrase carries loads of weight when it comes to project management. Without this formula, chaos and frustration would erupt in short order. Documenting and following a process made my life easier; outlined a roadmap for the team; and made it possible to aggressively upgrade the 50 sites -- and a few extra -- in 2018.

Establish Repeatable Processes

Wash, rinse, and repeat. Ford made it possible to mass produce vehicles using a production line. Likewise, I needed a repeatable process to upgrade 50 locations in 12 months. Build a project plan. Follow it. That’s fine -- nothing new. But I wanted to take it a step further.

I built a system, and used my workbook as the framework for EVERYTHING (more on this later.) As far as I was concerned, if something important about phones wasn’t documented in this workbook, it didn’t happen. My workbook contained multiple tabs to collect various information. It allowed me to apply the same process for each project over and over and over again.

To kick off a site upgrade, I’d begin with an all-hands team call to introduce the project and let the business know that the cost is covered in my budget. Next, I hit the FAQs. Then, I moved on to roles and responsibilities, expectations, timeframes, training, technical aspects, and the logistics of porting on the go-live date. After doing a few VoIP migrations, I began to memorize and give the same speech for the kickoff call of each project.

Determine and Prioritize Locations for Deployment

Our company has more than 250 offices in the U.S. I needed to select 50 of them. To help make my selections, I first reached out to our TEM provider to gather data. My question was simple: “What portion of our circuits and landlines could we eliminate with a cloud-based solution, and how much do we pay for them?

After gathering that data, I had to identify candidate sites and prioritize them. I probably could have analyzed 10 different perspectives and played with the data in 20 different ways (just ask any statistician). Paralysis by analysis could have been an easy trap, but time was ticking, and I had a sense of urgency riding on my back. To boil it down, I simply needed to grab the low-hanging fruit.

I mined our data looking for answers to a couple of questions:

And then I broke that down to a cost per phone.

Since our voice systems were as diverse as the fish in the sea, I had no way of connecting to each system to pull quantities of stations. The next-best method was to merge an employee count from PeopleSoft, our enterprise resource planning database, with a telecom operational cost per site to arrive at an average cost per user. While each employee doesn’t necessarily equate to one phone, I concluded that this method was going to provide the best average. Perfection is the enemy of good enough. Over-analyses would drag me down, and I knew it. With few exceptions, I found that the data supported what I already suspected.

In general, small offices had a much higher operating cost (per user) and upgrading these offices would produce a quicker ROI than upgrading larger offices. Dial tone in the cloud is nothing new to small businesses. But what about medium-sized or large companies? Where is that tipping point for keeping voice on premises rather than putting it in the cloud? The results to that question are very interesting… but a discussion topic for another day.

Several other factors influenced if and when a site would be upgraded to cloud-based voice. Questions I considered included:

What costs can be eliminated after moving to the cloud?

Formalize Discovery & Data Collection

Gathering all the necessary information to make each VoIP migration a smooth transition must be done ahead of the order.

The cut sheet must be completed as thoroughly as possible. I made it clear to the business that the cut sheet is what the carrier would use to program the new system. The programmer would be someone they never met, not me. So they couldn’t make any assumptions. Every field is a required field, and if it’s not documented in the cut sheet, it won’t get programmed in the new system.

The trunks, circuits, POTS lines, DIDs, toll-free numbers, etc. all must be identified and documented.

The DHCP server, VLANs, IP scopes, network switches, cabling limitations must all be known, documented, and shared with several team members.

Conduct a Thorough Analysis

I would begin analyzing site information as it came in. Sometimes we had to make a “no go” decision, at which point we’d drop the work on that site and I’d move on to another. But more often than not, we’d continue through to project completion.

The analysis is not just a cost analysis. Evaluation of each site’s wiring was also essential, as we often found Cat 3 cabling in use. A lack of Cat 5/6 cabling in some areas meant that we needed to stop and evaluate alternatives to VoIP phones when new wiring was cost prohibitive.

Our network engineer reviewed the switches and routers. Do the switches support LLDP and PoE? Are those features must-have, all the time? (Answer: Not always, but LLDP is usually a minimum requirement.)

Execute on the cutover

Execution, in my opinion, was the easiest step of the project because we had completed 90% of the work during normal business hours in the weeks leading up to the go-live date. Execution was the remaining 10% of the work on the morning of the VoIP migration. When things went smoothly, the cutover was mostly an uneventful conference call with the team while the phone company ported the numbers. The old PBX remained up. Existing TDM phones remained in use. The real event was that inbound calls that rang on the legacy PBX would now ring on the VoIP phones.

This was my favorite scenario (upgrading TDM phones to VoIP) because we placed and tested the VoIP phones well ahead of the go-live date. For about a week prior to the cutover, users had two working phones on their desks. I encouraged users to use their new phones for outbound and internal calls. I asked users to log into their Web portals and I encouraged them to install and use their mobile app. This gave the user community time to become acquainted with the new products before the go-live date and reduced stress when they forgot their voicemail or Web portal password. Inbound calls would still ring to their old phones until the port date. Taking this approach allowed us to complete most of the work ahead of the go-live date.

Some of the legwork still occurred on the morning of the cutover. For example, we had to connect analog telephone adapters to fax machines, external ringers, paging equipment, and analog phones. And, we followed a test plan, which included international and 911 calls, auto-attendant options, dial by name, etc.

Communication Is Crucial

Throughout this entire process, I can’t stress the importance of communicating with team members enough. Anybody can build a project plan, but what’s often undervalued is communications -- both verbal and written. Helping stakeholders see the big picture and understand their roles and responsibilities is one of the biggest challenges I faced. Since I couldn’t fill out a cut sheet myself; draft the script of the auto-attendants; identify network limitations; or provide a floor plan for each site, I motivated others to help with these tasks.

As much as possible, I used canned emails, or templates. I carefully drafted my messages for first contact, follow-ups, go-live expectations, training documents, and help desk process to be concise, but rich and valuable.

Weekly conference calls were equally valuable. Over time, I found, I had memorized my talking points, which helped to eliminate lulls and redundancy. Rather than just lecturing, I had best results when I peppered folks with questions every two or three minutes. Our company uses Google products, so using Google sheets allowed for everyone to be on the same page. I could easily see who was and wasn’t paying attention, and when I saw that someone wasn’t on the same tab of the workbook, I’d ask him or her a question to keep them focused.

I could have used a screen-sharing service, but these are working meetings and require participation from EVERYONE. I put ownership on the business representative to fill out the cut sheet and provide a floor plan. Field IT must provide DHCP server and switch information. Network engineering needs to request new scopes for voice VLANs and document those in the workbook. We all had our roles, and keeping everyone on task was my job.

At the start of each meeting, I’d ask each participant for a status check. If the stakeholder didn’t do his or her homework, I’d wrap up the meeting and say something like, “Well, I booked this meeting for an hour. I’ll give you 45 minutes back to work on it, and I’ll check back with you next week.” That’s it. Meeting immediately adjourned and my message was heard. If nothing was done the next week, I’d repeat the process. If this happened three times in a row, then I’d explain that I’m canceling the project because I have a couple hundred other locations that do want a system upgrade.

Usually by the end of the day I’d receive an apology, along with an email telling me that the requested work had been finished. And we'd continue on with the project.

In part two of this multipart series, I’ll share my lessons learned.