FiOS: Gigabit Ethernet Stalls and Challenges

So we got this extremely complex design into production and ready to go. We get it into Verizon's hands when we said we would. Then nothing happens.

I want to back up a second to talk about the 622 Mbps limitation and why we thought it was not a big deal for a first release. The first installations of FiOS had a single OC-3c (155 Mbps) connection at the Central Office. This moved to one per shelf or 3 per Optical Line Terminal (OLT). When we would go to Gigabit Ethernet, we would quadruple the downstream capacity of a shelf to 622 Mbps in the first release and quadruple it again to 2.5 Gbps. This seemed like a good plan.

Well, what stalled things had nothing to do with this. It had to do with the complexity involved in provisioning the systems. Altough the Verizon technical team had given us quite explicit instructions on what to do, apparently they never shared them with the OSS Development Team. That group of Verizon folks said that what we were doing was too complicated. As an aside, we asked costantly about presenting the new interface to the OSS group as WE thought it was too complicated.

So, we had a working card and software and Verizon could not deploy it. Talk about Facepalms. The best part is that we had rushed and expedited to get to this stalled place. If we had known that this extra time would be required, we could have delivered the 2.4 Gbps version first out. The only reason we didn't do that was the complexity of the provisioning model. This took so much time in software to develop that we didn't have the staff (nor did we want to add the risk) of doing that work.

On Verizon's side, this was also a big issue. The reason to go to Ethernet was a huge cost reduction in the Router Infrastructure. Between Ethernet and ATM interfaces, there was a big cost delta. On top of that Verizon had a plan to replace their existing Edge Routers with newer versions from Juniper. All of this had to go on hold while we awaited the software changes to the provisioning model. We eventually ended up with a model that looked a lot like one we had propsed at the beginning. We simply mapped VLANs to VCCs and all was good.

Now comes the downside and the lesson. The Juniper Router we plugged into for ATM had a 622 Mbps limitation on GigE interfaces. Our first release of the UMC reinforced that idea and Verizon never really understood that this would not be a long term limitation. Perhaps they were jaded by past promises, but the very next FiOS release was capable of 2.4 Gbps. Unfortunately this message never got around Verizon.

As you can see, this story relates to communications failures. The failure in this case was inside the customer and we could see it coming like a freight train. The mistake was not working to mitigating the problems up front. What we likely should have done is prompt the OSS team to take an early look at the work off the record. We were asked by the Technical Team not to bother them, and we were obedient. Later events will show this was a horrible mistake.

Eventually, the UMC GigE card made it into the Verizon network in excellent volume. We eventually ported it back to the DLC line and made it available for IPTV applications. Tellabs had effectively killed the IOC business for the UMC through its decisions and so we sold few GigEs to IP video applications. But the card sold sigificant volumes to Verizon alone.

Well, tonight is the Calix and Cyan calls. I will be listening to the Cyan one and will post about it tomorrow.

Jim Sackman
FocalPoint Business Coaching
http://www.jimsackman.focalpointcoaching.com/
We Focus On Your Business - Time, Team, Money, Exit
Business Coaching, Sales Training, Web Marketing, Behavioral Assessments, Financial Analysis
Lunch and Learn Tickets!