Our equipment - update #4 for 2018

Looks like it has been around 18 months since we last updated our equipment information and there have been a couple significant changes.

We changed our upstream ISP about 12 months ago. We used to buy fiber and Internet access from Comcast. While this worked fine, we started looking for more of a strategic partner for our bandwidth. We found another WISP in Colorado that was expanding their business connection footprint into Golden and after some discussions, we realized it made sense to buy bandwidth from them. They took over our Comcast contract to the Mountaineering Center downtown and then they added a 10G fiber link to our tower on Lookout mountain. This gave us access to significantly more bandwidth at cheaper pricing than Comcast. It also gave us redundancy between the two locations.

With this change, we had to move away from the Peplink core router. It was time to finish learning the Mikrotik and switch back to them. We started off with a CCR1009 at both the Mountaineering Center and the Lookout Mountain tower. These were connected via 1G fiber to our ISPs routers in each location (also Mikrotiks).

We also removed our AirFiber 24 link from the Mountaineering Center to Lookout and replaced it with a Siklu 1 Gbps link operating in the 70 GHz band. This Siklu link has two Ethernet ports on it. We use one to route our management VLAN between locations. Our ISP uses the other to build a redundant link for both of our networks incase of a fiber failure at either location. Our ISP took care of BGP routing of our public IP space - one less thing for us to worry about and it makes sense since they own the public range we use.

We recently upgraded both of our edge routers to Mikrotik CCR1016-12S-1S+ routers. We did this to remove a single point of failure at each location - the edge switch - and move all of our connections to fiber. Now, each CCR1016-12S-1S+ is connected to 2 or 3 Netonix switches that in turn hit all our radios. These Netonix all are home run wired back to the CCR1016-12S-1S+ via redundant fiber (using RSTP). If we loose a switch now, we only loose the radios attached to it. In the past, we had one main switch that fed other switches via copper. If we lost our main switch, we lost everything. As we keep adding radios and backhaul links, I needed to increase throughput to each switch, remove single points of failure, and get away from RF interference the longer Ethernet runs were experiencing on our towers. Plus, I like being electrically separated from switch to router on our towers by using glass.

Now that we have Mikrotik routers in both edge locations, we have our ISP routing around fiber failures via BGP and we are routing around failures of their hardware using simple internal route rules in the Mikrotiks.  In other words, if there is a fiber cut out in the wild, our ISP will self heal to the other fiber line via their link on the Siklu.  If our ISP has their own router failure or the connection between us and them fails at either location, we will self heal over our side of the Siklu link. The two fiber paths (Mountaineering Center and the Lookout tower) each take separate physical paths back to the main connection in an Internet hotel in downtown Denver.

So, while we are not 100% immune from problems, we are well protected from a large percentage of potential issues.

We were able to re-use the AirFiber 24 radios to become a point to point to a large condo building in Golden. From this building, we placed an IgniteNet 10G Omni (60 GHz radio) to supply 2 Gbps connections to a number of other condo buildings in the area - allowing us to grow our "on network" building footprint and offer speeds well in excess of 100 Mbps in each of these buildings. Many can now support eventual gigabit speeds with no additional hardware upgrades. We just need to buy more bandwidth...

In each of our "on network" buildings, we place another Mikrotik router (various models) to provide both private and public IP routing to the customers in each building. This allows us to bandwidth shape at each building as well.

Let's talk about network management now... That was also a big change in 2017. The network finally outgrew QuickBooks and manual management. Both from a simple invoicing and money collection standpoint to an automated work flow point. We contracted with Sonar and now use them for 100% of our customer management, billing, traffic shaping, sales, installation, etc. A perspective member signs up on our website via a form that links to TowerCoverage.com  That form pre-qualifies the person and then dumps all their information into Sonar for us. We then schedule roof visits and installs via Sonar and kick the work orders out to our installers automatically. Once installed, we activate the customer in Sonar, it assigns them an IP address and then programs the correct router and places them into the correct speed package and builds the interface queue. It bills them and allows them to pay online. If a customer is delinquent, it will automatically slow their connection to 1 Mbps and allow access only to our billing portal. The moment they pay, it automatically releases that restriction and they are back online. Everything is automated! Took a bunch of "busy work" from the admin.

We have added a couple more licensed point to point links as well as a handful of unlicensed 60 GHz links capable of 2 Gbps of throughput. We are removing as much 5 GHz point to point as possible.

2018 has brought a few new buildings on-line as well as expansion into two additional neighborhoods we did not service before.

No Point to Multi-point equipment changes are on the horizon. We continue to be very happy with the Cambium ePMP gear and look forward to the ePMP 3000 line coming out this year.