Testing Siklu Terragraph hardware

In July of this year, we were selected to field test Siklu’s entrance into the new 60 GHz 802.11ay standard, also referred to as Facebook Terragraph. This is a multi-point technology that has both the benefits of 60 GHz spectrum speed and GPS synchronization to help with self interference. The Terragraph “magic” allows meshing of nodes (called DNs or distribution nodes). I am not going to go into the details of Terragraph or the technology. There are far smarter people than me out there to better describe it. I am going to describe the hardware and our deployment.

We have been eyeing this technology for a few months now. We have a couple locations in our downtown area where we have enough 802.11ad gear deployed that we are seeing self interference. We have gear deployed from Mikrotik, IgniteNet and Kwikbit in both point to point and point to multi-point configurations. Our experience with 802.11ad multi-point has been less than impressive - with Kwikbit being the best of the bunch. We fully understand the limits of the spectrum. We have a hard stop on installations beyond 200 meters in multi-point. We typically run high gain dishes as clients. We don’t have rain fade issues, we have beam forming issues, firmware issues, interference issues (self and otherwise) and generally poor performance. Your mileage will very and I only speak for our deployments. Our luck in point to point is much better with all vendors. It’s the multi-point that is not great.

We have good fiber and 10G licensed backhauls in our core downtown. From here, we were looking to put up 802.11ay gear to serve as multi-point backhauls to our customers around this core area. These customers are small businesses, condo buildings and apartment buildings. We try to provide at least 700 Mbps service to these properties but prefer 1 Gbps or more.

Siklu Multihaul TG N366n distribution node radio

Our hope was one 360 degree 802.11ay radio could replace 4 to 6 existing 802.11ad radios. This would clean up our spectrum and simplify future deployment. It may also allow for future meshing of additional distributions nodes.

Enter Siklu Multihaul TG N366N distribution node (DN) and the T265 client node (CN). The N366 is a 360 degree radio with four 90 degree sector antennas. Each sector antenna could be on a different channel. The equipment supports channels 1-4 but does not support channel bonding as of the writing of this post. The T265 is a 90 degree client radio with a beam forming antenna.

What we liked about the Siklu product is the 360 degree DN radio. Where we were looking to deploy, we already had clients in all directions. Second, the Siklu product did not rely on IPv6 and also supported layer 2 bridging right out of the box. This was in line with our current deployment method. We also have Siklu products in our portfolio and are familiar with their build quality.

For an in depth video review of the unboxing and build quality, see our review here:

Build quality is in line with Siklu’s other products. Firmware is still early. We are running 1.1.0 and will focus on that version for this review.

After some channel planning, we decided to run with the suggested A-B-A-B channel plan for the 4 sectors. We are using channels 2 and 4 in our configuration. We have seven client sites selected with the furthest site being 180 meters from the DN. All within the specifications of the equipment and the technology.

Take a look at our installation video for both the DN and a client site.

Configuration of the system is pretty straightforward. Out of the box, our hardware was running an early release that did not have a mature web interface. So, we logged in via CLI and issued a command to load new firmware via FTP to the radio and then rebooted.

Once 1.1.0 was loaded, all future configuration can be done via the web interface. Very little configuration needs to be done for the system to work. We started with the DN. We first named the DN with a unique 8 character name (can be less) and then rebooted it. Next, we added our management IPs. Like other Siklu gear, the MultiHaul TN product line can have multiple IPs and VLANs on every interface. This allows us to program it using the default IP and not have to reboot and reconfigure our computer to move IPs.

Next, we assigned radio channels to each of the 4 sector antennas. There is one final step on the DN - entering your links but more on that later.

Next, take a T265 out of the box. Again, due to old firmware, we logged in via CLI and upgraded firmware. Now, log in to the web interface. Only one step is required - give the CN a unique 8 character name. That is the the name that the DN will use to connect and authorize the radio. You can get into changing the SSID, encryption keys, etc but we did not and I would not recommend it. You can give the CN an IP as well but it is not required to connect it.

Once you have the 8 character (or less) unique name of the CN, you now build a link in your DN. It is as simple as adding that unique CN name and telling the DN what sector antenna it should see. You can select 1 to 4 of them so if you are not sure, select the antennas you think it will connect to and then edit this once it connects.

That’s basically it. Repeat this for every CN you want to deploy. Just like other Siklu gear, you can get very granular with IPs, VLANs, access ports, etc. You build bridges, add interfaces, untag VLANs, etc. All very Siklu friendly.

Siklu T265 CN client node

Performance

What have we seen for throughput and performance with the system? Well, that has been a bit of a mixed bag. Let me start by saying Siklu engineers and support staff have been nothing short of wonderful to work with. They are responding to support emails at 11:00pm on a Saturday night, weekends, you name it. This is also a product early in its life and one where the manufacturers (Siklu, Cambium, IgniteNet) are taking a radio standard built by Facebook and engineering that into their product with their firmware. Not a simple task.

So, with that in mind, we have had some issues. Our first DN would disconnect clients every few days for no reason that we could discover. That DN was replaced with RMA and the new DN has not had those issues with over 2 months of uptime. Potentially a GPS related hardware issue that impacted a a small number of devices.

Second issue we have seen is throughput related and this is currently being blamed on the Terragraph standard itself - but we are told a firmware fix is pending. What we see is if you have a single CN on a sector, you get ~ 1 Gbps of actual TCP throughput. As soon as you add a second radio to that same sector, your throughput is cut by about 50%. It does not matter if that second radio is actually installed. As soon as it is administratively added to the DN as a link and activated, the throughput is cut.

Here is a TCP speed test from a laptop plugged directly into the T265 CN with a speedtest.net test back to our own server on our network. This is using the speedtest.net app, not the web interface. We are not seeing full ~950 Mbps of the laptop port due to some network congestion on this link.

TCP speed test with one CN on that sector

Now, we administratively turn on a second CN to that same sector on the DN and this is the next speed test:

TCP speed test from same CN with a second CN link turned on for the same sector

We can duplicate this result on every sector with every channel and using speedtest.net tests or TCP bandwidth testing in our Mikrotik routers just going through the link itself. It show very consistent 50% drop in throughput as soon as the 2nd (or third) link is activated in the DN on that sector. Just have one link on a sector? That sector will see full speed performance. Does not matter what the other sectors are doing.

Mikrotik bandwidth test through only Terragraph gear with two links active on that sector. Results show both one way tests and simultaneous.

If / when this is fixed with a firmware upgrade, I will post that information.

Conclusion

I love the hardware, I love the software and I love the concept of what Terragraph brings. But, from a performance standpoint, we are not there yet. I think that will change and we are leaving this gear up since it fixes our self interference issues, but I look forward to the speed bosts that are promised.

Update November 30, 2021

We have continued to work with Siklu on the speeds and the GPS issues.

On the speeds, firmware 1.1.4 was released and it supports flexible bandwidth control. Running some speed tests with 1.1.4, we are seeing the speed issues improve greatly. We can now just about max out the 1G Ethernet port on our testing laptop running TCP bandwidth tests back to our core test server. I would say the flexible bandwidth is working well.

On the GPS issue, we continue to have DN reboots and CN disconnects up through firmware 1.1.4. Siklu believes we may have faulty hardware on the DN. We have been sent our 4th and 5th DNs for testing. We already have RMAd one, we bought a third as a spare and both the second (RMA replacement) and the third will reboot randomly. Logs are showing it is related to GPS reception. There is a watchdog in the radio that will eventually reboot the DN if it looses GPS sync for a set period of time. We know there are no obstructions blocking GPS signals to this radio. We went so far as to put anti-bird spikes on the top of the radio to keep birds from landing on it and potentially blocking GPS signals. So, they sent us two more DNs so we can RMA #2 and #3 and send them in for support investigation.

However, we have been working on this test since July. We have yet to go more than a week without some piece of hardware rebooting. With snow coming, we are discussing what our next steps will be. Some of these radios will not be physically accessible once there is snow on roofs. We need to decide if we are going to leave these radios up through the winter…

I know at least one other operator with many more radios up than we have and they are not seeing issues like ours. We might just be in a weird spot for this test where some sort of RF issue is impacting these. I can’t explain why we have ongoing troubles and others do not. Siklu engineers are at least publicly baffled as well. I don’t think there is a system wide issue with Siklu Terragraph but we have not had a great experiment at our test location.

January 4, 2022 update

At the end of 2021, we made the decision to remove the Siklu Terragraph equipment from our network. We continued to have issues with both DNs and CNs rebooting on us. Siklu was incredibly responsive to this issue and event sent people onsite to troubleshoot. In the end, we needed to stabilize this part of our network and move on to other projects. We initially thought this test would be a month or so and we would move on. Six months later, we were still climbing on roofs and replacing hardware to try to narrow down what was going on. As a small shop, we had to move on and divert our resources to new projects. We continue to be very happy with our EtherHaul and MultiHaul systems we have in production but we just could'’t get the bugs worked out of the Terragraph at this location. I know other operators that have not had a single issue like ours.

Our equipment - update #4 for 2018

Looks like it has been around 18 months since we last updated our equipment information and there have been a couple significant changes.

We changed our upstream ISP about 12 months ago. We used to buy fiber and Internet access from Comcast. While this worked fine, we started looking for more of a strategic partner for our bandwidth. We found another WISP in Colorado that was expanding their business connection footprint into Golden and after some discussions, we realized it made sense to buy bandwidth from them. They took over our Comcast contract to the Mountaineering Center downtown and then they added a 10G fiber link to our tower on Lookout mountain. This gave us access to significantly more bandwidth at cheaper pricing than Comcast. It also gave us redundancy between the two locations.

With this change, we had to move away from the Peplink core router. It was time to finish learning the Mikrotik and switch back to them. We started off with a CCR1009 at both the Mountaineering Center and the Lookout Mountain tower. These were connected via 1G fiber to our ISPs routers in each location (also Mikrotiks).

We also removed our AirFiber 24 link from the Mountaineering Center to Lookout and replaced it with a Siklu 1 Gbps link operating in the 70 GHz band. This Siklu link has two Ethernet ports on it. We use one to route our management VLAN between locations. Our ISP uses the other to build a redundant link for both of our networks incase of a fiber failure at either location. Our ISP took care of BGP routing of our public IP space - one less thing for us to worry about and it makes sense since they own the public range we use.

We recently upgraded both of our edge routers to Mikrotik CCR1016-12S-1S+ routers. We did this to remove a single point of failure at each location - the edge switch - and move all of our connections to fiber. Now, each CCR1016-12S-1S+ is connected to 2 or 3 Netonix switches that in turn hit all our radios. These Netonix all are home run wired back to the CCR1016-12S-1S+ via redundant fiber (using RSTP). If we loose a switch now, we only loose the radios attached to it. In the past, we had one main switch that fed other switches via copper. If we lost our main switch, we lost everything. As we keep adding radios and backhaul links, I needed to increase throughput to each switch, remove single points of failure, and get away from RF interference the longer Ethernet runs were experiencing on our towers. Plus, I like being electrically separated from switch to router on our towers by using glass.

Now that we have Mikrotik routers in both edge locations, we have our ISP routing around fiber failures via BGP and we are routing around failures of their hardware using simple internal route rules in the Mikrotiks.  In other words, if there is a fiber cut out in the wild, our ISP will self heal to the other fiber line via their link on the Siklu.  If our ISP has their own router failure or the connection between us and them fails at either location, we will self heal over our side of the Siklu link. The two fiber paths (Mountaineering Center and the Lookout tower) each take separate physical paths back to the main connection in an Internet hotel in downtown Denver.

So, while we are not 100% immune from problems, we are well protected from a large percentage of potential issues.

We were able to re-use the AirFiber 24 radios to become a point to point to a large condo building in Golden. From this building, we placed an IgniteNet 10G Omni (60 GHz radio) to supply 2 Gbps connections to a number of other condo buildings in the area - allowing us to grow our "on network" building footprint and offer speeds well in excess of 100 Mbps in each of these buildings. Many can now support eventual gigabit speeds with no additional hardware upgrades. We just need to buy more bandwidth...

In each of our "on network" buildings, we place another Mikrotik router (various models) to provide both private and public IP routing to the customers in each building. This allows us to bandwidth shape at each building as well.

Let's talk about network management now... That was also a big change in 2017. The network finally outgrew QuickBooks and manual management. Both from a simple invoicing and money collection standpoint to an automated work flow point. We contracted with Sonar and now use them for 100% of our customer management, billing, traffic shaping, sales, installation, etc. A perspective member signs up on our website via a form that links to TowerCoverage.com  That form pre-qualifies the person and then dumps all their information into Sonar for us. We then schedule roof visits and installs via Sonar and kick the work orders out to our installers automatically. Once installed, we activate the customer in Sonar, it assigns them an IP address and then programs the correct router and places them into the correct speed package and builds the interface queue. It bills them and allows them to pay online. If a customer is delinquent, it will automatically slow their connection to 1 Mbps and allow access only to our billing portal. The moment they pay, it automatically releases that restriction and they are back online. Everything is automated! Took a bunch of "busy work" from the admin.

We have added a couple more licensed point to point links as well as a handful of unlicensed 60 GHz links capable of 2 Gbps of throughput. We are removing as much 5 GHz point to point as possible.

2018 has brought a few new buildings on-line as well as expansion into two additional neighborhoods we did not service before.

No Point to Multi-point equipment changes are on the horizon. We continue to be very happy with the Cambium ePMP gear and look forward to the ePMP 3000 line coming out this year.

Easley West is live

Easley West mini-POP

The Easley West mini-pop is live and servicing members previously hidden by terrain and houses from the Easley East site.  We have also upgraded equipment on both Easley Rd mini-pops and can now provide speeds up to 100 Mbps in the neighborhood.  

Easley Rd site is up!

Member antenna looking towards the tower on the house in the distance

After weeks of preparation, four volunteers helping with the install of the repeater site and a week of testing the site with an existing member, we finally added our first "new" member to the Easley Rd antenna site. We are officially calling this antenna "Easley-East" since there are already plans in the works for Easley-West. However, at this point, Easley-East should have no problems servicing the residents that have asked for service from us. Five additional new members are scheduled for installation prior to the July 4 holiday.

This is AU Wireless' second mini-pop (or neighborhood tower). The first was added in Harmony Village. We have plans in the works to add a third mini-pop in the Mountain Ridge neighborhood later this summer. Stay tuned for details on that.

Testing, testing and more testing

When we started this project in August, 2015, I wanted to make sure we were going to use the fastest wireless system we could find for the money.  That last bit is key - for the money. We did not want radios on members homes to go much past $100. For the access points on the towers, we wanted to stay under $500.  So, that narrowed the field to a handful of vendors.  I had years of experience with Ubiquiti hardware. I have deployed systems of a dozen radios up to 100+ radios  using their hardware. In my mind, it was going to be the direction we were going to go.

We started by deploying a couple Rocket 5AC PtMP radios with UBNT 60 degree AC sector antennas. We bought some Nanobeam 5AC radios (NBE-5AC-19) as well as a couple PowerbeamAC radios (PBE-5AC-300-ISO) and began testing signals from various points all over our intended coverage area.

Let me back up slightly and describe the setup...  We are a town in a valley. Our main antenna site is about 1700' above the town and coverage starts 1 mile out from the tower and goes about 5 miles out from the tower. For those that understand most sector antennas, you see an immediate issue.

However, we had other problems. First was firmware issues. Ubiquiti has been plagued with firmware issues. They develop hardware and software very quickly. Sometimes too quickly I think. Next came plain old performance issues. I was expecting to get 100+ Mbps radio to radio throughput with this set up. We were not seeing that in most locations we tested. We were getting 50 - 80 Mbps download and a little worse upload. The primary testing area was between 1.5 and 3 miles from the antenna. We made sure the narrow vertical beam of the sector was hitting where it needed to be. RF was clean on our 20 Mhz test channel.

This made me begin to doubt my immediate feeling to go with Ubiquiti and I started reading and reading. We tried LigoWave and a couple smaller vendors. Nothing we were happy with. I kept hearing Cambium get mentioned over and over again. After a long conversation with them, we decided to deploy some ePMP test gear. Through a local vendor who also had some expert advice for us, we started testing an ePMP 1000 GPS access point into a Cambium 120 degree sector. We started with the ePMP 1000 integrated CPE and then upgraded to add some new Force 180 radios along with the Force 110 into the mix. This was a very close match to the same CPEs offered by Ubiquiti.

Not to get into the weeds with details, I was amazed by how rock solid the ePMP gear is. We were getting 100 Mbps down (30 - 60 Mbps up) from nearly all of our test sites and there were no firmware issues. Also, signal to noise with the Cambium sector was about 10 to 15 dB better than what we were seeing with Ubiquiti.  Keep in mind, we did test using the same frequencies with antennas pointed in the same direction with CPEs in the same location.  It was a well orchestrated apples to apples test. And extremely time consuming. These tests ran nearly non-stop from late August into early December.

Then it came time for some antenna tests. The narrow vertical beam width of most traditional sectors was causing an issue due to our high antenna.  I contacted RF Elements and ended up on the beta test for the new TwistPort antennas. I'll let you do the reading on them but they (on paper) solved exactly what we needed to solve.  So, I deployed two of these, one on an ePMP and one on a Rocket 5AC.

The TwistPort performed significantly better than the UBNT AC sector. No lobes, great SNR and good vertical performance.  CPEs that would not lock up on the AC sector worked with the TwistPort (which was also a 60 degree version). With the ePMP gear, it also performed very well but it drops off around 5.4 Ghz. The 5.7 and 5.8 is very crowded in our area so we need to be in 5.1 and 5.2.  The TwistPort adapter is not yet tuned to go that low. This was a major bummer because the size, price and performance are there for this antenna - just not yet at the low bands.

But, we decided to buy a couple RF Elements carrier class sectors that have fantastic 5.1 performance. Putting these on the ePMP radios (we had decided to move past Ubiquiti by now) brought our signals from around -65 to -55 and added 10 dB to the SNR. Speeds increased around 25 Mbps. This is just by putting these sectors on the AP.  I am very pleased with them and would recommend anyone needing a sector antenna, look at RF Elements and if you are 5.7 / 5.8 Ghz, try a TwistPort. You won't be disappointed.

We created a detailed spreadsheet (thanks to the RF Elements beta program) were we tracked upload and download signal strength, SNR, radio to radio speed tests and speed tests to the Internet over a 100Mbps connection. We ran 2 full sets of tests with the Rocket 5AC PtMP radio - one into the UBNT AC sector and one into the RF Elements TwistPort. We had about 12 test sites in the coverage area and tested both the NBE and PBE radios at each location.

Then, we repeated this with the Cambium ePMP radio. Two full suits of tests with the Cambium 120 sector and then the RF Elements TwistPort.  CPEs were Force 180 and Force 110 radios at each test site. These are almost exact matches to the NBE and PBE products from Ubiquiti - except Cambium does not have an AC product.

After all this, two things were very clear:

1) My wife was getting very tired of the UPS guy stopping by every day with boxes.

2) Cambium ePMP products out performed Ubiquiti AC equipment in our test environment 95% of the time (there was one radio location where UBNT gear was slightly better in speeds).  

If you are interested in seeing the full spreadsheet, contact us and I will be happy to send it along.  It has been shared with RF Elements (as part of the beta program) and Cambium.

While none of this was written as a paid endorsement, I am very pleased to be working with Cambium for our co-op WISP and thank RF Elements for a well performing antenna line.

December update and a new partnership in Golden

Finally, news to report and our December update!

In July, we started the process of looking for a building in downtown Golden with fiber optics in it for our backbone connection to the Internet. I can count on one hand the number of buildings that have fiber. Three of them are banks and are not interested in talking to co-op Internet providers. One is managed by a national leasing company that does not return five phone calls (looking your direction Source Gas building). The fifth is a building many of you know and have been in numerous times.  The American Mountaineering Center on 10th St has fiber they never hooked up but it is still in perfect condition.

I started the conversation with the building management back in August. They are a building full of like minded co-op and non-profit organizations. It was a perfect fit for us.  After a number of meetings and email conversations, yesterday we signed a deal with them for access to the fiber lines as well as roof rights to place our antenna to get the Internet up Lookout Mountain.  To say the least, we are very excited to be working with them. Next time you stop by the center, let them know you appreciate them working with us!

So, where are we with timelines? The bad news is my wife has to endure even more UPS packages on the front door as additional equipment arrives.  The fiber has been ordered as have the routers and radios to get it up Lookout Mountain. We just received 2 more (and final) test radios that would go on members homes. We are committed to finding the fastest radio equipment available today. We are down to two vendors and hope to narrow it down to one in the next 7 to 10 days. You may see me around town with a tripod full of antennas and a laptop...

I expect to transition to our high-speed line from the Mountaineering Center in mid to late December. We are adding our last test customers this week and next with our current temporary line.

If you live in the city of Golden (north of the ~ golf course), I think we can start connecting new members right after the holidays. Just in time to hook up any new electronic toys.

For those in the Easley Rd area, we are working on a relay site on the hill over there. In addition, some new radio equipment is shipping in early January we want to deploy in your neighborhood.  Our goal is to get your service up and running in late January.

We will have some more updates and information on how the co-op will work in the next week or two once we know the radio vendor and the pricing of the equipment.