Adding ER-X between Cable Modem and Amplifi for SQM
This may be of interested to some. I recently added a TP-Link Easy managed switch for my wired devices so I could do 4 priority levels for the ports, so now I have basic QoS on my wired devices.
The next step was to add an Edgerouter between my Bridged Cable Modem and Amplifi HD so I could have some sort of SQM to help with my bufferbloat rating (mine ranged from D to E).
But I didn't want to bridge my Amplifi HD and loose all it's functionality and easy management from the app.
After a couple nights and testing different setups and settings I think I found the one that works best for my setup. I tried many different variations to see how different settings reacted with each other.
Per Edgerouter (ER-X):
Pay for 150/15
When speed testing 280/18
D-E bufferbloat rating (it spikes as high at 650ms)
Bridge Cable Modem -> Amplifi HD (with 2 ethernet backhaul Amplifi HDs) -> TP-Link Easy Managed switch (for wired QoS per port)
Post Edgerouter (ER-X):
When speed testing 150-170/15
A+ bufferbloat rating (max spike is about 35ms)
Bridge Cable Modem -> Bridge ER-X -> Amplifi HD (with 2 ethernet backhaul Amplifi HDs) -> TP-Link Easy Managed switch (for wired QoS per port)
Here's how I configured the ER-X (from what I can remember, I tried many different setups).
-Used the Wan+2Lan Wizard so I could leave eth0 as the main router port and plug my laptop into to monitor and setup the router (otherwise you have no way to access it from behind your Amplifi HD since it's in front of it).
eth1 was automatically setup as the WAN and then eth2, eth3, eth4 were dropped onto switch0 automatically.
-Changed switch0 to No IP and removed eth3 & eth4 from the switch0
-Disabled eth3 and eth4 ports since I'm not using them
-Removed DHCP from from eth1 (set it at No IP)
-Made a Br0 bridge and set it as DHCP and connected eth1, eth2 & switch0 to it (very important you connect switch0 otherwise you get slower speeds with SQM for some reason).
-Enabled hwnat and ipsec offloading through CLI (the only thing I used CLI for)
-I believe there was a DNS server for switch0 that the wizard made also, 192.168.2.1 so I disabled that since I don't need it.
Now I had my public IP showing on Br0, and with my Amplifi HD plugged into eth1 my public IP was being passed directly onto it and all my devices had internet.
With no SQM I would get my normal 280/17 and D bufferbloat rating. So I confirmed the ER-X was doing nothing more than passing the internet through it.
I then turned SQM on eth1, set it at 1 Mp up and 1 Mb down because I wanted to make sure it was working. Ran a speed test and didn't get more than 1Mb. Which is perfect because I've now confirmed it can apply SQM to a bridge interface.
I played with my SQM up and down numbers until I got an A+ on my bufferbloat rating.
I set the up to 17. But the down it didn't matter how high I set it as I always got an A+ rating as long as my up was set at 17 or lower. My down speeds were always 150-180 as long as I set the down higher than 200. So I know that I'm maxing out the Edgerouters SQM CPU capabilities on the down (the CPU was hitting around 70%, which seems like it has more headroom but I couldn't get anymore than 150-180 down out of it).
I'm thinking if I go to an ER-12 that has a better CPU I'll be able to get some down speed back. But for now I'm pretty happy with 150-180 down.
I ran a couple rounds on my sons Fortnite to watch the pings (sorry, easiest way for me to see lag and ping spikes, there's definitely better ways to test).
It had the lowest pings I've ever seen, normally it sits around 35-40ms, except when my NAS or desktop upload it would spike as high as 200ms. While the NAS was uploading last night consuming all the up bandwidth as usual, the Fortnite pings sat at an incredibly stable 27-30ms. I've never seen them so stable, sometimes even dropping as low as 19ms which I've never seen before.
It was only 1 test though late last night when I finished setting it up so maybe the network was just "quiet", so I have more tests to run and want to see what it does when the desktop uploads which used to cause the worst ping spikes. I should know for sure over the next few days.
We also have a nightly incoming backup from Filezilla on one of our NAS and our other NAS backups offsite to another QNAP and both backups had no issue getting through the ER-X -> Amplifi.
Showing promising results so far.
Oh and more more thing. I was able to enable DPI and see all the traffic on my system. If didn't seem to disable SQM either because I still got an A+ rating when I tested, but the CPU took a hit and speeds where slower as would be expected. I did leave it off though as I don't really need it, but just cool to see as I've never had anything that would break down traffic and show you exactly what your system is doing.
Hope this helps for anyone who would like some SQM on the Amplifi but don't want to loose all it's features by bridging it.
Thanks @LainB - nice write up, great results...and some some déjà vu
The ER-X has been running perfect for the last 4 days doing SQM on my non-bridged Amplifi HD. Super impressed with it.
I let my son put in a lot of Fortnite time on the weekend so he could report back the pings to me, if there's any issues I want to know about it as soon as possible. It was pretty staple around 20-30ms the whole weekend, better than it's ever been. The lowest he saw it was 16ms, which he's never seen before.
I still have to watch the ping when the desktop uploads, but the NAS was offsite backuping pretty much all weekend (there was a huge backup to do, about 200GB of work data completely saturating the upload) and didn't make a dent in his Fortnite pings.
I've ordered an ER-12 to see if it's better CPU will get me higher than 150-180Mbps down.
It's due in at the end of the month and I'll report back once I've tested it.
(The ER-X will be going to my work since it's only a 20/20 connection).
If Amplifi added SQM into the HD and/or the Alien it would be a game changer. Although I do think some home routers have SQM/fq_codel now. I'm kind of surprised a $70 ER-X router can pull it off where much more expensive routers can't.
Last update (unless I have issues).
- I just saw that I actually removed everything from switch0 not just eth3 & eth4, so it is empty, but I needed to still add it as part of the bridge to get my 150-170Mbps of down speed.'
- I was able watch the pings when the desktop was uploading. Sat around 30-40ms, normally would go to 100-150ms solid and sometimes spike as high as +200ms. So the ER-X is working perfect!
My ER-12 came in way sooner than expected, it arrived yesterday and I just got it setup last night.
Short version - it works awesome. I'm now around 250-270Mbps down (and still 17Mbps up which is my upload cap). So it got me about another 100Mbps down. I'm not sure if I'm maxing out the CPUs capabilities or if it's my down speed that's being max (which when I test without SQM I get about 280-290). But either way upgrading to the ER-12 got me a lot more down speed.
Long version - I thought it would be an easy drop in, set it up the same as the ER-X.... was I wrong! I tried the WAN-2LAN wizard like I did with the ER-X so I could plug my WAN into eth1, eth0 would be the ER-X login port and eth2 would go to my Amplifi. I could not get any internet to pass through to any ports. I'm not a network wizard though, so maybe I was missing something, but with the ER-X I had internet on those ports right away. I even looked at the ER-X and ER-12 side by side and they were setup identical.
I then noticed that the WAN-2LAN only setups up eth2, eth3 & eth4 as the 2nd LAN just like the ER-X. It makes no mention of all the other ports on the ER-12. So I'm wondering if WAN-2LAN is from the older ER-X days and when the ER-12 came out it was never updated, so maybe it's not compatible with the ER-12. Just a thought, but either way it burnt about 3 hrs of my night trying to get WAN-2LAN to work and even tried different firmwares.
I then used the Basic setup which puts internet on eth9 and then used eth8 as my ER-12 login port. And pluged Amplifi into eth1.
I was then able to proceed like I did with the ER-X, making the bridge, changing the DHCP, disabling the 2nd LAN, ect, and got it working great.
@LainB I just recently purchased a couple of Amplifi Alien routers and I'm considering doing this same thing with my EdgeRouter 4 so that I'll have access to the features of the Aliens. All I really need the ER4 for is SQM because it makes a huge difference in my bufferbloat. Is this method still working well for you?
Hi @Jason-Ellis - have you tried the Latency Optimization QoS Setting in the Alien's web UI?
@Derek-Saville I currently have the ER4 acting as a router and the Aliens as mesh access points. Before I found LainB's posts here and on reddit, I briefly took the ER4 out of the mix and enabled QoS on the Alien. The bufferbloat results were pretty terrible.
I've tried searching a lot over the past 2 weeks (time I've had the Aliens) to see exactly what the QoS is on the Alien. It's my understanding that it's just bandwidth prioritization and not any form of smart queue management.
With kids and a wife that are constantly using the internet, it's hard to shut things down for a few hours while I experiment, but I may do that this weekend. Do you have more info about the Alien QoS and whether or not it implements any kind of SQM?
Yes, I'm still using it and it's been working great.
As mentioned I upgraded from the ER-X to the ER-12 for more bandwidth/CPU.
This actually was the jumping point for me to build a ridiculous server rack for all my gear now.
Told my wife I "needed it" for the home network
I put in the ER-X at work the exact same way because we only have a 20/20 connection there and I have the same Amplifi HDs.
I also upgraded my internet at home to 1Gbps down 30Mbps up. So now if I have 2 computers run a speed test I can hit about 700-800 Mbps. So the ER-12 gave me the bandwidth I want.
But down speed was never really an issue, the 30Mbps up was what was killing me when my NASs would offsite backup it would crush my internet and the ER-12 cleaned that right up.
As for the Alien, Amplifi told me the way the Alien Latency Optimization works is all it does is limit connections to 100Mbps, which does nothing if your upload is below 100Mbps or have a slow download speed. There is no prioritization being done or SQM. Ethernet ALWAYS takes priority over WiFi, and Wifi has 3 levels of QoS (normal, streaming, gaming).
"this feature limits devices at 100Mbps so they do not cause latency spikes."
Good luck getting time to test. I had a few 1am nights of testing because was impossible to shut down the internet for hours to test at any other time.... can't live without internet apparently
@LainB That is an INCREDIBLE rack! I'd love to have something like that, but I've basically settled for just putting all of my equipment on an ikea bookshelf.
My situation is similar to yours, I have 1 gig down, 50 mbps up. I've been running the ER4 as the router with SQM for about a year to fix my bufferbloat issues. I have a synology DSM running 24/7, so I needed help prioritizing voice chat and games over the backups.
Thanks for the info on the Alien QoS. That's what my searches have been turning up, but without seeing official confirmation I was holding out hope that I could simplify my setup by eliminating the ER4 and letting the Alien do it. And thank you so much for your write up on how to bridge the EdgeRouter. It will be nice once I get this setup and can take advantage of the Alien convenience features, because I'm otherwise really enjoying how stable they've been compared to the 3 mesh products I've used over the past few years.
@Jason-Ellis Ha, at least you are using Ikea bookshelves. I was using Walmart wood plank shelf units.
It looks like the ER-4 offloading is the same as my ER-12 (which is different than the ER-X from above)
about half way down the page it says:
Used by the following EdgeRouter models:
I enabled all the offloading.
Here's what my setup looks like for my ER-12 (it's a bit different than my ER-X). I have DHCP setup on the WAN port rather than Br0. I believe when I was testing it worked setting Br0 to DCHP and eth9 to No IP also.
Br0 is no IP. eth 9 & eth 7 enabled in it, that's all.
eth7 goes to my Amplif (No IP)
eth9 goes to my cable modem (DHCP)
I believe I used the Basic setup for the ER-12 but I was on the newest firmware (I think read something about the Basic Wizard changed in a newer firmware so I could use that one to get it pre-setup to what I needed).
You don't have to do this but I did:
eth6 I set as DHCP and I plug that into my TP-link Managed Switch when I want to remote access the ER-12 using a static IP address (from my Amplifi LAN). This was super handy when running bufferbloat tests and adjusting SQM.
But I leave it unplugged when not in use because that's essentially putting eth6 behind my Amplifi firewall and I haven't been able to confirm that is someone hacked into into my ER-12 on eth9 WAN if they'd have access to eth6 behind the Amplifi firewall, so I just leave it disconnected when not in use.
I also setup eth8 as the routers default LAN it creates so I can direct plug in and log into the ER-12 using it's LAN if needed. It's the Optional secondary LAN in the basic wizard setup.
2 recommendations when you do this.
After you have the system setup and running.
- Create a backup image with eth9 NOT in the Br0.
- Create a backup image with eht9 in Br0. (you probably won't use this one though, see below).
I haven't nailed down exactly what causes it but when I cycled the power eth9 would completely disappear from the ER-12 (happened with the ER-X also) and I'd have to factory reset and set it all up again.
So I tried making a backup image and loading in the backup but the WAN eth9 port would not come back until I factory reset.
I then made a backup with eth9 NOT in Br0 and it loaded in perfect and then I just added eth9 to Br0
I have no idea if it's just a weird glitch but I didn't get deep into testing because I have enough UPSs in my server rack to keep the rack live for 6hrs. So it's not something I will encounter regularly.
I have the backups made without eth9 in Br0 that I can drop in quickly if needed and then just make that one change.
Then it's just a case of playing with SQM to see what works.
I used http://www.dslreports.com/speedtest until I got A+ across the board.
I did a lot of reading on SQM, but I'm no expert at all and a lot of stuff I didn't really get.
ER-12 has a good CPU so I set my targets tight and I got the other settings from sources on the internet. except for the 1000000 limit on the upload. I just maxed that to see what it did.
I don't do any VOIP, so your settings may vary.
Hi @Jason-Ellis - I used to run a transparent bridged ER for SQM with the HD but upgrades on the ISP backend eliminated the buffer bloat so the ER became unnecessary
There is no official confirmation of what the Alien implements for QoS on the WAN connection
You enter the bandwidth limits manually for both upload and download speeds - it’s only 100Mvps if you don’t change the default values
Well, I worked on this yesterday and got it mostly working, but I'm not entirely sure I've worked out the kinks because I have a few weird issues.
I did notice along the way that my ER-4 doesn't have a switch built in like the ER-X or ER-12, so that's a big difference. I connected my computer for direct access on eth0 and factory reset. eth0 was assigned an IP address, which is now outside of the main subnet. I had to use the CLI to add br0, bridging eth1 and eth2.
eth1 is connected to the modem, eth2 is connected to the Alien, and then one of the Alien's LAN ports connects to my smart switch, which then connects to a few PCs and my NAS.
I tried doing as you did and assigning eth1 (WAN) to DCHP, but it won't let me, it says I cannot do so to ports in bridge mode. I'm not sure if this is a result of me not having a switch in the ER-4.
I confirmed QoS is working by dropping it to 1mbps before adjusting it up. I have it set to 50mbps on upload, which is my limit, and it's getting me As on the dslreports bufferbloat test. I may go back in and try some of the advanced tweaks you did, but after messing with the for a few hours yesterday I forgot and was just trying to get it in a stable state. I don't have QoS on the download. Speeds are similar to what they were before trying all of this, somewhere in the 700-900 range for my 1Gbps down (or lower, assuming congestion), and around 40-44 for up. I'm happy with that.
- I'm not entirely certain I haven't left any security vulnerabilities. br0, eth1, and eth2 do not have IP addresses and I do not see a way to access them on the local network (not listed in the Alien's devices). If I try accessing my public IP from outside the network, it is correctly forwarding to my NAS per my port forwarding on the Alien. This is for 80 and 443, I haven't checked any other ports.
- This is a really weird one. I have several services running on my NAS with a reverse proxy, custom domain, and subdomains. I can access these on my phone over wifi and cellular (so, outside the network). On my PCs, I can access them over wifi, but NOT when I'm connected via ethernet through the smart switch (TP Link t1500g). When connected over ethernet, I can only access the NAS with the IP and port. I did some mostly unsuccessful googling last night that says it may have something to do with NAT/hairpinning, but I don't know if that's on the router or Alien. I'm not sure if there's some sort of firewall still being applied on the ER-4 br0 interface, but I guess I'll have to take a look (pain in the butt now that I can only access through eth0 direct connection).
- This one also baffles me. My Nest cameras are working just fine on the local network, but when I turn off wifi on my phone, they are mostly frozen and buffer a lot on cellular. I don't think that Nest makes the connections locally (I may be wrong), so even viewing on wifi would require a round trip to Nest servers during viewing, so I don't understand why wifi works great but cellular barely works. I had my wife turn off the wifi on her phone to view them and she said they work fine. I'm on Android and she's on iPhone, but I don't think that's a factor.
Anyways, I really appreciate your help, things are going mostly well, I just need to figure out the fine-tuning so I can stop messing with it. I already had the opportunity to use the Alien app to pause wifi on my kid's computer yesterday when he was misbehaving, so thumbs up.
Oh, I also made a config back up before I did any of this, and a couple along the way. After I had it in a usable state, I power cycled the ER-4 to check for the lost bridge you mentioned on yours and I don't seem to have that problem.
Great to hear you have it working (mostly).
I also got the error about assigning eth1 to DHCP. But after a bit of messing around I was able to get it to work. I believe I assigned DHCP to WAN port prior to adding it to the Br0, then once I pulled my public IP address I added it to Br0.
I believe this "loop hole" making it DHCP and then adding it to the bridge is what causes the eth WAN port to disappear on power cycle, which is why I mentioned to make a backup prior to adding it to Br0, so if you needed to restore then you only have 1 thing to do once you install the backup.
Did you just make the bridge DHCP? That works also.
(You also did all the hardware offloads for the ER-4 too right?)
My thought process was "IF" the ER has a firewall it would "probably" be assigned to traffic on the WAN port (there's a menu somewhere you tell it what the WAN port is), so by having DHCP on the WAN port (rather than Br0) "may" offer better protection. But I don't know all the ins and outs of the ER though, so I'm not 100% sure and there's no one to really ask because it's not really something Ubiquiti or Amplifi have done. That was just what I was thinking.
Good to hear you got QoS working. It's nice seeing the A+s
Made a big difference for my network when my NAS would offsite backup, doesn't cripple my internet connection anymore.
As for your lingering issues.
As mentioned I don't know all the ins and outs of the ER, so I'm not 100% sure about the security aspect of it. Which is why I made the WAN port DHCP, hoping that would put the ER behind it's own firewall and then everything else is behind the Amplifi's firewall. And I don't have anything from behind my Amplifi plugged into the ER (except that RJ45 jumper on my rack I have that I take on and off when I need to access the ER by an Amplifi IP address).
I can access my NAS externally by port and also my server through RDP. I can tell you that China and Russia (as per the IP addresses) found my NAS ports and brute force attacked it continually for a couple weeks before I saw it, so I changed the ports and now have notifications that tell me failed logins.
They also found my server's RDP port and started brute force attacking it also, Bitdefender notified me of that. So had to change that port also.
So they have tried to hack me and never got through, so there doesn't "seem" to be any security issues, however I'm sure if someone really wanted in they'd find a way no mater what you are using, hence the reason for daily offsite backups.
I don't know much about nat/hairpinning but if it's a loopback issue the Amplifi (at least the HD does) has a setting called "Enable STP - Spanning Tree Protocol - a protection against Ethernet loops". You can turn it on from the Amplifi web interface. http://amplifi.lan/
Maybe that will help.
Yes it is a bit of a pain to keep connecting to eth0 for setup, which is why I used one of the other ER ports set it to DHCP and connected it to my switch BEHIND the Amplifi, now I can access the ER from a normal Amplifi IP address for all my testing and setup. BUT this is the cable/jumper I remove when I'm not using it because it's putting the ER behind my Amplifi firewall.
No idea about the Nest. I'd probably try turning off QoS and see if that makes a difference. It it does then you can try playing with some of the QoS settings to see if you can clean it up.
But odd that it works fine on an iPhone but not on Android.
I'm running iphones and I have 5 Ring cams and they all stream perfectly.
Good luck on the rest of the setup. Took me a bit to fine tune it.
Would be so much easier if Amplifi just added SQM and ethernet QoS to the HD and Alien, wouldn't have to figure out all these work arounds
Hi @Jason-Ellis - I am curious, what was your BufferBloat score when you previously removed the ER from the mix and tried the Alien's Latency QoS set to 50 Mbps on the upload (same limit ER is using) and ~850 Mbps on the download (you can't enable them independently)?
I don't know of anyone who hasn't been able to achieve an 'A' or better BufferBloat score on DSLReports after tuning the Alien's Latency QoS settings so didn't realize the ER Bridge solution still had some potential benefits
@Derek-Saville I can't recall what it was when I previously tried, but I think it was mostly C. I'm going to run a test for a few days, removed the ER-4 from the flow and just rolling with the Alien to see how it handles real world conditions and whether I notice anything detrimental. It's possible that I'm over optimizing based on data and wouldn't really notice an impact in normal usage.
Currently, with the ER-4 removed and the QoS on the Alien set to my ISP limits (1Gbps/50Mbps, I haven't done any tweaking), I am getting scattered results, one A, one C, mostly Bs. Of course, my whole family is at home today, so these tests aren't being run in a quiet network environment. I'll do the fornite test tonight. Just like LainB, fortnite latency when playing with my kids has been the primary driver of getting this wireless environment fixed over the past year or so and the easiest way to test real world performance.
If this works well enough, I'll consider retiring the ER-4 from the setup. I really like the router, but would like a simpler environment if possible. I'm concerned that my unfamiliarity with the bridge setup may open me up to vulnerabilities.
A few changes thus far.... Taking the ER-4 out of the mix has fixed my problem with my NAS custom domains, they're now acceptable on the network for wired devices. Unfortunately, Nest cams still freeze outside of the network. I borrowed my wife's phone and see that she really does not have this problem, so maybe it's a Nest android app problem.
Anyways, will update here when I decide to stick to the Alien or add the ER-4 back in.
Hi @Jason-Ellis - thanks for the comparison
For the Alien’s Latency QoS to provide a benefit we have found you need to manually reserve some bandwidth versus your real world ISP throughput
When I did the ER SQM a couple years ago the firmware at the the time would automatically reserve something like ~10% off of the values you entered (ISP speeds) but I have no idea how their code has evolved since
On the Alien if you enter your ISP speeds it won’t actually do anything unless your real world throughput is significantly higher
At the time for SQM we also had to set artificially lower limits to improve performance
My applications are latency sensitive so I sacrifice as much bandwidth as necessary to achieve the lowest ping times, but most people probably don’t like the trade off
You need to find those values experimentally by testing and then dial them back to what you are comfortable with
At one location i sacrifice 25% of my bandwidth to reduce ping times by 66% (BuffetBloat goes from ‘B’ to ‘A+’) during periods of high ISP congestion, but that makes the difference in being able to Teleport live HDTV on some streaming apps that don’t buffer (since they assume they are local and not being redirected around the world)
Since Alien doesn’t have any fine tuning control like SQM the trade off may not be worth it to meet your needs, but the feature does work to reduce latency (most of the time if the firmware doesn’t break it)
Jason Ellis last edited by Jason Ellis
Yeah, I went through a lot of the QoS/SQM finer details last year when I figured out that may be the problem I had with previous equipment (I went through a few different mesh networking kits before adding the ER-4 to eliminate bufferbloat). If the Alien-only setup looks promising, I'll try to fine-tune it to allow for some headroom, but I am not able to do that with a relatively traffic free environment while my wife and kids are here.
Regarding the issue accessing custom domains: I was wrong, not fixed. I don't know why it seemed to work at some point, but it still does not. I DID find out the problem though.
This seems to be something the devs are aware of, as referenced here: https://community.amplifi.com/topic/3733/hairpin-nat-loopback-not-working and discussed further here: https://community.amplifi.com/topic/4221/alien-firmware-3-5-0rc5/59?page=3
Basically, NAT loopback is not working for me because I have modem -> Alien -> 8 port switch (with NAS) -> 4 port switch (with my 2 PCs). I did as suggested in the post above and connected my 4 port switch to another LAN port on the Alien and all of a sudden I can access my custom domains. Hopefully, this is something that gets fixed in firmware soon, but I think I can live with it if the Alien otherwise performs well without the ER-4 assisting it.
SerLowenbrau last edited by
@LainB Thank you for this post. I recently went from 2 ISPs to just 1, so the edgerouter X was reduced to a underused switch. I wanted to keep the nice interface options for the Amplifi HD, but still put the X to use. I followed your instructions and this is working like a champ. Bufferbloat has gone from F to A, and I can easily pause my kids' internet at any time to get their attention.
Thanks again for taking the time to write this up.
@SerLowenbrau No problem at all! Great to hear you got it up and running.
My ER-12 at home and ER-X at work are still working great.
Don't forget to make a backup with the WAN port not in BR0. As mentioned in the above posts, if it looses power (cycles power) the WAN port is completely removed from the EdgeRouter and the only way to get it back is factory reset and setting it up all over again or installing the backup with the WAN port not in BR0 and then putting it back into BR0.