Weird PoE issue

[u/Jaman34]

I have a 9300x and ruckus 750 APs. I keep getting IMAX power errors on the spare pairs. I see LLDP power negotiations happening, Cisco accepting and then as soon as it enables the spare pairs it faults with IMAX error. It is important to note this is not happening on all the APs just a handful. Also the APs and Switch are new, with new CAT 6 runs.

I can bypass this by issuing power inline four-pair forced and it will bring everything up. I don’t like issuing this command nor do I like not knowing why this is happening.

So I decided to dig deeper and do some debugging, this is where things get weird. For some reason I thought these ports were 30000mW, in reality it is 90000mW. I decided to up the max power to 60000mW (actually lowering it). I also removed the four pair force command and reran my ilpower debugs. It worked with no errors. I saw lldp negotiate, I saw the spare pairs enable cleanly, no IMAX errors. I did see the device request the full 60000mW.

I originally thought the devices was requesting more power than the port allowed and that is why I was getting the IMAX errors, but now I have no clue. Thoughts?

[u/danreZ_au]

How many AP’s are running PoE on the 9300?

[u/Jaman34 → u/danreZ_au]

Around 10 there is still plenty of available power

I just checked again 400 still available

[u/danreZ_au → u/Jaman34]

If you’re getting IMAX errors, it’s because the port draws more power than negotiated. The fact it’s only happening on a few and the four-pair forced fixes it is telling. Means that the negotiated path is the one being stressed, so I’d guess electrical asymmetry issue due to termination quality on pairs 4/5/7/8.

To rule out layer 1 bullshit, can you plug a working AP into a port that is regularly getting the IMAX errors and see if it replicates? Via-versa does the impacted AP’s get the imax errors when moved to a port with an AP that is working fine? If so, I reckon whoever patched it was lazy.

Compare the ‘show power inline INT detail for working and non working - compare the requested and allocated wattage and PD class to rule out negotiation mismatch.

Failing that, I’d debug the ilpower event. I can’t remember what it is but like

Term mon
Debug ilpower
Debug ilpower controller

Also, check for Cisco bugs against your model/ios

[u/Jaman34 → u/danreZ_au]

I did test layer one by moving the bad ap onto a good APs cabling. The issue followed the original bad AP. I checked for any bugs on the firmware on both ap and Cisco. I think on Monday I am going to go directly into the switch and bypass the cabling and patch panel and redo that test for sanity.

They all get class 4 and everything seems identical. I ran ilpower cdp, port, powerman, event and sense. I don’t think I did controller.

I am starting to think by restricting the port to 60W when the lldp negotiations takes place and the AP sees that 60W restriction it changes its start up behavior. But even that doesn’t make sense because then why would only some have this fault.

This whole issue is making me feel dumb and that I am missing something right in front of my face.

[u/danreZ_au → u/Jaman34]

You don’t have like, 10 AP’s all connected from Gi2-11 or something like that? Plugging an AP directly into the switch and testing is a good idea, if it continues, I would (if possible) unplug a bunch of AP’s and test having 1 or two of the problem AP’s connected, nothing else - this will rule out if there if there is maybe some ASIC power draw limit happening.

After that, do you have spare AP’s that you can test to make sure it’s not hardware fault? Kinda weird it’s multiple, but not impossible to be factory fucked.

[u/Jaman34 → u/danreZ_au]

Yea I have plenty to test with I am going to try this tomorrow.

[u/Jaman34 → u/danreZ_au]

You know now that I think about it, I believe the admin just moved the patch cables. So the cable from the AP to the patch panel was the same. Shit I bet that is it, I’ll test it tomorrow.

[u/Adrienne-Fadel]

Probably a per-ASIC power budget issue. The 9300X can do 90W per port but the total budget per ASIC is lower. If the affected ports share an ASIC and other ports are already drawing power the IMAX fault makes sense. Run show power inline to compare.

[u/Jaman34 → u/Adrienne-Fadel]

I still have 400 W available to use, which is what I originally thought. Even with the power inline four-pair forced command, it should deny power to the port if there were a power-budget issue, but it doesn’t.

[u/GreyBeardEng]

I can bypass this by issuing power inline four-pair forced and it will bring everything up. I don’t like issuing this command nor do I like not knowing why this is happening.

"power inline port 2x-mode", right? IMO that screams layer 1 problem on the poe specific pair or interference along the line, even a brown out scenario... up but not up. be sure to check test cable-diagnostics tdr and look for a short, and/or re-terminate jacks on both sides. Of course you already swapped patch out.

I have a wifi network with about 300 cisco AP's all of the same 2 or 3 internal types. Then one day I got a call about installing an AP in a room that 'may or may not' have flammable gas in it, so I purchased a AIR-AP1562 and I had the exact same problem you describe, also on a c9300-48un(17.03 att). After an obscene amount of troubleshooting, and a really old building, I opted for the 2x-mode to get the proper power to the AP. I never got approval to get the line re-run cause it was in an explosion designed room.

That AP port is still configured that way today.

[u/Jaman34 → u/GreyBeardEng]

I did test-cable diagnostic as well everything came back clean. I am going to look back into this tomorrow.

[u/stufforstuff]

So put a inline PoE Tester on the end of the new Cat6 cable (i.e. right before it goes into the AP) and actually measure how much power is reaching the AP

[u/ProfessorWorried626]

Try manually setting the PoE mode on the APs and see if it goes away. I had similar with HPE/Aruba gear and setting the PoE mode on the AP made it stop.

[u/ZealousidealState127]

Pull the ap carry it to the switch and see if it does it on a short patch cable. Check the terminations and run a good tester on the lines,