My blog has never been particularly homogeneous. This post is probably weirder than usual.
These days most people know me for work around Linux kernel safety and the Elisa Community. That’s the professional side. But like many engineers, I also have a homelab where I experiment, break things, and occasionally learn something useful while pretending it’s just procrastination.
This story comes from when I decided my network needed an upgrade.
And it unexpectedly reminded me why networking used to be my field in the first place.
The Pascal Incident
To explain why I even cared about this migration, we need to go back a bit.
I originally started as a Assembly-Pascal programmer.
In 1997 I had a small disagreement with my computer architecture professor about whether Pascal could be used for low-level work.
Naturally, the only rational response was to write an arcade machine emulator in Pascal.
This one: a Space Invaders emulator. Which I proudly demonstrated to the professor. I still remember that conversation.
Me: Hey professor, look what I built using Pascal. If used properly, you don’t actually need C.
To be fair, this was only partially true. I did use the Pascal compiler to build the executable, but most of the low-level work ended up written as large blocks of inline assembly inside the Pascal source. Anyway, I showed it to the professor.
The professor watched the screen for a moment.
Professor: Well yes. Nice game.
That was not the reaction I expected.
Me: Look, I didn’t make this game. Someone in Japan did… back in 1978.
At that point the professor looked mildly annoyed.
Professor: Then what exactly are you showing me today?
Which honestly left me unarmed.
I would have liked to answer: the emulator, you idiot… But I didn’t. For the record it neither improved my grade nor impressed the professor. But the experiment left me with a fascination for understanding systems by emulating them. Which later led to another emulator.
Instead of arcade hardware, I tried to emulate a Cisco router.
That project became my computer engineering master’s thesis: a Cisco 2500 router emulator, again written in Pascal… the last time before burying it forever.
Reverse engineering parts of the router firmware to understand the hardware architecture forced me to dive into routing logic…
That was basically my accidental entry point into network engineering. So yes, my networking career technically started with a router emulator written in Pascal. Which probably explains a few things.
Back to the Homelab
Fast forward a few decades.
The task that triggered this story looked simple enough:
Replace an old Cisco 3560, sitting in my lab and running almost continuously for 15 years with something newer… a Ruckus ICX-7150, and finally enter the gigabit era.
Which mostly meant replacing one relic with a slightly younger relic. The 3560 was already ancient. The ICX-7150 is not ancient yet… but it’s already end-of-sale, so it’s clearly on the right trajectory.
At first glance the CLI felt familiar. Cisco and Ruckus commands look similar enough that you instinctively think:
“Ah yes. I know exactly what this does.”
That confidence usually lasts about five minutes. After that you discover that while the commands look similar, the behavior is not. Part of the reason is historical.
This network used to be entirely Cisco. Phones, switches, everything. In that environment life was easy. Phones learned their VLAN via CDP, everything powered correctly, and nobody asked too many questions.
The moment I started replacing pieces, I discovered which parts of that convenience were standards, and which parts were simply Cisco assumptions.
Some of that I expected. Some of it I discovered the hard way.
The Troubleshooting Story
The setup was simple:
- Cisco 7940 phones
- SCCP firmware
- Asterisk
- configuration via TFTP
- DHCP instead of CDP
The goal was to run the phones in a vendor-neutral network.
Initially everything worked. Phones booted and registered. Then they started doing something strange. They would run normally for a while…
disconnect… then reconnect.
Phone logs showed events like:
Code:8103
Packet captures showed:
- SCCP keepalive retransmissions
- TCP resets
- reconnect cycles
Then another symptom appeared: occasionally the phone did not respond to ARP requests. That pointed suspicion directly at the phone. Old 79xx firmware is known to be… quirky. So the investigation went there first.
The Firmware Detour
The phones were running:
P00308000100
Which triggered the usual Cisco firmware archaeology. Finding firmware for older Cisco devices is a bit like looking for historical artifacts: you know it exists, you’re just not sure where.
Eventually a newer firmware image was located and installed.
The upgrade worked… And the problem remained.
So the firmware theory collapsed.
At this point the investigation had already consumed a fair amount of time… and it was still focused on the phone.
The Turning Point
The turning point came after a longer packet capture session. Something about the ARP traffic looked wrong. So the next step was obvious: check what the switch actually saw. A mirror session was configured on the ICX and the phone port was captured.
The mirrored traffic showed:
- unicast traffic on VLAN 2
- broadcast traffic from the phone
But something was missing.
The ARP broadcast from Asterisk never appeared. To confirm this, traffic was captured in several places:
- the ICX mirror
- a VLAN-2 host
- a Cisco mirror upstream
The result was consistent. The ARP broadcast existed on the Cisco side, but never reached the ICX access port. At that moment the investigation pivoted. This was no longer a phone problem.
It was a switch problem.
Finding the Culprit
To confirm the path, the Cisco 3560 uplink was mirrored. The ARP request was clearly visible leaving the Cisco trunk:
Who has 10.1.5.57? Tell 10.1.5.1
But the broadcast never appeared on the ICX.
The packet was disappearing between the switches.
Attention turned to the ICX uplink configuration:
vlan 2 tagged ethernet 1/2/2 ! interface ethernet 1/2/2 dual-mode
VLAN configuration looked correct: VLAN 2 tagged 1/2/2
Various things were tested:
- VLAN membership
- DHCP snooping
- spanning tree
- broadcast filtering
Nothing changed.
The Fix
Eventually attention returned to one small detail on the ICX uplink:
interface ethernet 1/2/2 dual-mode
In FastIron (Brocade/Ruckus) this is roughly equivalent to the native VLAN on a Cisco trunk.
And in fact, if you come from a Ruckus-native background, that line probably would not raise any alarms at all.
The only reason it caught my attention is precisely because I wasn’t thinking like a Ruckus engineer.
So, mostly out of prejudice rather than conviction, I tried removing it.
Immediately:
- ARP broadcasts appeared
- VLAN-2 hosts received them
- the phone stopped disconnecting
The note I wrote at the time was simply:
was that! no dual-mode on 1/2/2 solved the problem.
And that closed the investigation.
Which strongly suggests the real root cause was not the configuration itself, but likely a firmware bug in the switch triggered by that combination of trunking behavior.
NOTE: On Brocade/Ruckus FastIron
switches, the dual-mode command is used to allow a port
that carries tagged VLANs to also accept and transmit
untagged traffic. Unlike Cisco trunks, which always
include a native VLAN by default, a FastIron port with tagged VLANs
normally sends only tagged frames unless
dual-mode is enabled. The command itself takes no
arguments and has remained syntactically stable across FastIron
versions; it simply enables the ability for the port to carry
untagged frames. The actual native (PVID) VLAN is
determined separately by the port’s untagged VLAN
membership, which can be changed under the interface with
vlan-config move untagged vlan <vid>. For engineers
familiar with Cisco, the combination of dual-mode plus the
port’s untagged VLAN is functionally equivalent to configuring a
native VLAN on a trunk.
A Note About AI
One final note.
I have been skeptical about AI tools for quite a while and avoided using them in my workflow.
This debugging session, and a few others, are changing my mind a bit.
The AI suggested ICX commands I didn’t know, reminded me of Cisco syntax I had not used in years, and helped narrow the search space.
Without that assistance I would probably have spent days digging through documentation.
Instead the whole problem unraveled in one night.
Which, for a homelab experiment, is an excellent outcome; and significantly better than spending a weekend reading switch manuals.
