Saturday, October 17, 2026

The Battle Chess Disk That Was Full, Empty, and Absolutely Fine

There is a particular kind of debugging problem that only old computers can provide.

The software is nearly forty years old. The media is magnetic. The operating system lives partly in ROM. The documentation assumes you remember what a key is. And somewhere along the way somebody installed an antivirus written when “computer virus” was still a sufficiently new concept that people were not always entirely sure how one moved from machine to machine.

This investigation started with an old Amiga copy of Battle Chess.

Which is appropriate, because Battle Chess gave software development one of its better debugging metaphors: the duck.

Battle Chess and the duck

According to a piece of Interplay lore, the artist responsible for the queen’s animations knew that the producer had a tendency to request some change during every review.

So he prepared the queen exactly as he wanted her, and added one deliberately unnecessary feature.

A pet duck.

The duck was carefully animated alongside the queen, but kept separate from the actual animation. When the producer reviewed the work, everything important survived untouched. There was just one requested change:

lose the duck.

The trick was not that the duck hid a defect. Quite the opposite: it provided an obvious, harmless target for scrutiny so that the thing the artist actually cared about could pass through review unchanged.

Years later, Battle Chess provided me with another duck.

This one lived in its boot block.

And I fell for it completely.

Enter Greaseweazle

The whole investigation started because I was preserving old Amiga floppies.

For that, I was using a Greaseweazle.

This is particularly interesting with Amiga disks because the Amiga and the PC approached floppy disks almost from opposite directions.

A traditional PC floppy controller is heavily format-aware. Software normally asks the controller for a particular cylinder, head and sector, and the controller takes care of finding the sector, decoding the MFM data, checking the CRC and returning the contents.

That works very well as long as the disk looks like something the controller understands.

If it does not, software has very little room to negotiate.

The Amiga floppy subsystem was deliberately much less opinionated.

The hardware can DMA the encoded track data into memory and leave interpretation to software. Normal AmigaDOS tracks have a defined format, of course, but software is not limited to that format.

That freedom is one reason Amiga software accumulated such an impressive collection of custom track layouts, unusual sync words, protections and other creative abuses of magnetic media.

Very approximately:

PC: software: give me sector 7 controller: certainly I know what a sector is

while the Amiga is closer to:

hardware: here is the encoded track data software: excellent I'll decide what it means

Greaseweazle goes lower still for preservation purposes.

Instead of returning decoded sectors, it records the timing of the magnetic flux transitions coming from the drive.

For a normal AmigaDOS disk, it can decode directly into an ADF:

gw read --format=amiga.amigados disk.adf

For preservation, however, I also like keeping a flux-level image:

gw read --revs 5 disk.scp

That records several revolutions of the disk.

Those transitions can later be decoded again, compared between revolutions, interpreted by other software, or written back to a floppy.

So, roughly:

PC floppy controller: sectors Amiga floppy subsystem: encoded track data Greaseweazle: flux-transition timing

For old-media archaeology, that lowest layer is extremely useful.

Battle Chess, however, did not initially look like a magnetic-media problem.

The disk read cleanly.

The filesystem was there.

The strange part came afterward.

The original mystery

Opening the Battle Chess disk window in Workbench produced:

Guru Meditation #87000008

The timing was interesting.

Workbench opened the window.

The icons appeared.

Then, just before it seemed to finish opening the drawer:

Guru.

My first thought was damaged media.

So I found another Battle Chess image online.

It crashed in exactly the same way.

That was the first major clue.

And the beginning of several excellent wrong ideas.

Comparing the two disks

The next tool entering the story was amitools, particularly xdftool.

If Greaseweazle lets me ask what was physically recorded on the floppy, xdftool lets me ask what the Amiga filesystem structures say about it.

It understands OFS and FFS volumes and can inspect directories, files, root blocks, allocation bitmaps and other filesystem structures directly from an image.

For example:

xdftool battle_chess.adf list / detail

showed:

Battlechess BattleChess.info Chesstodh0 ChessToDH0.info Disk.info EndCLI Icon.Library LoadWB s System-Configuration

Nothing obviously missing.

Then I compared my image against the one found online.

They were remarkably similar.

The actual game and filesystem contents were essentially the same. Apart from a few administrative differences in the root block, the significant difference was at the beginning of the disk.

My copy contained a custom boot block:

This disk was installed with TRISTAR-Viruskiller V1.1 Written 1988 by Transformer (Disk is protected against the SCA-Virus)

The online image did not.

There were also a few differences in the root-block metadata around block 880.

But the content of the disk was, for practical purposes, the same.

That made the boot block extremely difficult to ignore.

The virus hypothesis — and my own bias

The antivirus explanation was especially attractive to me because I grew up in exactly that period.

Computer viruses in the late 1980s and early 1990s occupied a peculiar place in popular computing culture.

They were very real.

But the understanding of how they propagated was not always equally real.

I still remember a computer-science teacher in high school, around 1992, asking us to unplug the PCs from the mains because a virus had been discovered and there was concern that it might propagate through the electrical wiring.

That memory matters because it explains my own bias here.

When I found:

TRISTAR-Viruskiller SCA-Virus

inside the boot block of a disk producing a mysterious Guru, my brain immediately assembled a satisfying story:

The disk had a virus.
Somebody tried to clean it.
Something survived.
Now that something is upsetting the system.

It was plausible.

It fit the period.

And it was dramatically better than “some filesystem bit is wrong.”

That should probably have been a warning.

All the boot block actually proved was:

Someone, at some point, was concerned enough about viruses to install TRISTAR-Viruskiller.

It did not prove that this particular disk had been infected.

It did not prove what the owner understood about boot-block viruses.

And it certainly did not prove that the present failure had anything to do with malware.

Still, it was difficult to resist.

It was our first duck.

Another suspicious character: chesstodh0

The disk startup sequence was very short:

chesstodh0 LoadWB endcli > nil:

LoadWB and EndCLI are ordinary enough.

chesstodh0 was more interesting.

Looking into the executable revealed:

UNIT A && World of Wonder presents NR. 47: BATTLECHESS Thanx to Blackie for the original ...

It also contained references to graphics.library, dos.library and icon.library.

A crack-group executable.

Running before Workbench.

On a disk that crashes Workbench.

This was an exceptionally good duck.

Perhaps it installed something resident.

Perhaps it changed DOS state.

Perhaps it left something behind that later failed.

So I removed it from startup-sequence.

Its messages disappeared.

The crash did not.

Remove the whole Battle Chess boot environment

There was an even cleaner way to eliminate that branch.

I booted a known Workbench 1.3.2 disk instead.

Once Workbench was running, I removed that disk, inserted Battle Chess and opened its drawer.

Now none of the Battle Chess boot-time software had run:

no Battle Chess boot block no TRISTAR code no chesstodh0 no Battle Chess startup-sequence no Battle Chess LoadWB no Battle Chess EndCLI

And yet:

87000008

So whatever the problem was, simply mounting and opening the Battle Chess disk was enough.

The surprisingly important write-protect tab

Then came the observation that changed the investigation.

Make the ADF write-protected:

works.

Make exactly the same image writable:

Guru.

That was much more interesting than another ROM revision or another executable.

Nothing about the visible contents had changed.

Workbench could clearly read the disk.

It could find the directory.

It could load the .info files.

It could draw the icons.

But whether the disk was allowed to be written determined whether the system survived opening the drawer.

At this point there were two facts:

same disk same Workbench operation write-protected -> works writable -> crashes

The obvious next question was no longer “what strange thing is the emulator doing?”

It was:

Does the real Amiga do the same thing?

Ask the Amiga

My first physical-hardware test had been misleading for a wonderfully mundane reason:

the disk had been write-protected.

So I returned to the physical Amiga.

Same disk.

Same operation.

But this time I opened the write-protect tab.

Workbench opened the drawer.

The icons appeared.

Then:

Guru Meditation.

Exactly the same behaviour.

That result was decisive.

There was no longer an emulator versus hardware problem to explain.

The real machine and the emulator agreed.

More importantly, they agreed on the one variable that mattered:

write-protected -> works writable -> fails

Only now did it make sense to stop looking at execution environment and start asking what the Amiga itself was complaining about.

What does 87000008 mean?

The Guru was not a random CPU exception.

87000008 is a DOS alert corresponding to:

key already free

Now the investigation had moved decisively into filesystem territory.

Classic Amiga filesystems use block numbers as keys.

They also maintain an allocation bitmap recording which blocks are occupied and which are available.

So “key already free” can be read fairly literally:

DOS is trying to release or manipulate a block which its allocation data already says is free.

That was a remarkably specific clue.

Especially given the previous experiment.

A read-only disk worked.

A writable disk did not.

Time to ask the filesystem what it believed.

Asking the filesystem

The volume is an Amiga OFS filesystem: DOS\0.

An Amiga double-density floppy contains 1760 blocks of 512 bytes.

The root block is conventionally placed in the middle:

1760 / 2 = 880

So block 880 is the volume root.

Among other things, the filesystem root leads us to the allocation bitmap: the structure responsible for recording which disk blocks are free.

xdftool can look at the same filesystem from two useful directions.

First, which blocks are actually referenced by files and directories?

$ xdftool battle_chess.adf bitmap node / all entries brief 0: .........D. ........... 880: V.......... ........... 1408: ........... .....HdHdHd 1430: dHdEEdddddd ddddddddddd 1452: ddddddddddd ddddddddddd ...

Here:

V = volume/root D = directory H = file header d = file data E = extension block

Clearly the disk contains a filesystem.

Clearly many blocks are occupied by files.

Then I asked the allocation bitmap which blocks were free:

$ xdftool battle_chess.adf bitmap free brief 0: ..FFFFFFFFF FFFFFFFFFFF 22: FFFFFFFFFFF FFFFFFFFFFF 44: FFFFFFFFFFF FFFFFFFFFFF ... 880: FFFFFFFFFFF FFFFFFFFFFF ... 1430: FFFFFFFFFFF FFFFFFFFFFF ...

Almost the whole disk was marked free.

Including blocks that the filesystem tree was actively using.

There it was.

A full disk convinced that it was empty

The filesystem said:

these blocks contain my files

The allocation bitmap said:

never heard of them

That contradiction explains the strange behaviour remarkably well.

Reading the disk did not require repairing the inconsistency.

The directory structure existed.

The file headers pointed to the correct data.

The icon files were readable.

Workbench could populate the window.

As long as the disk remained read-only, nobody had to reconcile the filesystem tree with the allocator’s version of reality.

But once writes were permitted, AmigaDOS eventually entered a path involving allocation bookkeeping.

At that point it encountered a block that the filesystem considered occupied while the bitmap already called it free.

And DOS complained, rather accurately:

KEY ALREADY FREE

Now all the experiments lined up:

Amiberry + write-protected disk -> works Amiberry + writable disk -> Guru real Amiga + write-protected disk -> works real Amiga + writable disk -> Guru

The machine did exactly what its metadata told it to do.

The metadata simply happened to contradict itself.

Greaseweazle and amitools: two layers of archaeology

One thing I particularly like about this case is how well the tools complemented each other.

Greaseweazle answers questions about the physical recording:

Can I read the disk reliably? What flux transitions are actually present? Can I preserve several revolutions? Can I return to the raw evidence later?

amitools answers questions about the filesystem constructed from that evidence:

What does AmigaDOS think this block is? Which blocks belong to a file? Where is the root block? Which blocks does the allocation bitmap call free?

Those are very different layers.

And keeping them separate is useful.

In this case the magnetic media was readable.

The logical file structures were readable.

The allocation metadata was wrong.

Greaseweazle gave confidence that the image represented the physical disk.

xdftool made it possible to interrogate what the filesystem itself believed.

Together they turned a thirty-something-year-old floppy into something surprisingly close to a modern forensic exercise.

What broke the bitmap?

That part remains archaeology.

Possible explanations include:

  • a badly mastered crack release;
  • a copying tool that reproduced the filesystem contents without correctly rebuilding the allocation bitmap;
  • an interrupted filesystem operation;
  • virus activity;
  • antivirus activity;
  • deliberate construction of a disk expected always to remain write-protected;
  • or some forgotten utility doing something perfectly reasonable by the standards of 1989.

The comparison with the online image is particularly interesting here.

My physical image and the online copy were almost identical in their actual filesystem contents.

The conspicuous difference was the boot block: mine contained the TRISTAR Viruskiller installation.

Yet both copies exhibited the same writable-disk failure.

That makes the antivirus boot block very poor evidence for the cause of the problem.

But excellent evidence for how easily an investigation can be distracted.

It was the duck.

Looks great. Lose the duck.

This investigation had no shortage of attractive suspects:

  • old magnetic media,
  • an emulator,
  • different Kickstart revisions,
  • cracked software,
  • a strange startup utility,
  • an antivirus boot block,
  • the SCA virus,
  • possible resident malware.

Some were reasonable hypotheses.

Some were historically fascinating.

One of them literally contained the word:

VIRUS

And none was responsible for the crash.

The actual problem was much quieter:

filesystem tree: these blocks contain my files allocation bitmap: those blocks are free

For decades that contradiction could remain harmless as long as the disk stayed write-protected.

Nobody asked the filesystem to reconcile the two stories.

The moment we did, AmigaDOS answered with surprising precision:

KEY ALREADY FREE

Battle Chess had already provided the appropriate debugging lesson.

When something conspicuous is absorbing all the attention:

Looks great. Lose the duck. Check the bitmap.

Thursday, June 18, 2026

One Terminal to Rule Them All

Or: how an ADM-3A became the cockpit of my homelab

Usually, when I write here, I try to talk about serious technical things.

  • Kernel memory.
  • Build systems.
  • Functional safety.

The kind of topics where a small mistake can become a paper, a patch, or a very long afternoon.

This time, I want to write about something equally important … to me, at least:

  • old electronic stuff.

Not “old” as in “deprecated by a vendor who would like me to buy the new cloud-enabled subscription version.”

I mean properly old… Old enough to have a CRT.

Old enough that when it displays a prompt, it does not feel like a terminal emulator. It feels like the computer is talking to you directly.

The arrival of the dumb terminal

At some point, I bought myself a Lear Siegler ADM-3A terminal. Inside I found a label that says 1977, meaning we are almost the same age.

This object is not just “an old terminal”, it is one of those objects that quietly left fingerprints all over computing history.

For example, if your shell abbreviates $HOME as ~, the ADM-3A keyboard is part of that story: the ~ character lived on the key labeled HOME.

And if your fingers know that h, j, k, and l are movement keys in vi, that is not random ancient wizardry either. On the ADM-3A keyboard, those letters had cursor symbols printed on them. Bill Joy developed vi using this kind of terminal, and apparently the muscle memory never left the building.

So yes, technically it is a dumb terminal… But “dumb” feels unfair.

It is more like a historical device with very few opinions and excellent posture.

  • A chunky case.
  • A real keyboard.
  • A screen that does not care about your GPU.
  • No tabs.
  • No browser.
  • No notifications.
  • No “your session has expired.”

Just characters… beautiful, honest, monochrome characters and a hardware keyboard where you can even find a key labeled HERE IS, close to RETURN, which sounds like the terminal is politely introducing itself at a party.

And that is almost what it did. With the optional answer-back feature, the terminal could send a predefined identification string, either when asked by the host or when the operator pressed the key.

A tiny serial-age “hello, it’s me” fingerprint.

The ADM-3A had manners.

The ADM-3A is often called a dumb terminal, but I think that is unfair. It is not dumb! It is focused. It has one job:

“Give me bytes, and I will show them.”

First contact with the homelab

Of course, after buying an old serial terminal, the obvious next question was:

“What should I connect it to?”

The boring answer would have been: nothing. Put it on a shelf. Take pictures. Occasionally say, “yes, it still works,” like people do with classic cars that never leave the garage.

Not what I intended for it: a terminal wants a computer.

So I connected it to my VM host machine, and it worked.

There is something very satisfying about controlling a modern Linux system through a terminal from another era. It feels like time travel, except instead of meeting dinosaurs, you meet systemd, sometimes even scarier.

It feels weird and cute at the same time to have a machine from the past controlling a machine running virtual machines. But then reality arrived, wearing the traditional hat of all homelab problems:

“Nice. But you have more than one machine.”

The problem with having “just one more box”

My VM host was not the only machine in the lab with a serial console; there was also a network switch… and another server… if we want to call it such. It is my home PBX, running on an Atom machine. So it is technically a server, but also spiritually closer to an appliance that refuses to die.

The point is: I had more serial devices than terminals, and I did not want to keep moving cables around.

Moving serial cables by hand is fun the first three times. After that, well, I leave the rest to the reader.

What I was thinking was something like an old KVM switch… but for serial. One terminal, multiple machines and a magic key sequence to switch between them.

The old KVM idea, but with serial

Old KVM switches had a very nice property: they allowed you to keep one keyboard, one video output, and one mouse, and move between machines. How to do this was not always straightforward: sometimes with a button, sometimes with a key sequence and sometimes with a ritual known only to the previous owner.

I wanted the same experience, in serial form but perhaps using a more deterministic way, the key sequence, for example. So the ADM-3A would be the keyboard and screen and behind it, a small box would route traffic to one of several serial ports so that the active machine would receive what I type and the terminal would display what the active machine says.

It sounds simple.

It is because of that that I started the project just after buying the ADM-3A and I finished it only now.

Enter the small box

For the hardware side, I used a tiny FriendlyElec ZeroPi.

It is one of those small ARM boards that exists in the dangerous region between “cheap enough to use casually” and “capable enough to become infrastructure by accident.”

Mine is connected to a multi-port PL2303 USB serial adapter.

My past in the telephony appliances is payback, at last, and I have a good plan to reinvent an old telephony switch but for data instead of voice.

So, ladies and gentlemen, we are about to reinvent the wheel.

Writing the little tunnel

The software became serial_tunnel.

It is a small utility that connects one terminal side to multiple remote serial devices. The terminal talks to the selected device. The selected device talks back. If I press the configured escape sequence followed by a number, the active device changes.

Plain and oldish, but for this kind of project, boring is good. Boring means I can trust it at 1 a.m. when the switch is unhappy and the only thing I want is a console, not a distributed systems lecture.

The configuration defines the serial ports and the tunnels. Each remote has a name, a device path, a speed, and the usual serial parameters. The tunnel then says: these are the devices, this is the terminal, and this is the escape sequence.

The beauty of limits

Using an ADM-3A is a wonderful reminder that computers were not always chatty. A machine from that era did not print five screens of logs just to say “hello.”

It did not emit JSON, YAML, warnings, hints, stack traces, deprecation notices, and one cheerful message from a package manager telling you that funding is available.

It was more like:

login:

That was enough.

Modern machines are different, have feelings, and they express those feelings through logs… Tons of logs.

Still, the ADM-3A provides only 80 columns and 24 lines, beautifully retrĂ², elegant if you wish, but also disappointing if you consider modern system verbosity. In modern system metrics that 80x24 translates to approximately three boot messages and half a warning, fine if everything is working, very annoying if anything goes wrong.

So the first time I connected it to a Linux machine and watched output fly by, I had a thought:

“This is charming, but I have no idea what just happened.”

screen or tmux can help there”…

The heresy: adding SSH

This is where I committed a small act of retrocomputing heresy.

I added an SSH interface, not to replace the ADM-3A, that would be rude and the ADM-3A is the star of the show, but I wanted an SSH terminal that mirrors the same console hub.

Why?

Because sometimes the old terminal is perfect.

And sometimes I need to scroll back, copy a log line, or look at the same session from a laptop without standing in front of the rack like a 1970s operator waiting for the mainframe to confess.

So serial_tunnel also has an optional SSH terminal interface.

The idea is: the physical terminal and the SSH client share the same selected console. The ADM-3A is still the star of the lab, but the SSH side can observe and, when needed, interact using the same switching mechanism.

This gives me the best of both worlds:

  • the tactile joy of the old terminal,
  • the practical convenience of a modern SSH client,
  • and the emotional conflict of knowing I added network access to a project whose whole charm is a serial cable.

Technology is compromise.

Sometimes compromise listens on port 2222. If you happen to have a terminal, a homelab, and what to use them together, check this out.

So now the dumb terminal is in charge of the smart machines. Which, frankly, feels like a healthy correction.

If you liked the story, please check out the code here

Thursday, May 21, 2026

When “Probably Safe” Is Not Safe Enough

If you ever worked with Linux, you probably heard something like:

“User space and kernel space are isolated.”

Which is true. And also… not entirely comforting.

Because, as Murphy politely reminds us:

“Anything that can go wrong, will go wrong.”

At the ELISA LFSCS working group , we try to look at Linux from a functional safety perspective. That means we are not satisfied with “it usually works”… we want to understand what could go wrong.

From Munich: A Suspicious Idea

During the Munich 2025 session, we discussed what we called the linear mapping threat.

The idea is simple:

  • Linux keeps a linear mapping of physical memory
  • This mapping includes all RAM
  • Kernel objects and user pages come from the same pool

So in theory:

A kernel object page could sit right next to a user page.

Not a fault… Not a bug… Just… how memory is laid out.

And that alone raises a question.

Because if a kernel bug causes an overflow, adjacency matters much more than isolation:

  • overflow → likely direction → next page
  • underflow → possible, but statistically less common

So the interesting case becomes:

kernel page → next → user page

At that point, this was still a hypothesis.

A reasonable one… But still a hypothesis.

From Hypothesis to Observation

So we asked:

Can we observe this situation?

Not in theory. Not in diagrams. But in actual memory.

To be clear, not the fault happening… just the fact that Kernel objects pages and userspace pages can sit one next to the other.

The Tool (a.k.a. “Let’s Peek at PFNs”)

Initially, the plan was simple:

“Let’s write a kernel module and scan the linear mapping.”

Because… that’s what kernel people do. (Maslow’s Hammer: If the only tool you have is a hammer, you tend to see every problem as a nail)

Then came the slightly embarrassing realization:

“Wait… I can already see this from userspace.”

Using /proc/kpageflags.

And suddenly:

  • no kernel module needed
  • no patching
  • no special hooks

Just a userspace tool reading kernel-exposed data.

Small Technical Box: /proc/kpageflags

/proc/kpageflags exposes metadata for each physical page frame (PFN).

With it, you can:

  • iterate over physical memory
  • classify pages (anonymous, slab, huge, etc.)
  • build a PFN-level view of the system

In practice, the tool:

  • scans PFNs sequentially
  • assigns a category per page
  • renders a colored raster

Think of it as:

a “topographic map” of memory… where mountains are slab pages and plains are user memory

(Yes, the legend is still under construction.)

First Results: “This Looks… Too Clean”

Using the tool on a real system (x86), results were promising:

  • memory looked mixed
  • user and kernel pages were interleaved
  • adjacency was visible

Good.

But the LFSCS working group set its reference platform as aarch64 many moons ago and when if came to the ARM64 VM we’re targeting:

  • almost no user pages
  • almost no slab pages
  • everything looked… suspiciously clean

At this point, either:

  1. the tool was wrong
  2. the kernel was lying
  3. or the system was too nice

What We Learned (a.k.a. “The Plot Thickens”)

Huge Pages Were Hiding Reality

Disabling THP:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

suddenly changed everything.

Why?

Because:

  • userspace sees memory at page granularity (4KB)
  • linear mapping may represent the same memory as huge pages

So we had:

fine-grained user memory vs coarse-grained linear mapping view

Disabling THP aligned the two worlds and revealed the real distribution.

Slab Pages Don’t Like Attention

The SLAB flag is only set on compound head pages.

Meaning:

most kernel objects are there… just not explicitly labeled

So the absence of slab in the visualization was… misleading.

The System Was Too Clean

The VM had:

  • low workload
  • low fragmentation
  • limited kernel activity

In short:

not enough chaos to trigger interesting behavior

Forcing Reality (a.k.a. “Adding Some Chaos”)

One missing piece was understanding that:

a real system naturally creates fragmentation an idle virtual machine… usually does not

So to observe the phenomenon in a controlled environment, we needed to give to the kernel an helping foot.

The approach was to introduce controlled memory pressure and fragmentation using a small kernel-side helper.

The idea is simple:

  • allocate many temporary objects
  • insert a few persistent objects in between
  • free the temporary ones

This leaves behind memory pages that are:

  • partially used
  • not reclaimable
  • and therefore fragmented

This creates a layout where:

kernel allocations are spread across many partially filled pages

Which is exactly the condition needed to increase the chance of:

kernel page → next → user page adjacency

Think of memory like a block of cheese:

  • temporary allocations = drilling holes
  • persistent objects = small solid parts left behind

After removing the temporary allocations:

you don’t get a clean empty block you get a holey structure the kernel cannot easily compact

The Key Result

After:

  • fixing classification
  • aligning granularity
  • adding controlled fragmentation

We can say:

Kernel and user pages can be physically adjacent.

Not always. Not everywhere. But definitely possible.

And importantly:

this can happen naturally on a real system while in a VM it may require forcing conditions

Looking Closer (a.k.a. “What Exactly Is Sitting at the Edge?”)

At this point, we knew that:

kernel pages and user pages can be physically adjacent.

Which is already interesting.

But adjacency alone does not tell us how easy it is for trouble to cross the border.

If the last object in the kernel page ends hundreds of bytes before the page boundary, then an overflow would need to be fairly enthusiastic.

If instead the object ends exactly at the page boundary…

then even a tiny mistake can immediately step into the next page.

So the obvious next question was:

“What is actually sitting at the very end of the slab page?”

Teaching the Tool Some Slab Archaeology

The original direct_map_view PoC was good at classifying pages.

The updated version became slightly more nosy.

Given a user page, it now:

  • looks at the preceding PFN,
  • checks whether that PFN is slab-backed,
  • scans the page,
  • and identifies the kernel object closest to the end of the page.

Using kmem_dump_obj(), the probe can determine which slab cache owns the object and where the object begins.

In other words:

we stopped looking only at pages and started looking at what is actually living inside them.

A Typical Result

[ 66.010479] pfn_slab_probe: input_pfn=1078469 checked_pfn=1078468 flags=0xbfffe0000000200 [ 66.010862] pfn_slab_probe: checked PFN 1078468 is slab-backed, scanning [ffff0000c74c4000 - ffff0000c74c5000) [ 66.011652] slab kmalloc-512 start ffff0000c74c4e00 pointer offset 504 size 512 [ 66.013007] pfn_slab_probe: nearest candidate object at ffff0000c74c4ff8, offset 0xff8 into PFN 1078468

The probe did not find an object starting at 0xff8.

It found itself standing 8 bytes before the edge of the page, inside a kmalloc-512 object, at offset 504 from the beginning of that object.

So the math is:

0xff8 - 504 = 0xe00

The object starts at 0xe00 and, being 512 bytes long, ends at 0x1000.

Exactly at the page boundary.

Which is the memory-management equivalent of parking with the bumper touching the wall.

Turning Logs into Pictures

Because hexadecimal offsets are excellent at convincing computers but less effective with humans, a small script was added to convert the logs into SVG diagrams.

The result is a picture showing:

  • the slab page,
  • the last kernel object,
  • the page boundary,
  • and the user page immediately after it.

Sometimes, a drawing is worth several hundred printk()s.

From “Interesting Layout” to “Well… That Escalated Quickly”

At this point, we had demonstrated that:

  • a kernel object can sit at the end of a page,
  • the next physical page can belong to a user process.

The next question was unavoidable:

“What happens if the kernel writes past that boundary?”

Fortunately for science, and slightly less fortunately for the victim process, we now have a PoC for that too.

The Victim (a.k.a. “The Simplest Program We Could Get Away With”)

The userspace target is a tiny assembly program, available for both:

Its behavior is intentionally boring:

  1. print the address of a string,
  2. wait,
  3. print the string,
  4. exit.

That is all.

No threads. No libraries. No excuses.

If something changes, we know exactly who to blame.

Was it that simple?

The PoC script: poc.sh

Coordinates with the kernel module: umem_poke.c

The script does not do magic address translation in bash.

Which is probably good news for everyone.

It only coordinates the experiment: it starts the victim, gets the address printed by the process, and passes that VMA address to the module.

The module is the one doing the kernel-side work. It uses GUP to resolve the userspace address to the underlying page, computes the linear mapping address destination byte, and then performs the write.

After all that machinery, the corruption itself is still disappointingly simple:

*dst_target = act.value;

That is it.

  • No ptrace().
  • No copy_to_user().
  • No access_ok().
  • No “Dear Kernel, may I please touch userspace?”

Just a normal assignment.

From the kernel’s point of view, this is simply a write to a valid pointer.

Which leads to a slightly uncomfortable realization:

once a userspace page is reached through the linear mapping, it stops being special.

It is just memory.

And the kernel is very good at writing to memory.

“But Surely Read-Only Memory Is Safe?”

A reasonable objection is:

“Fine, but this probably works only on writable pages.”

That would be reassuring… Would it really?

Even then, the wall would still have a door.

Unfortunately, reality is even less reassuring.

The experiment works even when the target string is moved from .bss to .rodata.

In other words, the process sees the page as read-only.

The process itself cannot modify the string.

And yet, after the userspace address has been translated to its corresponding location in the linear mapping, changing the supposedly immutable data still requires nothing more sophisticated than:

*dst_target = act.value;

No page permissions are changed. No special write mechanism is invoked. No dedicated “modify userspace memory” API is used.

Just a normal store through a valid kernel pointer.

So the surprising part is not that the kernel, in principle, has enough privilege to modify the page.

The surprising part is that once the page is reached through the linear mapping, the actual modification is indistinguishable from writing to any other ordinary kernel address.

Same physical page. Different virtual permissions. Same single dereference.

The Grand Finale

Before the write, the program prints one string.

After the write, it prints a different one.

buildroot login: root # /usr/umem_poke/poc.sh running without corruption message address: 0x0000000000400168 <TOKEN> running again with corruption [ 27.675393] umem_poke: loading out-of-tree module taints kernel. message address: 0x0000000000400168 [ 28.751270] umem_poke: req: poke memory for pid=107 mem[400168]=1074707572726f63 [ 28.752020] umem_poke: act: poke memory for pid=107 mem[400168]=1074707572726f63 corrupt #

The process does not crash.

The kernel does not panic.

No alarms ring. No dramatic music starts.

The application simply continues running with modified data.

Which, from a functional safety perspective, is often the most interesting outcome.

Not:

“The system exploded.”

But:

“The system kept working… incorrectly.”

From Layout Curiosity to Concrete Effect

At this point, the full chain becomes visible:

+---------------------------+ | shared physical memory | +---------------------------+ | \ / V +---------------------------+ | possible adjacency | +---------------------------+ | \ / V +---------------------------+ | kernel object at page end | +---------------------------+ | \ / V +---------------------------+ | kernel write defect | +---------------------------+ | \ / V +---------------------------+ | ordinary pointer store | +---------------------------+ | \ / V +---------------------------+ | userspace data corruption | +---------------------------+ | \ / V +---------------------------+ | incorrect but apparently | | normal behavior | +---------------------------+

So the original question:

“Can kernel and user pages be physically adjacent?”

Now has a more complete answer:

Yes.

And if a kernel write goes in the wrong direction, the next page may belong to a perfectly innocent process.

That process may continue running as if nothing happened, except for the small detail that its data is no longer what it thinks it is.

Why This Matters

From a safety perspective, the concern is not:

user → kernel corruption (blocked by MMU)

The concern is:

kernel → anything else

Because in the linear mapping:

  • everything shares the same physical space
  • adjacency is not controlled
  • allocators optimize for performance, not isolation

So when something goes wrong:

the target is simply “the next page”

And what that page is… depends on the moment.

Bigger Picture

This PoC is not proving a vulnerability.

It is proving something more subtle:

The system does not guarantee separation at the physical level.

And in safety:

lack of guarantees is already a problem.

Final Thought

The tool didn’t find a bug.

It found something more annoying:

an assumption that is not always true.

And those are usually the ones that hurt the most.

NOTE: The same article also appears in the Elisa Blog.

Tuesday, April 28, 2026

Native vs Cross Compilation: a Love Story, a War Story, and a Debugging Nightmare

Let’s talk about something that has caused more arguments than tabs vs spaces, more confusion than ./configure && make && make install, and more gray hair than undefined behavior:

native vs cross compilation

This is not just a technical topic. This is culture. This is history. This is… mild trauma for distro maintainers.

In the beginning, there was only native

To understand the present landscape, we have to look back to the 90s, when the foundations were laid. So…

A long time ago in a galaxy far, far away…

(read: before everything was ARM), life was simple.

You had:

  • one machine
  • one architecture
  • one compiler

You wrote code, you compiled it, you ran it.

All on the same machine.

echo -e '#include <stdio.h>\nint main(){printf("hello\\n");}' | cc -x c - -o hello && ./hello
No "build vs host vs target." No sysroots. No cross toolchains.

Just vibes… No CI, No containers… Just you, your compiler, and a questionable amount of confidence.

Then came the rise of the Linux distributions

Early Linux was not a “distribution” in the modern sense.

It was:

  • a kernel
  • plus a pile of userland tools (mostly from GNU) (Ever wondered why GNU Linux? This would leave the place for another war story: what is Linux? The kernel or the complete system? But we need to focus, this story will be for another time.)
  • manually assembled by whoever was brave enough

Installing Linux in the early 90s was less “setup” and more “ritual”.

But Linux grew, and things changed:

  • more users
  • more software
  • more versions
  • more dependencies

Suddenly, maintainers were not compiling one program.

They were compiling:

  • hundreds → then thousands → then tens of thousands of packages

And not just once, but:

  • for every release
  • for every update
  • for every security fix

At this point, distributions like Slackware, Debian, and later Red Hat appeared with a mission:

Take all this chaos and make it installable, updatable, and consistent.

Then things started to evolve, the panorama quickly changed and new requirements became necessary.

Binary distribution

Distributions don’t ship source (only).

They ship:

  • precompiled binaries

So they must ensure:

  • the binary you download is:
    • correct
    • consistent
    • built in a known environment

Later you will need a few more of these, but we were not at the point yet.

Reproducibility becomes a necessity, not a luxury

Here is the key shift: Distributions needed to be able to rebuild the same software reliably, “it works on my machine” was no more enough.

It seems trivial:

Given this source, we can rebuild the exact package again.

If you’re wonder why this requirement, here’s a few reasons:

  • Debugging - User reports “Program crashes” and Maintainer needs to:
    • rebuild the exact binary
    • reproduce the issue
    • inspect symbols, offsets, behavior
  • Security - Security fixes require:
    • rebuilding packages
    • ensuring nothing unexpected changed
  • Updates and upgrades - When updating:
    • package A depends on B
    • B changes ABI subtly
    • A must be rebuilt consistently

Reproducibility is not the default state… you have to actively remove sources of entropy.

For example, even changing only the source filename can be enough to break bit-for-bit identity:

$ echo -e '#include <stdio.h>\nint main(){printf("hello\\n");}' >1.c $ echo -e '#include <stdio.h>\nint main(){printf("hello\\n");}' >2.c $ gcc 1.c -o 1 $ gcc 2.c -o 2 $ md5sum 1 2 6aa0460c6e9d029b5c40f27c86fadfa3 1 ada5719e11d8b6a17f020353f23f12d6 2 $ hexdump -C 1 >1.hex $ hexdump -C 2 >2.hex $ diff 1.hex 2.hex 258c258 < 000026d0 65 6e 74 72 79 00 31 2e 63 00 63 72 74 65 6e 64 |entry.1.c.crtend| --- > 000026d0 65 6e 74 72 79 00 32 2e 63 00 63 72 74 65 6e 64 |entry.2.c.crtend|

Same code, same compiler, different hashes. In this instance, the mere difference in filenames caused the compiler to embed unique strings into the binary, breaking bit-for-bit identity.

Once reproducibility became a requirement, the question was no longer how to build, but where and under what conditions to build.

The solution: controlled build environments

There were times when builds were done by the “build computer”. A machine developers used to build the new software. It was untouchable, update or change anything could break the reproducibility of the software. It was evidently not a scalable solution, if only because that eventually broke. To solve all of this, distributions invented:

  • build roots (clean environments)
  • packaging systems (RPM, DEB)
  • build farms (many machines building packages)
  • strict dependency tracking

The idea was:

Every package is built in a clean, controlled environment.

This is where tools like:

come into play.

Then Linux escaped x86

Linux did not stay on x86.

It spread like a highly portable virus (in a good way):

Year Arch
1993 Dec Alpha
1995 Sparc
1995 MIPS
1996 M68k
1996 PowerPC
1998 ARM

…

Now we had a problem:

What if your build machine is x86, but your target is ARM?

Buying one machine per architecture works… until it doesn’t. And by the way, a starting architecture, likely has not any native environment… system does not exist yet.

Enter cross-compilation

Before even moving forward, have you ever stopped to consider what cross compilation actually is?

If you compile a package using a container running Alpine Linux for your Fedora, are you cross compiling things?

In common usage, people often reserve “cross-compilation” for different architectures. In toolchain terms, however, changing the ABI (e.g. glibc vs musl) is already enough.

Real-world example: I once hit an underlinking issue with sem_init() on MIPSEL caused by symbol versioning differences in glibc. Everything compiled fine, nothing obvious was missing, but the binary failed at runtime in ways that made no sense until you looked at the symbol versions.

That’s the kind of problem you only get when ABI assumptions don’t actually match reality.

The Three Pillars of a Platform

In toolchain development, a “Platform” is defined by a triplet that includes more than just the CPU:

  1. Architecture (ISA): e.g., x86_64, arm64, riscv64.
  2. Operating System: e.g., linux, windows, darwin, none (bare metal).
  3. C Library / Environment (ABI): e.g., gnu (glibc), musl, uclibc, msvc.

If any of these three differ between the machine doing the work and the machine running the code, you are technically in the “cross-compilation landscape.”

Back to the original question, even if you are on the same physical CPU, and even the same OS, containers share the kernel, a compiler running on an Alpine Linux system (which uses musl) targeting a Fedora system (which uses glibc), you are technically cross compiling.

If not convinced yet, here are a few points to consider:

  • The Linker needs to look for different library paths.
  • The Headers define structures (like struct stat) differently.
  • The Startup Code (crt0.o) that initializes the program before main() is unique to each C library.

Now it is safe to name cross compilation

In a nutshell, Cross-compilation means:

I build here, but I target somewhere else.

You now have:

  • a compiler that produces aarch64 binaries on x86_64
  • a sysroot with aarch64 libraries
  • a growing sense of complexity

Which means you can:

aarch64-linux-gnu-gcc hello.c -o hello

Congratulations, you’ve successfully built a binary you can’t run, can’t test, and aren’t entirely sure is correct.

Oops.

The religion war begins

Two camps emerge:

Native camp

  • Reality is the best test.
  • If it runs, it works.
  • Just build on the target.

Cross camp

  • Define your environment properly.
  • Stop relying on luck.
  • Your build system is broken, not cross.

Both are right.

Both are wrong.

Both have scars.

At some point, every engineer becomes opinionated about this. Not because they studied it, because they suffered through it.

Linux Distribution prefer Native

Most major distributions prefer native builds:

  • Fedora / RHEL → Koji + Mock
  • Debian → buildd network
  • Ubuntu → Launchpad

Why?

Not because cross is evil, but because:

Native builds tolerate messy software better.

And let’s be honest:

A lot of upstream software is messy.

Is there any foundation why native is better than cross?

Let’s see the Native supporters claim:

  • Cross compilation produces inferior grade quality executables.
  • Cross compilation can fail to configure.
  • Cross compilation can use wrong configuration and you only realize it when binary won’t work or crashes.
  • Cross compile implies Library Pollution.
  • Native compilation allows in place testing.
  • Can be situation you need to distribute intermediate products of the build, cross compilation does produce no usable artifacts.

And also Cross supporters’ claims:

  • It is not feasible to have a native builder of all the architecures you want to support.
  • In embedded world target are typically very small machines, build on them will take forever.
  • Cross builds can give the same exact result as Native compilation.
  • No bonds or ties, you can build anywhere, for anything.

Let’s address these claims and let’s see how things really are.

Do cross builds produce same binaries as native builds?

Short answer:

They can, but it won’t typically happen.

It can be actually demonstrated that if cross and native run:

  • same toolchain
  • controlled environment
  • And you add the secret sauce (compiler flags)

The result can be bitwise identical binaries.

Here comes the fun part: same toolchain and controlled environment are not enough to have bitwise identical executable. You might need to add the ‘secret sauce’ and control sources of non-determinism to approach reproducible builds.

But the important point is that for “identical” .text and .data same toolchain and controlled environment can be enough.

Common reasons why binaries can differ despite same toolchain and controlled environment are:

  • Timestamps differences: build time embedded is embedded in ELF comments and __DATE__, __TIME__ could be used by both compiler and usercode.
  • Hostnames / usernames: embedded in debug info and sometimes in build notes.
  • Compiler fingerprints: Compilers are like Dogs, they use to pee where the step, which means: linker version, build IDs, etc.
  • File ordering: This one is subtle and evil. make collects object files, filesystem returns them in some order, and linker consumes them in that order. Change file creation order leads to changes in binary layout.
  • Parallel build non-determinism: modern builds are heavily parallelized (make -j, LTO, linker jobs). Jobs don’t always finish in the same order, and neither does the linker. Sometimes, your binary depends on which core was faster that day.

If you want identical binaries, you need to remove as many sources of entropy as possible.

Part of the entropy can be handled with compiler and linker flags, using what I called the secret sauce.

Syndrome Cure Notes
Timestamps SOURCE_DATE_EPOCH Instead of using “Now,” Compilers will use the timestamp you provide.
Timestamps -Wno-builtin-macro-redefined Allows you to manually redefine __DATE__ and __TIME__ via -D flags if you can’t use SOURCE_DATE_EPOCH
Polluting sections -fno-ident Tells the compiler not to generate the .ident or .comment section containing the compiler version.
Build Paths -fdebug-prefix-map=/old/=/new/ Replaces absolute paths in debug info and in __FILE__ with a generic string.

But some sources of non-determinism live in the build system itself, and require discipline, not just flags.

Does this really exist cross compiler configuration problem?

Before we can understand why people distrust cross-compilation, we need to talk about a legendary creature Autotools.

Specifically:

autoconf automake ./configure

If you’ve ever typed:./configure && make …you’ve already met it.

So, what problem Autotools was solving?

To understand that, we need to go back to the 90s when there was no:

  • standardized Linux
  • consistent libc behavior
  • reliable compiler features
  • unified system APIs

Instead, you had a nice zoo:

  • different UNIX systems
  • different compilers
  • different headers
  • different kernel quirks

Back then it was common to answering questions like:

  • Does this system have strlcpy?
  • Is long 32 or 64 bits?
  • Does malloc(0) behave sanely?
  • Is fork() broken here?

And the answer was often: Maybe

Autotools solved the problem by saying:

Don’t guess. Ask the system.

But instead of asking politely, they do this:

  • Generate a tiny C program
  • Compile it
  • Run it
  • Observe the result
  • Decide how to build the real software

Example:

int main() { return sizeof(long) == 8 ? 0 : 1; }

If it runs and returns 0 then we know long is 64-bit.

It was clever, for its time.

Because documentation was unreliable, headers lied and systems were inconsistent.

But as you might guess, this model does not meet cross compilation because you build on x86, but your target is ARM, to name one.

Autotools would compile a test program… …and then tries to run it.

This is the origin of some native supporters’ claims about about configuration.

Question is: are we still at that point?

The answer? Spin the wheel, because Autotools has evolved, but it hasn’t entirely escaped its roots.

Current autotool can:

  • use cached answers: Instead of running code the code above it can use ac_cv_sizeof_long=8
  • Cross-aware configuration: by telling it --host=aarch64-linux-gnu it disables some runtime checks, switches to safer assumptions
  • Replace execution with knowledge: many checks were rewritten to inspect headers and use known architecture traits

But it still, sometimes, when execution fails, just… guesses…

and here’s where subtle bugs, wrong assumptions, architecture-specific breakage come.

When the system can’t answer at build time, we can either guess… or we can simulate the answer. Therefore, sometimes if you want to cross build, you need also to emulate using qemu.

Qemu enables autotools to:

  • Probe executables can run successfully under emulation
  • Configuration becomes more accurate and less dependent on assumptions
  • Complex or runtime-dependent checks become feasible again

This effectively restores the original Autotools probing model, even in cross environments.

Library pollution, what is this thing.

Native supporters calims cross builds to suffer from library pollution.

During a cross build, the build system accidentally links against host libraries instead of target libraries.

So you think you built for ARM (or another target), but headers came from one place, libraries came from another and the result is inconsistent or outright broken.

Making it less “pollution” and more “you accidentally linked against the wrong universe”.

Library pollution is usually discussed in the context of cross-compilation, but the underlying issue is broader:

it’s really about insufficient isolation between dependency domains.

At its core, library pollution happens whenever: the compiler and linker pick up inconsistent headers and libraries from different environments.

That can occur in:

  • cross builds (host vs target)
  • native builds (system vs custom prefixes, multiple versions, etc.)

The difference is not whether it happens, but how it fails.

The real root cause: isolation failure

The key variable is: > how well your build environment is isolated

This can involve:

  • sysroots
  • prefix separation (/usr vs /usr/local vs /opt/…)
  • environment variables
  • build system discipline

When isolation breaks, pollution becomes possible…

regardless of cross/native.

But then, why only the cross build gets the spotlight?

Cross builds make the problem obvious because architectures differ and binaries are often not runnable on the host. So when pollution happens linker may fail immediately or binary won’t even start or crashes instantly.

In practice the failure is loud and early… And you can’t deploy your package.

But beware that if happen on native builds thing can be even worse because headers and libraries are often compatible enough to compile, ABI differences may be subtle and symbol versions may partially match.

The result is silent inconsistency, executable can work, and perhaps only break on a few corner cases.

Cross vs Native pollution chart:

Aspect Cross build Native build
Architecture mismatch Yes No
Build failure likelihood High Low
Runtime failure Immediate Delayed / conditional
Debug difficulty Moderate Very high
Typical symptom Won’t start / crashes instantly Sporadic crashes

Intermediate prducts of the building

In a native build, all generated tools, scripts, and metadata are produced for the same environment in which they will later be used. This makes it possible to package and reuse the entire build context, such as kernel headers, symbol information, and helper binaries… for downstream tasks like out-of-tree module compilation.

In contrast, cross builds split the world into host and target components.

While the final binaries target the desired architecture, many intermediate tools are built for the host and are not portable. This makes the resulting build artifacts harder to redistribute or reuse in a different environment, especially without reconstructing the original build setup.

The concrete example for the Kernel (few other may exist) When you build something like the Linux kernel, you’re not just producing:

  • vmlinuz / *Image

You’re also generating a whole ecosystem of intermediate tools and metadata, such as:

  • scripts/ utilities (e.g. modpost, genksyms)
  • generated headers (include/generated/*)
  • configuration state (.config, Module.symvers)
  • build glue for modules (kbuild infrastructure)

These are not optional, they are required for:

building out-of-tree modules later

Some of these artifacts are binaries targeting the build architecture. Because in users’ world OOT modules are built on the machine that is going to use them cross build artifacts are not usable.

Testing

Testing is one of the weak, if not the weakest, point of the cross build. In native build, it’s easy just the test and you’re done.

On the Cross builds, on the other hand, testing is possible, but you need to work a little.

The use of QEMU user-mode emulation, in particular qemu-user-static, provides a practical way to execute binaries built for a foreign architecture directly on a host system. This capability is especially valuable in cross-compilation workflows, where executing target binaries is otherwise not possible.

In a traditional cross-compilation setup, test binaries produced during the build process cannot be executed because they target a different architecture than the host. This limitation affects both:

  • Test suites (unit and integration tests)
  • Build-time probing, such as those performed by GNU Autoconf

By integrating QEMU user-mode with the Linux binfmt_misc mechanism, foreign binaries can be executed transparently. When combined with a target root filesystem (sysroot), this allows:

  • Running test suites for the target architecture without dedicated hardware
  • Executing Autotools-generated probe binaries, enabling runtime checks (e.g. via AC_RUN_IFELSE) that would otherwise fail during cross-compilation

As a result, builds can behave similarly to native builds, reducing the need for manual overrides such as cached configuration variables.

Emulation Performance and setup Considerations

Execution under QEMU user-mode is generally slower than native execution due to instruction translation. However:

  • The overhead is often acceptable for build-time testing.
  • In some cases, the difference is negligible compared to overall build time.
  • When targeting inherently slower architectures, the perceived slowdown may be less significant, and in some constrained comparisons the host may still outperform real hardware.

Despite this, performance-sensitive or timing-dependent tests may exhibit different behavior under emulation.

While QEMU simplifies execution, it introduces additional setup requirements:

Configuration of binfmt_misc to associate foreign binaries with the appropriate QEMU interpreter Provision of a complete target root filesystem, including: * Dynamic linker * Shared libraries * Correct filesystem layout

This effectively shifts complexity from build configuration (e.g. Autotools cross-compilation issues) to environment preparation.

Summary of Cross vs Native build

This is a comparison synthesis between the two approaches. No judgement is given by choice, native vs build is a religion, and I don’t want to be hostile to any.

Issue Native Synthetic Cross Notes
Artifact Reproduciblity Reproducible = Reproducible Both build methods can lead to reproducible artifacts.
Configuration problems Mature <== Minor problems exist Historically Autotool is using host execution go guess the configuration. In cross environment this is not possible, butnewer versions of autotool support cross build enviroment through caching. Other approaches use qemu-user-static to make some complicated autotool setup work on cross environments.
Binary Artifacts Quality Same = Same If setup correctly artifact can be bitwise identical.
Build Enviroment Pollution Bad Setup = Bad Setup Not a Cross build specific issue. it is more a sysroot isolation problem. though, failing this, can have worse consequences in Native than cross.
Build Product Testing Easy <== Complex Test a build product, means executes it under some conditions. Native enviroment execute the artifact is trivial, in cross enviroment it is not, unless you cheat with qemu-user-static.
Build performance (speed) Variable ==> Optimal It is a fact, some minor architectures are slow. Iot devices can target slow architectures where build can be challenging.
Per architecture builder Issues ==> none A Linux distro targeting many architecures using Native compilation, needs to have at least a worker per architecure, and this scales bad.

Linux build systems overview

As last step, lets see ta some of more common build system that distro uses to generate linux.

System / Tool Used by Native / Cross Core tech Notes
Koji Fedora, RHEL, CentOS Native (per-arch workers) RPM, Mock, chroot, distributed scheduler Centralized build farm; builds SRPMs on target-arch machines
Mock Fedora/RHEL (via Koji) Native (isolated) chroot, RPM Creates clean build roots; prevents host contamination
Open Build Service openSUSE, SLE Mostly native (multi-arch workers), supports cross RPM/DEB, chroot, distributed builds Public multi-distro build platform
Debian buildd network Debian Native dpkg, debhelper, buildd daemons Distributed autobuilders per architecture
Launchpad Ubuntu Native dpkg, chroot/containers CI + packaging + build farm
Arch Build System Arch Linux Native (mostly) PKGBUILD, makepkg Simple recipe-based builds; less centralized
Portage Gentoo Native (user builds), supports cross ebuild, bash Source-based; builds happen on user machine
Buildroot Embedded (various) Cross (primary) Make, Kconfig Builds full rootfs + toolchain; minimal, deterministic
Yocto Project / BitBake Embedded, industrial distros Cross (primary) BitBake, metadata layers Highly configurable, large-scale embedded builds
OpenWrt build system OpenWrt Cross Make, custom infra Router-focused; cross toolchains + package feeds
Nix NixOS, others Hybrid (native + cross) functional DSL, isolated store Strong reproducibility model
GNU Guix Guix System Hybrid (native + cross) Scheme, functional builds Similar to Nix, emphasizes purity

How’s the future?

Newer ecosystems like Rust or Go make cross-compilation feel almost… reasonable.

They remove a lot of the historical friction, but they don’t change the fundamentals. You still need to control your environment, your dependencies, and your assumptions. The difference is that now the toolchain fails less creatively.

Native vs cross is still a debate. The difference is that now we have better tools… and the same old problems.

Tuesday, March 10, 2026

When a Homelab Migration Awakens Your Inner Networker

My blog has never been particularly homogeneous. This post is probably weirder than usual.

These days most people know me for work around Linux kernel safety and the Elisa Community. That’s the professional side. But like many engineers, I also have a homelab where I experiment, break things, and occasionally learn something useful while pretending it’s just procrastination.

This story comes from when I decided my network needed an upgrade.

And it unexpectedly reminded me why networking used to be my field in the first place.

The Pascal Incident

To explain why I even cared about this migration, we need to go back a bit.

I originally started as a Assembly-Pascal programmer.

In 1997 I had a small disagreement with my computer architecture professor about whether Pascal could be used for low-level work.

Naturally, the only rational response was to write an arcade machine emulator in Pascal.

This one: a Space Invaders emulator. Which I proudly demonstrated to the professor. I still remember that conversation.

Me: Hey professor, look what I built using Pascal. If used properly, you don’t actually need C.

To be fair, this was only partially true. I did use the Pascal compiler to build the executable, but most of the low-level work ended up written as large blocks of inline assembly inside the Pascal source. Anyway, I showed it to the professor.

The professor watched the screen for a moment.

Professor: Well yes. Nice game.

That was not the reaction I expected.

Me: Look, I didn’t make this game. Someone in Japan did… back in 1978.

At that point the professor looked mildly annoyed.

Professor: Then what exactly are you showing me today?

Which honestly left me unarmed.

I would have liked to answer: the emulator, you idiot… But I didn’t. For the record it neither improved my grade nor impressed the professor. But the experiment left me with a fascination for understanding systems by emulating them. Which later led to another emulator.

Instead of arcade hardware, I tried to emulate a Cisco router.

That project became my computer engineering master’s thesis: a Cisco 2500 router emulator, again written in Pascal… the last time before burying it forever.

Reverse engineering parts of the router firmware to understand the hardware architecture forced me to dive into routing logic…

That was basically my accidental entry point into network engineering. So yes, my networking career technically started with a router emulator written in Pascal. Which probably explains a few things.

Back to the Homelab

Fast forward a few decades.

The task that triggered this story looked simple enough:

Replace an old Cisco 3560, sitting in my lab and running almost continuously for 15 years with something newer… a Ruckus ICX-7150, and finally enter the gigabit era.

Which mostly meant replacing one relic with a slightly younger relic. The 3560 was already ancient. The ICX-7150 is not ancient yet… but it’s already end-of-sale, so it’s clearly on the right trajectory.

At first glance the CLI felt familiar. Cisco and Ruckus commands look similar enough that you instinctively think:

“Ah yes. I know exactly what this does.”

That confidence usually lasts about five minutes. After that you discover that while the commands look similar, the behavior is not. Part of the reason is historical.

This network used to be entirely Cisco. Phones, switches, everything. In that environment life was easy. Phones learned their VLAN via CDP, everything powered correctly, and nobody asked too many questions.

The moment I started replacing pieces, I discovered which parts of that convenience were standards, and which parts were simply Cisco assumptions.

Some of that I expected. Some of it I discovered the hard way.

The Troubleshooting Story

The setup was simple:

  • Cisco 7940 phones
  • SCCP firmware
  • Asterisk
  • configuration via TFTP
  • DHCP instead of CDP

The goal was to run the phones in a vendor-neutral network.

Initially everything worked. Phones booted and registered. Then they started doing something strange. They would run normally for a while…

disconnect… then reconnect.

Phone logs showed events like:

Code:8103

Packet captures showed:

  • SCCP keepalive retransmissions
  • TCP resets
  • reconnect cycles

Then another symptom appeared: occasionally the phone did not respond to ARP requests. That pointed suspicion directly at the phone. Old 79xx firmware is known to be… quirky. So the investigation went there first.

The Firmware Detour

The phones were running:

P00308000100

Which triggered the usual Cisco firmware archaeology. Finding firmware for older Cisco devices is a bit like looking for historical artifacts: you know it exists, you’re just not sure where.

Eventually a newer firmware image was located and installed.

The upgrade worked… And the problem remained.

So the firmware theory collapsed.

At this point the investigation had already consumed a fair amount of time… and it was still focused on the phone.

The Turning Point

The turning point came after a longer packet capture session. Something about the ARP traffic looked wrong. So the next step was obvious: check what the switch actually saw. A mirror session was configured on the ICX and the phone port was captured.

The mirrored traffic showed:

  • unicast traffic on VLAN 2
  • broadcast traffic from the phone

But something was missing.

The ARP broadcast from Asterisk never appeared. To confirm this, traffic was captured in several places:

  • the ICX mirror
  • a VLAN-2 host
  • a Cisco mirror upstream

The result was consistent. The ARP broadcast existed on the Cisco side, but never reached the ICX access port. At that moment the investigation pivoted. This was no longer a phone problem.

It was a switch problem.

Finding the Culprit

To confirm the path, the Cisco 3560 uplink was mirrored. The ARP request was clearly visible leaving the Cisco trunk:

Who has 10.1.5.57? Tell 10.1.5.1

But the broadcast never appeared on the ICX.

The packet was disappearing between the switches.

Attention turned to the ICX uplink configuration:

vlan 2 tagged ethernet 1/2/2 ! interface ethernet 1/2/2 dual-mode

VLAN configuration looked correct: VLAN 2 tagged 1/2/2

Various things were tested:

  • VLAN membership
  • DHCP snooping
  • spanning tree
  • broadcast filtering

Nothing changed.

The Fix

Eventually attention returned to one small detail on the ICX uplink:

interface ethernet 1/2/2 dual-mode

In FastIron (Brocade/Ruckus) this is roughly equivalent to the native VLAN on a Cisco trunk.

And in fact, if you come from a Ruckus-native background, that line probably would not raise any alarms at all.

The only reason it caught my attention is precisely because I wasn’t thinking like a Ruckus engineer.

So, mostly out of prejudice rather than conviction, I tried removing it.

Immediately:

  • ARP broadcasts appeared
  • VLAN-2 hosts received them
  • the phone stopped disconnecting

The note I wrote at the time was simply:

was that! no dual-mode on 1/2/2 solved the problem.

And that closed the investigation.

Which strongly suggests the real root cause was not the configuration itself, but likely a firmware bug in the switch triggered by that combination of trunking behavior.

NOTE: On Brocade/Ruckus FastIron switches, the dual-mode command is used to allow a port that carries tagged VLANs to also accept and transmit untagged traffic. Unlike Cisco trunks, which always include a native VLAN by default, a FastIron port with tagged VLANs normally sends only tagged frames unless dual-mode is enabled. The command itself takes no arguments and has remained syntactically stable across FastIron versions; it simply enables the ability for the port to carry untagged frames. The actual native (PVID) VLAN is determined separately by the port’s untagged VLAN membership, which can be changed under the interface with vlan-config move untagged vlan <vid>. For engineers familiar with Cisco, the combination of dual-mode plus the port’s untagged VLAN is functionally equivalent to configuring a native VLAN on a trunk.

A Note About AI

One final note.

I have been skeptical about AI tools for quite a while and avoided using them in my workflow.

This debugging session, and a few others, are changing my mind a bit.

The AI suggested ICX commands I didn’t know, reminded me of Cisco syntax I had not used in years, and helped narrow the search space.

Without that assistance I would probably have spent days digging through documentation.

Instead the whole problem unraveled in one night.

Which, for a homelab experiment, is an excellent outcome; and significantly better than spending a weekend reading switch manuals.