One Terminal to Rule Them All

Or: how an ADM-3A became the cockpit of my homelab

Usually, when I write here, I try to talk about serious technical things.

• Kernel memory.
• Build systems.
• Functional safety.

The kind of topics where a small mistake can become a paper, a patch, or a very long afternoon.

This time, I want to write about something equally important … to me, at least:

• old electronic stuff.

Not “old” as in “deprecated by a vendor who would like me to buy the new cloud-enabled subscription version.”

I mean properly old… Old enough to have a CRT.

Old enough that when it displays a prompt, it does not feel like a terminal emulator. It feels like the computer is talking to you directly.

The arrival of the dumb terminal

At some point, I bought myself a Lear Siegler ADM-3A terminal. Inside I found a label that says 1977, meaning we are almost the same age.

This object is not just “an old terminal”, it is one of those objects that quietly left fingerprints all over computing history.

For example, if your shell abbreviates $HOME as ~, the ADM-3A keyboard is part of that story: the ~ character lived on the key labeled HOME.

And if your fingers know that h, j, k, and l are movement keys in vi, that is not random ancient wizardry either. On the ADM-3A keyboard, those letters had cursor symbols printed on them. Bill Joy developed vi using this kind of terminal, and apparently the muscle memory never left the building.

So yes, technically it is a dumb terminal… But “dumb” feels unfair.

It is more like a historical device with very few opinions and excellent posture.

• A chunky case.
• A real keyboard.
• A screen that does not care about your GPU.
• No tabs.
• No browser.
• No notifications.
• No “your session has expired.”

Just characters… beautiful, honest, monochrome characters and a hardware keyboard where you can even find a key labeled HERE IS, close to RETURN, which sounds like the terminal is politely introducing itself at a party.

And that is almost what it did. With the optional answer-back feature, the terminal could send a predefined identification string, either when asked by the host or when the operator pressed the key.

A tiny serial-age “hello, it’s me” fingerprint.

The ADM-3A had manners.

The ADM-3A is often called a dumb terminal, but I think that is unfair. It is not dumb! It is focused. It has one job:

“Give me bytes, and I will show them.”

First contact with the homelab

Of course, after buying an old serial terminal, the obvious next question was:

“What should I connect it to?”

The boring answer would have been: nothing. Put it on a shelf. Take pictures. Occasionally say, “yes, it still works,” like people do with classic cars that never leave the garage.

Not what I intended for it: a terminal wants a computer.

So I connected it to my VM host machine, and it worked.

There is something very satisfying about controlling a modern Linux system through a terminal from another era. It feels like time travel, except instead of meeting dinosaurs, you meet systemd, sometimes even scarier.

It feels weird and cute at the same time to have a machine from the past controlling a machine running virtual machines. But then reality arrived, wearing the traditional hat of all homelab problems:

“Nice. But you have more than one machine.”

The problem with having “just one more box”

My VM host was not the only machine in the lab with a serial console; there was also a network switch… and another server… if we want to call it such. It is my home PBX, running on an Atom machine. So it is technically a server, but also spiritually closer to an appliance that refuses to die.

The point is: I had more serial devices than terminals, and I did not want to keep moving cables around.

Moving serial cables by hand is fun the first three times. After that, well, I leave the rest to the reader.

What I was thinking was something like an old KVM switch… but for serial. One terminal, multiple machines and a magic key sequence to switch between them.

The old KVM idea, but with serial

Old KVM switches had a very nice property: they allowed you to keep one keyboard, one video output, and one mouse, and move between machines. How to do this was not always straightforward: sometimes with a button, sometimes with a key sequence and sometimes with a ritual known only to the previous owner.

I wanted the same experience, in serial form but perhaps using a more deterministic way, the key sequence, for example. So the ADM-3A would be the keyboard and screen and behind it, a small box would route traffic to one of several serial ports so that the active machine would receive what I type and the terminal would display what the active machine says.

It sounds simple.

It is because of that that I started the project just after buying the ADM-3A and I finished it only now.

Enter the small box

For the hardware side, I used a tiny FriendlyElec ZeroPi.

It is one of those small ARM boards that exists in the dangerous region between “cheap enough to use casually” and “capable enough to become infrastructure by accident.”

Mine is connected to a multi-port PL2303 USB serial adapter.

My past in the telephony appliances is payback, at last, and I have a good plan to reinvent an old telephony switch but for data instead of voice.

So, ladies and gentlemen, we are about to reinvent the wheel.

Writing the little tunnel

The software became serial_tunnel.

It is a small utility that connects one terminal side to multiple remote serial devices. The terminal talks to the selected device. The selected device talks back. If I press the configured escape sequence followed by a number, the active device changes.

Plain and oldish, but for this kind of project, boring is good. Boring means I can trust it at 1 a.m. when the switch is unhappy and the only thing I want is a console, not a distributed systems lecture.

The configuration defines the serial ports and the tunnels. Each remote has a name, a device path, a speed, and the usual serial parameters. The tunnel then says: these are the devices, this is the terminal, and this is the escape sequence.

The beauty of limits

Using an ADM-3A is a wonderful reminder that computers were not always chatty. A machine from that era did not print five screens of logs just to say “hello.”

It did not emit JSON, YAML, warnings, hints, stack traces, deprecation notices, and one cheerful message from a package manager telling you that funding is available.

It was more like:

login:

That was enough.

Modern machines are different, have feelings, and they express those feelings through logs… Tons of logs.

Still, the ADM-3A provides only 80 columns and 24 lines, beautifully retrò, elegant if you wish, but also disappointing if you consider modern system verbosity. In modern system metrics that 80x24 translates to approximately three boot messages and half a warning, fine if everything is working, very annoying if anything goes wrong.

So the first time I connected it to a Linux machine and watched output fly by, I had a thought:

“This is charming, but I have no idea what just happened.”

screen or tmux can help there”…

The heresy: adding SSH

This is where I committed a small act of retrocomputing heresy.

I added an SSH interface, not to replace the ADM-3A, that would be rude and the ADM-3A is the star of the show, but I wanted an SSH terminal that mirrors the same console hub.

Why?

Because sometimes the old terminal is perfect.

And sometimes I need to scroll back, copy a log line, or look at the same session from a laptop without standing in front of the rack like a 1970s operator waiting for the mainframe to confess.

So serial_tunnel also has an optional SSH terminal interface.

The idea is: the physical terminal and the SSH client share the same selected console. The ADM-3A is still the star of the lab, but the SSH side can observe and, when needed, interact using the same switching mechanism.

This gives me the best of both worlds:

• the tactile joy of the old terminal,
• the practical convenience of a modern SSH client,
• and the emotional conflict of knowing I added network access to a project whose whole charm is a serial cable.

Technology is compromise.

Sometimes compromise listens on port 2222. If you happen to have a terminal, a homelab, and what to use them together, check this out.

So now the dumb terminal is in charge of the smart machines. Which, frankly, feels like a healthy correction.

If you liked the story, please check out the code here

When “Probably Safe” Is Not Safe Enough

If you ever worked with Linux, you probably heard something like:

“User space and kernel space are isolated.”

Which is true. And also… not entirely comforting.

Because, as Murphy politely reminds us:

“Anything that can go wrong, will go wrong.”

At the ELISA LFSCS working group , we try to look at Linux from a functional safety perspective. That means we are not satisfied with “it usually works”… we want to understand what could go wrong.

From Munich: A Suspicious Idea

During the Munich 2025 session, we discussed what we called the linear mapping threat.

The idea is simple:

• Linux keeps a linear mapping of physical memory
• This mapping includes all RAM
• Kernel objects and user pages come from the same pool

So in theory:

A kernel object page could sit right next to a user page.

Not a fault… Not a bug… Just… how memory is laid out.

And that alone raises a question.

Because if a kernel bug causes an overflow, adjacency matters much more than isolation:

• overflow → likely direction → next page
• underflow → possible, but statistically less common

So the interesting case becomes:

kernel page → next → user page

At that point, this was still a hypothesis.

A reasonable one… But still a hypothesis.

From Hypothesis to Observation

So we asked:

Can we observe this situation?

Not in theory. Not in diagrams. But in actual memory.

To be clear, not the fault happening… just the fact that Kernel objects pages and userspace pages can sit one next to the other.

The Tool (a.k.a. “Let’s Peek at PFNs”)

Initially, the plan was simple:

“Let’s write a kernel module and scan the linear mapping.”

Because… that’s what kernel people do. (Maslow’s Hammer: If the only tool you have is a hammer, you tend to see every problem as a nail)

Then came the slightly embarrassing realization:

“Wait… I can already see this from userspace.”

Using /proc/kpageflags.

And suddenly:

• no kernel module needed
• no patching
• no special hooks

Just a userspace tool reading kernel-exposed data.

Small Technical Box: /proc/kpageflags

/proc/kpageflags exposes metadata for each physical page frame (PFN).

With it, you can:

• iterate over physical memory
• classify pages (anonymous, slab, huge, etc.)
• build a PFN-level view of the system

In practice, the tool:

• scans PFNs sequentially
• assigns a category per page
• renders a colored raster

Think of it as:

a “topographic map” of memory… where mountains are slab pages and plains are user memory

(Yes, the legend is still under construction.)

First Results: “This Looks… Too Clean”

Using the tool on a real system (x86), results were promising:

• memory looked mixed
• user and kernel pages were interleaved
• adjacency was visible

Good.

But the LFSCS working group set its reference platform as aarch64 many moons ago and when if came to the ARM64 VM we’re targeting:

• almost no user pages
• almost no slab pages
• everything looked… suspiciously clean

At this point, either:

1. the tool was wrong
2. the kernel was lying
3. or the system was too nice

What We Learned (a.k.a. “The Plot Thickens”)

Huge Pages Were Hiding Reality

Disabling THP:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

suddenly changed everything.

Why?

Because:

• userspace sees memory at page granularity (4KB)
• linear mapping may represent the same memory as huge pages

So we had:

fine-grained user memory vs coarse-grained linear mapping view

Disabling THP aligned the two worlds and revealed the real distribution.

Slab Pages Don’t Like Attention

The SLAB flag is only set on compound head pages.

Meaning:

most kernel objects are there… just not explicitly labeled

So the absence of slab in the visualization was… misleading.

The System Was Too Clean

The VM had:

• low workload
• low fragmentation
• limited kernel activity

In short:

not enough chaos to trigger interesting behavior

Forcing Reality (a.k.a. “Adding Some Chaos”)

One missing piece was understanding that:

a real system naturally creates fragmentation an idle virtual machine… usually does not

So to observe the phenomenon in a controlled environment, we needed to give to the kernel an helping foot.

The approach was to introduce controlled memory pressure and fragmentation using a small kernel-side helper.

The idea is simple:

• allocate many temporary objects
• insert a few persistent objects in between
• free the temporary ones

This leaves behind memory pages that are:

• partially used
• not reclaimable
• and therefore fragmented

This creates a layout where:

kernel allocations are spread across many partially filled pages

Which is exactly the condition needed to increase the chance of:

kernel page → next → user page adjacency

Think of memory like a block of cheese:

• temporary allocations = drilling holes
• persistent objects = small solid parts left behind

After removing the temporary allocations:

you don’t get a clean empty block you get a holey structure the kernel cannot easily compact

The Key Result

After:

• fixing classification
• aligning granularity
• adding controlled fragmentation

We can say:

Kernel and user pages can be physically adjacent.

Not always. Not everywhere. But definitely possible.

And importantly:

this can happen naturally on a real system while in a VM it may require forcing conditions

Looking Closer (a.k.a. “What Exactly Is Sitting at the Edge?”)

At this point, we knew that:

kernel pages and user pages can be physically adjacent.

Which is already interesting.

But adjacency alone does not tell us how easy it is for trouble to cross the border.

If the last object in the kernel page ends hundreds of bytes before the page boundary, then an overflow would need to be fairly enthusiastic.

If instead the object ends exactly at the page boundary…

then even a tiny mistake can immediately step into the next page.

So the obvious next question was:

“What is actually sitting at the very end of the slab page?”

Teaching the Tool Some Slab Archaeology

The original direct_map_view PoC was good at classifying pages.

The updated version became slightly more nosy.

Given a user page, it now:

• looks at the preceding PFN,
• checks whether that PFN is slab-backed,
• scans the page,
• and identifies the kernel object closest to the end of the page.

Using kmem_dump_obj(), the probe can determine which slab cache owns the object and where the object begins.

In other words:

we stopped looking only at pages and started looking at what is actually living inside them.

A Typical Result

[   66.010479] pfn_slab_probe: input_pfn=1078469 checked_pfn=1078468 flags=0xbfffe0000000200
[   66.010862] pfn_slab_probe: checked PFN 1078468 is slab-backed, scanning [ffff0000c74c4000 - ffff0000c74c5000)
[   66.011652] slab kmalloc-512 start ffff0000c74c4e00 pointer offset 504 size 512
[   66.013007] pfn_slab_probe: nearest candidate object at ffff0000c74c4ff8, offset 0xff8 into PFN 1078468

The probe did not find an object starting at 0xff8.

It found itself standing 8 bytes before the edge of the page, inside a kmalloc-512 object, at offset 504 from the beginning of that object.

So the math is:

0xff8 - 504 = 0xe00

The object starts at 0xe00 and, being 512 bytes long, ends at 0x1000.

Exactly at the page boundary.

Which is the memory-management equivalent of parking with the bumper touching the wall.

Turning Logs into Pictures

Because hexadecimal offsets are excellent at convincing computers but less effective with humans, a small script was added to convert the logs into SVG diagrams.

The result is a picture showing:

• the slab page,
• the last kernel object,
• the page boundary,
• and the user page immediately after it.

Sometimes, a drawing is worth several hundred printk()s.

From “Interesting Layout” to “Well… That Escalated Quickly”

At this point, we had demonstrated that:

• a kernel object can sit at the end of a page,
• the next physical page can belong to a user process.

The next question was unavoidable:

“What happens if the kernel writes past that boundary?”

Fortunately for science, and slightly less fortunately for the victim process, we now have a PoC for that too.

The Victim (a.k.a. “The Simplest Program We Could Get Away With”)

The userspace target is a tiny assembly program, available for both:

• AArch64 version
• x86_64 version

Its behavior is intentionally boring:

1. print the address of a string,
2. wait,
3. print the string,
4. exit.

That is all.

No threads. No libraries. No excuses.

If something changes, we know exactly who to blame.

Was it that simple?

The PoC script: poc.sh

Coordinates with the kernel module: umem_poke.c

The script does not do magic address translation in bash.

Which is probably good news for everyone.

It only coordinates the experiment: it starts the victim, gets the address printed by the process, and passes that VMA address to the module.

The module is the one doing the kernel-side work. It uses GUP to resolve the userspace address to the underlying page, computes the linear mapping address destination byte, and then performs the write.

After all that machinery, the corruption itself is still disappointingly simple:

*dst_target = act.value;

That is it.

• No ptrace().
• No copy_to_user().
• No access_ok().
• No “Dear Kernel, may I please touch userspace?”

Just a normal assignment.

From the kernel’s point of view, this is simply a write to a valid pointer.

Which leads to a slightly uncomfortable realization:

once a userspace page is reached through the linear mapping, it stops being special.

It is just memory.

And the kernel is very good at writing to memory.

“But Surely Read-Only Memory Is Safe?”

A reasonable objection is:

“Fine, but this probably works only on writable pages.”

That would be reassuring… Would it really?

Even then, the wall would still have a door.

Unfortunately, reality is even less reassuring.

The experiment works even when the target string is moved from .bss to .rodata.

In other words, the process sees the page as read-only.

The process itself cannot modify the string.

And yet, after the userspace address has been translated to its corresponding location in the linear mapping, changing the supposedly immutable data still requires nothing more sophisticated than:

*dst_target = act.value;

No page permissions are changed. No special write mechanism is invoked. No dedicated “modify userspace memory” API is used.

Just a normal store through a valid kernel pointer.

So the surprising part is not that the kernel, in principle, has enough privilege to modify the page.

The surprising part is that once the page is reached through the linear mapping, the actual modification is indistinguishable from writing to any other ordinary kernel address.

Same physical page. Different virtual permissions. Same single dereference.

The Grand Finale

Before the write, the program prints one string.

After the write, it prints a different one.

buildroot login: root
# /usr/umem_poke/poc.sh 
running without corruption
message address: 0x0000000000400168
<TOKEN>

running again with corruption
[   27.675393] umem_poke: loading out-of-tree module taints kernel.
message address: 0x0000000000400168
[   28.751270] umem_poke: req: poke memory for pid=107 mem[400168]=1074707572726f63
[   28.752020] umem_poke: act: poke memory for pid=107 mem[400168]=1074707572726f63
corrupt
# 

The process does not crash.

The kernel does not panic.

No alarms ring. No dramatic music starts.

The application simply continues running with modified data.

Which, from a functional safety perspective, is often the most interesting outcome.

Not:

“The system exploded.”

But:

“The system kept working… incorrectly.”

From Layout Curiosity to Concrete Effect

At this point, the full chain becomes visible:

+---------------------------+
|   shared physical memory  |
+---------------------------+
              |
             \ /
              V
+---------------------------+
|     possible adjacency    |
+---------------------------+
              |
             \ /
              V
+---------------------------+
| kernel object at page end |
+---------------------------+
              |
             \ /
              V
+---------------------------+
|   kernel write defect     |
+---------------------------+
              |
             \ /
              V
+---------------------------+
|   ordinary pointer store  |
+---------------------------+
              |
             \ /
              V
+---------------------------+
| userspace data corruption |
+---------------------------+
              |
             \ /
              V
+---------------------------+
|  incorrect but apparently |
|       normal behavior     |
+---------------------------+

So the original question:

“Can kernel and user pages be physically adjacent?”

Now has a more complete answer:

Yes.

And if a kernel write goes in the wrong direction, the next page may belong to a perfectly innocent process.

That process may continue running as if nothing happened, except for the small detail that its data is no longer what it thinks it is.

Why This Matters

From a safety perspective, the concern is not:

user → kernel corruption (blocked by MMU)

The concern is:

kernel → anything else

Because in the linear mapping:

• everything shares the same physical space
• adjacency is not controlled
• allocators optimize for performance, not isolation

So when something goes wrong:

the target is simply “the next page”

And what that page is… depends on the moment.

Bigger Picture

This PoC is not proving a vulnerability.

It is proving something more subtle:

The system does not guarantee separation at the physical level.

And in safety:

lack of guarantees is already a problem.

Final Thought

The tool didn’t find a bug.

It found something more annoying:

an assumption that is not always true.

And those are usually the ones that hurt the most.

NOTE: The same article also appears in the Elisa Blog.

Native vs Cross Compilation: a Love Story, a War Story, and a Debugging Nightmare

Let’s talk about something that has caused more arguments than tabs vs spaces, more confusion than ./configure && make && make install, and more gray hair than undefined behavior:

native vs cross compilation

This is not just a technical topic. This is culture. This is history. This is… mild trauma for distro maintainers.

In the beginning, there was only native

To understand the present landscape, we have to look back to the 90s, when the foundations were laid. So…

A long time ago in a galaxy far, far away…

(read: before everything was ARM), life was simple.

You had:

• one machine
• one architecture
• one compiler

You wrote code, you compiled it, you ran it.

All on the same machine.

echo -e '#include <stdio.h>\nint main(){printf("hello\\n");}' | cc -x c - -o hello && ./hello

No "build vs host vs target."
No sysroots.
No cross toolchains.

Just vibes… No CI, No containers… Just you, your compiler, and a questionable amount of confidence.

Then came the rise of the Linux distributions

Early Linux was not a “distribution” in the modern sense.

It was:

• a kernel
• plus a pile of userland tools (mostly from GNU) (Ever wondered why GNU Linux? This would leave the place for another war story: what is Linux? The kernel or the complete system? But we need to focus, this story will be for another time.)
• manually assembled by whoever was brave enough

Installing Linux in the early 90s was less “setup” and more “ritual”.

But Linux grew, and things changed:

• more users
• more software
• more versions
• more dependencies

Suddenly, maintainers were not compiling one program.

They were compiling:

• hundreds → then thousands → then tens of thousands of packages

And not just once, but:

• for every release
• for every update
• for every security fix

At this point, distributions like Slackware, Debian, and later Red Hat appeared with a mission:

Take all this chaos and make it installable, updatable, and consistent.

Then things started to evolve, the panorama quickly changed and new requirements became necessary.

Binary distribution

Distributions don’t ship source (only).

They ship:

• precompiled binaries

So they must ensure:

• the binary you download is:
  • correct
  • consistent
  • built in a known environment

Later you will need a few more of these, but we were not at the point yet.

Reproducibility becomes a necessity, not a luxury

Here is the key shift: Distributions needed to be able to rebuild the same software reliably, “it works on my machine” was no more enough.

It seems trivial:

Given this source, we can rebuild the exact package again.

If you’re wonder why this requirement, here’s a few reasons:

• Debugging - User reports “Program crashes” and Maintainer needs to:
  • rebuild the exact binary
  • reproduce the issue
  • inspect symbols, offsets, behavior
• Security - Security fixes require:
  • rebuilding packages
  • ensuring nothing unexpected changed
• Updates and upgrades - When updating:
  • package A depends on B
  • B changes ABI subtly
  • A must be rebuilt consistently

Reproducibility is not the default state… you have to actively remove sources of entropy.

For example, even changing only the source filename can be enough to break bit-for-bit identity:

$ echo -e '#include <stdio.h>\nint main(){printf("hello\\n");}' >1.c 
$ echo -e '#include <stdio.h>\nint main(){printf("hello\\n");}' >2.c 
$ gcc 1.c -o 1 
$ gcc 2.c -o 2 
$ md5sum 1 2 
6aa0460c6e9d029b5c40f27c86fadfa3 1 
ada5719e11d8b6a17f020353f23f12d6 2 
$ hexdump -C 1 >1.hex 
$ hexdump -C 2 >2.hex 
$ diff 1.hex 2.hex 
258c258 
< 000026d0 65 6e 74 72 79 00 31 2e 63 00 63 72 74 65 6e 64 |entry.1.c.crtend| 
--- 
> 000026d0 65 6e 74 72 79 00 32 2e 63 00 63 72 74 65 6e 64 |entry.2.c.crtend|

Same code, same compiler, different hashes. In this instance, the mere difference in filenames caused the compiler to embed unique strings into the binary, breaking bit-for-bit identity.

Once reproducibility became a requirement, the question was no longer how to build, but where and under what conditions to build.

The solution: controlled build environments

There were times when builds were done by the “build computer”. A machine developers used to build the new software. It was untouchable, update or change anything could break the reproducibility of the software. It was evidently not a scalable solution, if only because that eventually broke. To solve all of this, distributions invented:

• build roots (clean environments)
• packaging systems (RPM, DEB)
• build farms (many machines building packages)
• strict dependency tracking

The idea was:

Every package is built in a clean, controlled environment.

This is where tools like:

• Koji
• Mock

come into play.

Then Linux escaped x86

Linux did not stay on x86.

It spread like a highly portable virus (in a good way):

Year	Arch
1993	Dec Alpha
1995	Sparc
1995	MIPS
1996	M68k
1996	PowerPC
1998	ARM

…

Now we had a problem:

What if your build machine is x86, but your target is ARM?

Buying one machine per architecture works… until it doesn’t. And by the way, a starting architecture, likely has not any native environment… system does not exist yet.

Enter cross-compilation

Before even moving forward, have you ever stopped to consider what cross compilation actually is?

If you compile a package using a container running Alpine Linux for your Fedora, are you cross compiling things?

In common usage, people often reserve “cross-compilation” for different architectures. In toolchain terms, however, changing the ABI (e.g. glibc vs musl) is already enough.

Real-world example: I once hit an underlinking issue with sem_init() on MIPSEL caused by symbol versioning differences in glibc. Everything compiled fine, nothing obvious was missing, but the binary failed at runtime in ways that made no sense until you looked at the symbol versions.

That’s the kind of problem you only get when ABI assumptions don’t actually match reality.

The Three Pillars of a Platform

In toolchain development, a “Platform” is defined by a triplet that includes more than just the CPU:

1. Architecture (ISA): e.g., x86_64, arm64, riscv64.
2. Operating System: e.g., linux, windows, darwin, none (bare metal).
3. C Library / Environment (ABI): e.g., gnu (glibc), musl, uclibc, msvc.

If any of these three differ between the machine doing the work and the machine running the code, you are technically in the “cross-compilation landscape.”

Back to the original question, even if you are on the same physical CPU, and even the same OS, containers share the kernel, a compiler running on an Alpine Linux system (which uses musl) targeting a Fedora system (which uses glibc), you are technically cross compiling.

If not convinced yet, here are a few points to consider:

• The Linker needs to look for different library paths.
• The Headers define structures (like struct stat) differently.
• The Startup Code (crt0.o) that initializes the program before main() is unique to each C library.

Now it is safe to name cross compilation

In a nutshell, Cross-compilation means:

I build here, but I target somewhere else.

You now have:

• a compiler that produces aarch64 binaries on x86_64
• a sysroot with aarch64 libraries
• a growing sense of complexity

Which means you can:

aarch64-linux-gnu-gcc hello.c -o hello

Congratulations, you’ve successfully built a binary you can’t run, can’t test, and aren’t entirely sure is correct.

Oops.

The religion war begins

Two camps emerge:

Native camp

• Reality is the best test.
• If it runs, it works.
• Just build on the target.

Cross camp

• Define your environment properly.
• Stop relying on luck.
• Your build system is broken, not cross.

Both are right.

Both are wrong.

Both have scars.

At some point, every engineer becomes opinionated about this. Not because they studied it, because they suffered through it.

Linux Distribution prefer Native

Most major distributions prefer native builds:

• Fedora / RHEL → Koji + Mock
• Debian → buildd network
• Ubuntu → Launchpad

Why?

Not because cross is evil, but because:

Native builds tolerate messy software better.

And let’s be honest:

A lot of upstream software is messy.

Is there any foundation why native is better than cross?

Let’s see the Native supporters claim:

• Cross compilation produces inferior grade quality executables.
• Cross compilation can fail to configure.
• Cross compilation can use wrong configuration and you only realize it when binary won’t work or crashes.
• Cross compile implies Library Pollution.
• Native compilation allows in place testing.
• Can be situation you need to distribute intermediate products of the build, cross compilation does produce no usable artifacts.

And also Cross supporters’ claims:

• It is not feasible to have a native builder of all the architecures you want to support.
• In embedded world target are typically very small machines, build on them will take forever.
• Cross builds can give the same exact result as Native compilation.
• No bonds or ties, you can build anywhere, for anything.

Let’s address these claims and let’s see how things really are.

Do cross builds produce same binaries as native builds?

Short answer:

They can, but it won’t typically happen.

It can be actually demonstrated that if cross and native run:

• same toolchain
• controlled environment
• And you add the secret sauce (compiler flags)

The result can be bitwise identical binaries.

Here comes the fun part: same toolchain and controlled environment are not enough to have bitwise identical executable. You might need to add the ‘secret sauce’ and control sources of non-determinism to approach reproducible builds.

But the important point is that for “identical” .text and .data same toolchain and controlled environment can be enough.

Common reasons why binaries can differ despite same toolchain and controlled environment are:

• Timestamps differences: build time embedded is embedded in ELF comments and __DATE__, __TIME__ could be used by both compiler and usercode.
• Hostnames / usernames: embedded in debug info and sometimes in build notes.
• Compiler fingerprints: Compilers are like Dogs, they use to pee where the step, which means: linker version, build IDs, etc.
• File ordering: This one is subtle and evil. make collects object files, filesystem returns them in some order, and linker consumes them in that order. Change file creation order leads to changes in binary layout.
• Parallel build non-determinism: modern builds are heavily parallelized (make -j, LTO, linker jobs). Jobs don’t always finish in the same order, and neither does the linker. Sometimes, your binary depends on which core was faster that day.

If you want identical binaries, you need to remove as many sources of entropy as possible.

Part of the entropy can be handled with compiler and linker flags, using what I called the secret sauce.

Syndrome	Cure	Notes
Timestamps	SOURCE_DATE_EPOCH	Instead of using “Now,” Compilers will use the timestamp you provide.
Timestamps	-Wno-builtin-macro-redefined	Allows you to manually redefine __DATE__ and __TIME__ via -D flags if you can’t use SOURCE_DATE_EPOCH
Polluting sections	-fno-ident	Tells the compiler not to generate the .ident or .comment section containing the compiler version.
Build Paths	-fdebug-prefix-map=/old/=/new/	Replaces absolute paths in debug info and in __FILE__ with a generic string.

But some sources of non-determinism live in the build system itself, and require discipline, not just flags.

Does this really exist cross compiler configuration problem?

Before we can understand why people distrust cross-compilation, we need to talk about a legendary creature Autotools.

Specifically:

autoconf
automake
./configure

If you’ve ever typed:./configure && make …you’ve already met it.

So, what problem Autotools was solving?

To understand that, we need to go back to the 90s when there was no:

• standardized Linux
• consistent libc behavior
• reliable compiler features
• unified system APIs

Instead, you had a nice zoo:

• different UNIX systems
• different compilers
• different headers
• different kernel quirks

Back then it was common to answering questions like:

• Does this system have strlcpy?
• Is long 32 or 64 bits?
• Does malloc(0) behave sanely?
• Is fork() broken here?

And the answer was often: Maybe

Autotools solved the problem by saying:

Don’t guess. Ask the system.

But instead of asking politely, they do this:

• Generate a tiny C program
• Compile it
• Run it
• Observe the result
• Decide how to build the real software

Example:

int main() {
    return sizeof(long) == 8 ? 0 : 1;
}

If it runs and returns 0 then we know long is 64-bit.

It was clever, for its time.

Because documentation was unreliable, headers lied and systems were inconsistent.

But as you might guess, this model does not meet cross compilation because you build on x86, but your target is ARM, to name one.

Autotools would compile a test program… …and then tries to run it.

This is the origin of some native supporters’ claims about about configuration.

Question is: are we still at that point?

The answer? Spin the wheel, because Autotools has evolved, but it hasn’t entirely escaped its roots.

Current autotool can:

• use cached answers: Instead of running code the code above it can use ac_cv_sizeof_long=8
• Cross-aware configuration: by telling it --host=aarch64-linux-gnu it disables some runtime checks, switches to safer assumptions
• Replace execution with knowledge: many checks were rewritten to inspect headers and use known architecture traits

But it still, sometimes, when execution fails, just… guesses…

and here’s where subtle bugs, wrong assumptions, architecture-specific breakage come.

When the system can’t answer at build time, we can either guess… or we can simulate the answer. Therefore, sometimes if you want to cross build, you need also to emulate using qemu.

Qemu enables autotools to:

• Probe executables can run successfully under emulation
• Configuration becomes more accurate and less dependent on assumptions
• Complex or runtime-dependent checks become feasible again

This effectively restores the original Autotools probing model, even in cross environments.

Library pollution, what is this thing.

Native supporters calims cross builds to suffer from library pollution.

During a cross build, the build system accidentally links against host libraries instead of target libraries.

So you think you built for ARM (or another target), but headers came from one place, libraries came from another and the result is inconsistent or outright broken.

Making it less “pollution” and more “you accidentally linked against the wrong universe”.

Library pollution is usually discussed in the context of cross-compilation, but the underlying issue is broader:

it’s really about insufficient isolation between dependency domains.

At its core, library pollution happens whenever: the compiler and linker pick up inconsistent headers and libraries from different environments.

That can occur in:

• cross builds (host vs target)
• native builds (system vs custom prefixes, multiple versions, etc.)

The difference is not whether it happens, but how it fails.

The real root cause: isolation failure

The key variable is: > how well your build environment is isolated

This can involve:

• sysroots
• prefix separation (/usr vs /usr/local vs /opt/…)
• environment variables
• build system discipline

When isolation breaks, pollution becomes possible…

regardless of cross/native.

But then, why only the cross build gets the spotlight?

Cross builds make the problem obvious because architectures differ and binaries are often not runnable on the host. So when pollution happens linker may fail immediately or binary won’t even start or crashes instantly.

In practice the failure is loud and early… And you can’t deploy your package.

But beware that if happen on native builds thing can be even worse because headers and libraries are often compatible enough to compile, ABI differences may be subtle and symbol versions may partially match.

The result is silent inconsistency, executable can work, and perhaps only break on a few corner cases.

Cross vs Native pollution chart:

Aspect	Cross build	Native build
Architecture mismatch	Yes	No
Build failure likelihood	High	Low
Runtime failure	Immediate	Delayed / conditional
Debug difficulty	Moderate	Very high
Typical symptom	Won’t start / crashes instantly	Sporadic crashes

Intermediate prducts of the building

In a native build, all generated tools, scripts, and metadata are produced for the same environment in which they will later be used. This makes it possible to package and reuse the entire build context, such as kernel headers, symbol information, and helper binaries… for downstream tasks like out-of-tree module compilation.

In contrast, cross builds split the world into host and target components.

While the final binaries target the desired architecture, many intermediate tools are built for the host and are not portable. This makes the resulting build artifacts harder to redistribute or reuse in a different environment, especially without reconstructing the original build setup.

The concrete example for the Kernel (few other may exist) When you build something like the Linux kernel, you’re not just producing:

• vmlinuz / *Image

You’re also generating a whole ecosystem of intermediate tools and metadata, such as:

• scripts/ utilities (e.g. modpost, genksyms)
• generated headers (include/generated/*)
• configuration state (.config, Module.symvers)
• build glue for modules (kbuild infrastructure)

These are not optional, they are required for:

building out-of-tree modules later

Some of these artifacts are binaries targeting the build architecture. Because in users’ world OOT modules are built on the machine that is going to use them cross build artifacts are not usable.

Testing

Testing is one of the weak, if not the weakest, point of the cross build. In native build, it’s easy just the test and you’re done.

On the Cross builds, on the other hand, testing is possible, but you need to work a little.

The use of QEMU user-mode emulation, in particular qemu-user-static, provides a practical way to execute binaries built for a foreign architecture directly on a host system. This capability is especially valuable in cross-compilation workflows, where executing target binaries is otherwise not possible.

In a traditional cross-compilation setup, test binaries produced during the build process cannot be executed because they target a different architecture than the host. This limitation affects both:

• Test suites (unit and integration tests)
• Build-time probing, such as those performed by GNU Autoconf

By integrating QEMU user-mode with the Linux binfmt_misc mechanism, foreign binaries can be executed transparently. When combined with a target root filesystem (sysroot), this allows:

• Running test suites for the target architecture without dedicated hardware
• Executing Autotools-generated probe binaries, enabling runtime checks (e.g. via AC_RUN_IFELSE) that would otherwise fail during cross-compilation

As a result, builds can behave similarly to native builds, reducing the need for manual overrides such as cached configuration variables.

Emulation Performance and setup Considerations

Execution under QEMU user-mode is generally slower than native execution due to instruction translation. However:

• The overhead is often acceptable for build-time testing.
• In some cases, the difference is negligible compared to overall build time.
• When targeting inherently slower architectures, the perceived slowdown may be less significant, and in some constrained comparisons the host may still outperform real hardware.

Despite this, performance-sensitive or timing-dependent tests may exhibit different behavior under emulation.

While QEMU simplifies execution, it introduces additional setup requirements:

Configuration of binfmt_misc to associate foreign binaries with the appropriate QEMU interpreter Provision of a complete target root filesystem, including: * Dynamic linker * Shared libraries * Correct filesystem layout

This effectively shifts complexity from build configuration (e.g. Autotools cross-compilation issues) to environment preparation.

Summary of Cross vs Native build

This is a comparison synthesis between the two approaches. No judgement is given by choice, native vs build is a religion, and I don’t want to be hostile to any.

Issue	Native	Synthetic	Cross	Notes
Artifact Reproduciblity	Reproducible	=	Reproducible	Both build methods can lead to reproducible artifacts.
Configuration problems	Mature	<==	Minor problems exist	Historically Autotool is using host execution go guess the configuration. In cross environment this is not possible, butnewer versions of autotool support cross build enviroment through caching. Other approaches use qemu-user-static to make some complicated autotool setup work on cross environments.
Binary Artifacts Quality	Same	=	Same	If setup correctly artifact can be bitwise identical.
Build Enviroment Pollution	Bad Setup	=	Bad Setup	Not a Cross build specific issue. it is more a sysroot isolation problem. though, failing this, can have worse consequences in Native than cross.
Build Product Testing	Easy	<==	Complex	Test a build product, means executes it under some conditions. Native enviroment execute the artifact is trivial, in cross enviroment it is not, unless you cheat with qemu-user-static.
Build performance (speed)	Variable	==>	Optimal	It is a fact, some minor architectures are slow. Iot devices can target slow architectures where build can be challenging.
Per architecture builder	Issues	==>	none	A Linux distro targeting many architecures using Native compilation, needs to have at least a worker per architecure, and this scales bad.

Linux build systems overview

As last step, lets see ta some of more common build system that distro uses to generate linux.

System / Tool	Used by	Native / Cross	Core tech	Notes
Koji	Fedora, RHEL, CentOS	Native (per-arch workers)	RPM, Mock, chroot, distributed scheduler	Centralized build farm; builds SRPMs on target-arch machines
Mock	Fedora/RHEL (via Koji)	Native (isolated)	chroot, RPM	Creates clean build roots; prevents host contamination
Open Build Service	openSUSE, SLE	Mostly native (multi-arch workers), supports cross	RPM/DEB, chroot, distributed builds	Public multi-distro build platform
Debian buildd network	Debian	Native	dpkg, debhelper, buildd daemons	Distributed autobuilders per architecture
Launchpad	Ubuntu	Native	dpkg, chroot/containers	CI + packaging + build farm
Arch Build System	Arch Linux	Native (mostly)	PKGBUILD, makepkg	Simple recipe-based builds; less centralized
Portage	Gentoo	Native (user builds), supports cross	ebuild, bash	Source-based; builds happen on user machine
Buildroot	Embedded (various)	Cross (primary)	Make, Kconfig	Builds full rootfs + toolchain; minimal, deterministic
Yocto Project / BitBake	Embedded, industrial distros	Cross (primary)	BitBake, metadata layers	Highly configurable, large-scale embedded builds
OpenWrt build system	OpenWrt	Cross	Make, custom infra	Router-focused; cross toolchains + package feeds
Nix	NixOS, others	Hybrid (native + cross)	functional DSL, isolated store	Strong reproducibility model
GNU Guix	Guix System	Hybrid (native + cross)	Scheme, functional builds	Similar to Nix, emphasizes purity

How’s the future?

Newer ecosystems like Rust or Go make cross-compilation feel almost… reasonable.

They remove a lot of the historical friction, but they don’t change the fundamentals. You still need to control your environment, your dependencies, and your assumptions. The difference is that now the toolchain fails less creatively.

Native vs cross is still a debate. The difference is that now we have better tools… and the same old problems.

When a Homelab Migration Awakens Your Inner Networker

My blog has never been particularly homogeneous. This post is probably weirder than usual.

These days most people know me for work around Linux kernel safety and the Elisa Community. That’s the professional side. But like many engineers, I also have a homelab where I experiment, break things, and occasionally learn something useful while pretending it’s just procrastination.

This story comes from when I decided my network needed an upgrade.

And it unexpectedly reminded me why networking used to be my field in the first place.

The Pascal Incident

To explain why I even cared about this migration, we need to go back a bit.

I originally started as a Assembly-Pascal programmer.

In 1997 I had a small disagreement with my computer architecture professor about whether Pascal could be used for low-level work.

Naturally, the only rational response was to write an arcade machine emulator in Pascal.

This one: a Space Invaders emulator. Which I proudly demonstrated to the professor. I still remember that conversation.

Me: Hey professor, look what I built using Pascal. If used properly, you don’t actually need C.

To be fair, this was only partially true. I did use the Pascal compiler to build the executable, but most of the low-level work ended up written as large blocks of inline assembly inside the Pascal source. Anyway, I showed it to the professor.

The professor watched the screen for a moment.

Professor: Well yes. Nice game.

That was not the reaction I expected.

Me: Look, I didn’t make this game. Someone in Japan did… back in 1978.

At that point the professor looked mildly annoyed.

Professor: Then what exactly are you showing me today?

Which honestly left me unarmed.

I would have liked to answer: the emulator, you idiot… But I didn’t. For the record it neither improved my grade nor impressed the professor. But the experiment left me with a fascination for understanding systems by emulating them. Which later led to another emulator.

Instead of arcade hardware, I tried to emulate a Cisco router.

That project became my computer engineering master’s thesis: a Cisco 2500 router emulator, again written in Pascal… the last time before burying it forever.

Reverse engineering parts of the router firmware to understand the hardware architecture forced me to dive into routing logic…

That was basically my accidental entry point into network engineering. So yes, my networking career technically started with a router emulator written in Pascal. Which probably explains a few things.

Back to the Homelab

Fast forward a few decades.

The task that triggered this story looked simple enough:

Replace an old Cisco 3560, sitting in my lab and running almost continuously for 15 years with something newer… a Ruckus ICX-7150, and finally enter the gigabit era.

Which mostly meant replacing one relic with a slightly younger relic. The 3560 was already ancient. The ICX-7150 is not ancient yet… but it’s already end-of-sale, so it’s clearly on the right trajectory.

At first glance the CLI felt familiar. Cisco and Ruckus commands look similar enough that you instinctively think:

“Ah yes. I know exactly what this does.”

That confidence usually lasts about five minutes. After that you discover that while the commands look similar, the behavior is not. Part of the reason is historical.

This network used to be entirely Cisco. Phones, switches, everything. In that environment life was easy. Phones learned their VLAN via CDP, everything powered correctly, and nobody asked too many questions.

The moment I started replacing pieces, I discovered which parts of that convenience were standards, and which parts were simply Cisco assumptions.

Some of that I expected. Some of it I discovered the hard way.

The Troubleshooting Story

The setup was simple:

• Cisco 7940 phones
• SCCP firmware
• Asterisk
• configuration via TFTP
• DHCP instead of CDP

The goal was to run the phones in a vendor-neutral network.

Initially everything worked. Phones booted and registered. Then they started doing something strange. They would run normally for a while…

disconnect… then reconnect.

Phone logs showed events like:

Code:8103

Packet captures showed:

• SCCP keepalive retransmissions
• TCP resets
• reconnect cycles

Then another symptom appeared: occasionally the phone did not respond to ARP requests. That pointed suspicion directly at the phone. Old 79xx firmware is known to be… quirky. So the investigation went there first.

The Firmware Detour

The phones were running:

P00308000100

Which triggered the usual Cisco firmware archaeology. Finding firmware for older Cisco devices is a bit like looking for historical artifacts: you know it exists, you’re just not sure where.

Eventually a newer firmware image was located and installed.

The upgrade worked… And the problem remained.

So the firmware theory collapsed.

At this point the investigation had already consumed a fair amount of time… and it was still focused on the phone.

The Turning Point

The turning point came after a longer packet capture session. Something about the ARP traffic looked wrong. So the next step was obvious: check what the switch actually saw. A mirror session was configured on the ICX and the phone port was captured.

The mirrored traffic showed:

• unicast traffic on VLAN 2
• broadcast traffic from the phone

But something was missing.

The ARP broadcast from Asterisk never appeared. To confirm this, traffic was captured in several places:

• the ICX mirror
• a VLAN-2 host
• a Cisco mirror upstream

The result was consistent. The ARP broadcast existed on the Cisco side, but never reached the ICX access port. At that moment the investigation pivoted. This was no longer a phone problem.

It was a switch problem.

Finding the Culprit

To confirm the path, the Cisco 3560 uplink was mirrored. The ARP request was clearly visible leaving the Cisco trunk:

Who has 10.1.5.57? Tell 10.1.5.1

But the broadcast never appeared on the ICX.

The packet was disappearing between the switches.

Attention turned to the ICX uplink configuration:

vlan 2
 tagged ethernet 1/2/2
!
interface ethernet 1/2/2  
 dual-mode  

VLAN configuration looked correct: VLAN 2 tagged 1/2/2

Various things were tested:

• VLAN membership
• DHCP snooping
• spanning tree
• broadcast filtering

Nothing changed.

The Fix

Eventually attention returned to one small detail on the ICX uplink:

interface ethernet 1/2/2  
 dual-mode

In FastIron (Brocade/Ruckus) this is roughly equivalent to the native VLAN on a Cisco trunk.

And in fact, if you come from a Ruckus-native background, that line probably would not raise any alarms at all.

The only reason it caught my attention is precisely because I wasn’t thinking like a Ruckus engineer.

So, mostly out of prejudice rather than conviction, I tried removing it.

Immediately:

• ARP broadcasts appeared
• VLAN-2 hosts received them
• the phone stopped disconnecting

The note I wrote at the time was simply:

was that! no dual-mode on 1/2/2 solved the problem.

And that closed the investigation.

Which strongly suggests the real root cause was not the configuration itself, but likely a firmware bug in the switch triggered by that combination of trunking behavior.

NOTE: On Brocade/Ruckus FastIron switches, the dual-mode command is used to allow a port that carries tagged VLANs to also accept and transmit untagged traffic. Unlike Cisco trunks, which always include a native VLAN by default, a FastIron port with tagged VLANs normally sends only tagged frames unless dual-mode is enabled. The command itself takes no arguments and has remained syntactically stable across FastIron versions; it simply enables the ability for the port to carry untagged frames. The actual native (PVID) VLAN is determined separately by the port’s untagged VLAN membership, which can be changed under the interface with vlan-config move untagged vlan <vid>. For engineers familiar with Cisco, the combination of dual-mode plus the port’s untagged VLAN is functionally equivalent to configuring a native VLAN on a trunk.

A Note About AI

One final note.

I have been skeptical about AI tools for quite a while and avoided using them in my workflow.

This debugging session, and a few others, are changing my mind a bit.

The AI suggested ICX commands I didn’t know, reminded me of Cisco syntax I had not used in years, and helped narrow the search space.

Without that assistance I would probably have spent days digging through documentation.

Instead the whole problem unraveled in one night.

Which, for a homelab experiment, is an excellent outcome; and significantly better than spending a weekend reading switch manuals.

Cheri & Fil-C, are they the C memory-safe solution?

Looking at the past

Pointers Were Never Meant to Be Just Numbers

(And C Didn’t Kill That Idea… We Did)

Ask a C programmer what a pointer is, and you’ll almost certainly hear:

“It’s just a memory address.”

That statement is true in practice, but historically wrong.

The idea that a pointer is “just a number” is not a law of computing. It is the result of a long chain of economic and engineering decisions that happened to align in the 1970s and 1980s.

Before that, and increasingly again today, a pointer was understood as something richer: a reference with rules, a capability, a guarded object.

And crucially: C did not invent flat memory. It merely adapted itself to it extremely well.

Before C: When Pointers Carried Meaning

Early computer systems had no illusion that memory access was harmless.

Machines such as the Burroughs B5000, the Unisys 1100/2200 series, and later the Lisp Machines all treated pointers as structured entities:

• bounds-checked
• tagged
• validated by hardware
• often unforgeable

A pointer was not an integer you could increment freely. It was closer to a capability, a permission to access a specific object.

This wasn’t academic purity. It was a necessity:

• multi-user systems
• shared memory
• batch scheduling
• safety over speed

These machines enforced correctness by design.

C Did Not “Flatten” Memory… It Adapted to It

It’s tempting to say:

“C introduced flat memory and unsafe pointers.”

That’s not quite true.

C was designed on the PDP-11, a machine that already had:

• a flat address space
• no hardware memory tagging
• no segmentation protection at the language level

C didn’t invent this model… It embraced it.

But here’s the key point that often gets missed:

C was explicitly designed to be portable across architectures with very different memory models.

And that includes machines that did not have flat memory.

C on Non-Flat Architectures: The Forgotten Chapter

C was successfully implemented on:

• segmented machines
• descriptor-based systems
• capability-like architectures

Including Unisys systems, where pointers were not simple integers.

As documented in historical work (and summarized well in begriffs.com – “C Portability”), early C compilers adapted to the host architecture rather than forcing a universal memory model.

On Unisys systems:

• pointers were implemented using descriptors
• arithmetic was constrained
• bounds and access rules were enforced by hardware
• the compiler handled translation

This worked because the C standard never required pointers to be raw addresses.

It required:

• comparability
• dereferenceability
• consistent behavior

Not bit-level identity.

Even Henry Rabinowitz warned, in Portable C, that assuming pointer arithmetic behaved like integer arithmetic was already non-portable, even in the late 1980s.

So what changed?

The Real Shift: Economics, Not Language Design

The shift didn’t come from C.

It came from:

• cheap RAM
• fast CPUs
• simple pipelines
• RISC philosophy
• UNIX portability

Flat memory was faster to implement and easier to optimize.

Once x86 and similar architectures dominated, the hardware stopped enforcing:

• bounds
• provenance
• validity

And since C mapped perfectly onto that model, it became the dominant systems language.

From that point on:

• pointers became integers
• safety became a software problem
• memory bugs became a security industry

Not because C demanded it, but because the hardware no longer helped.

The Long Detour We Are Now Undoing

For decades, the industry tried to patch this with:

• ASLR
• stack canaries
• DEP
• sanitizers
• fuzzers

All useful. None fundamental.

They treat symptoms, not causes.

Which brings us back, full circle, to the idea that started this story:

A pointer should carry metadata.

A Short Detour: Even x86 Wasn’t Always Flat

Before moving forward, it’s worth correcting one more common simplification.

Even the architecture most associated with “flat pointers”, x86, did not start that way.

In real mode, x86 used segmented addressing:

physical_address = segment × 16 + offset

This meant:

• pointers were effectively split into two components
• address calculation wasn’t trivial
• different segments could overlap
• the same physical memory could be referenced in multiple ways

It wasn’t a capability system, there were no bounds or permissions, but it was a reminder that pointer arithmetic was never universally “just an integer add.”

What changed wasn’t the hardware’s ability to support structure.

What changed was that:

• segmentation was seen as inconvenient
• flat addressing was faster
• compilers and operating systems optimized for simplicity

By the time protected mode and later 64-bit mode arrived, segmentation had been mostly sidelined. The industry standardized on:

Flat memory + software discipline

That decision stuck.

And that’s the world CHERI and FIL-C are now challenging.

Are CHERI and FIL-C fixing the problem?

The Common Idea Behind CHERI and FIL-C

At first glance, CHERI and FIL-C look very different.

One changes the CPU. The other changes the compiler.

But conceptually, they start from exactly the same premise:

A pointer is not an address. A pointer is authority.

Everything else follows from that.

The Shared Heritage: Capability Thinking

Both CHERI and FIL-C descend from the same historical lineage:

• Burroughs descriptors
• Lisp machine object references
• Capability-based operating systems
• Hardware-enforced segmentation

The core idea is simple:

A program should only be able to access memory it was explicitly given access to.

That means a pointer must carry:

• where it points
• how far it can go
• what it is allowed to do
• whether it is still valid

In other words: metadata.

The only real disagreement between CHERI and FIL-C is where that metadata lives.

Cheri: Making Capabilities a Hardware Primitive

CHERI takes the most direct route possible.

It says:

“If pointers are capabilities, the CPU should understand them.”

So CHERI extends the architecture itself.

A CHERI pointer (capability) contains:

• an address (cursor)
• bounds
• permissions
• a validity tag

The tag is critical:

• it is stored out-of-band
• it cannot be forged
• it is cleared automatically if memory is corrupted
• the CPU refuses to dereference invalid capabilities

This means:

• no buffer overflows
• no out-of-bounds accesses
• no forged pointers
• no accidental privilege escalation

And all of this happens without software checks.

The hardware enforces it.

This is not “fat pointers” in the C++ sense. This is architectural memory safety.

Importantly, CHERI preserves C semantics:

• pointers still look like pointers
• code still compiles
• performance is predictable

But the machine simply refuses to execute illegal memory operations.

It’s the return of the capability machine, this time built with modern CPUs, caches, and toolchains.

By design, CHERI enforces only what can reasonably belong to the instruction set, leaving higher-level memory semantics to software.

FIL-C: Capability Semantics Through Compiler and Runtime

FIL-C starts from the same premise:

“C pointers need metadata.”

But instead of changing the hardware, it changes the compiler and runtime.

This choice allows FIL-C to enforce stronger guarantees than CHERI, but at the cost of changing not only pointer representation, but also object lifetime and allocation semantics.

In FIL-C:

• pointers become InvisiCaps
• bounds are tracked invisibly
• provenance is preserved
• invalid accesses trap

From the programmer’s point of view:

• it’s still C
• code still compiles
• the ABI mostly stays intact

From the runtime’s point of view:

• every pointer has hidden structure
• every access is validated
• dangling pointers are detected

FIL-C and CHERI start from the same idea: pointers as capabilities, but deliberately apply it at very different semantic depths.

At this point, the similarity ends.

While CHERI limits itself to enforcing spatial safety and capability integrity, FIL-C necessarily goes further. In order to provide temporal safety, FIL-C must change the object lifetime model itself.

In FIL-C, deallocation is decoupled from object lifetime: freed objects are often quarantined, delayed, or kept alive by a garbage collector. Memory reuse is intentionally conservative, because temporal safety cannot coexist with eager reuse.

This is not an implementation choice but a semantic requirement, and it has consequences that go well beyond pointer representation.

The Reality Check: It’s a Runtime, Not Just a Compiler

It is important to distinguish FIL-C from a simple “safe compiler”. Because it enforces temporal safety via an object lifetime model, it must bypass standard allocators like glibc. This means the high-performance concurrency optimizations (arenas, caches) that developers expect are gone, replaced by a managed runtime.

Furthermore, because FIL-C requires this complex runtime support, it is currently a user-space tool. Using it to build, for example, a kernel like Linux is architecturally unfeasible.

For standard userspace applications, the trade-offs can be summarized as follows:

Aspect	CHERI	FIL-C
Enforcement	Hardware	Software
Pointer metadata	In registers & memory	In runtime structures
Performance	2%-5% overhead	50%–200% overhead
Deployment	Requires new hardware	Works today

The higher cost of FIL-C is not primarily due to pointer checks, but to the changes required in allocation, lifetime management, and runtime semantics to enforce temporal safety.

CHERI makes the CPU safe. FIL-C makes the language safe.

Same Idea, Two Execution Models

CHERI and FIL-C are not competitors. They are two implementations of the same philosophy.

They both assert that:

• pointer provenance matters
• bounds must be enforced
• safety must be deterministic
• memory errors are architectural, not stylistic

They differ only in where that logic lives.

You can think of it this way:

• CHERI -> capabilities in silicon
• FIL-C -> capabilities in software
• MTE -> capabilities with probabilities

Different tradeoffs. Same destination.

Why This Matters Now: The ELISA Perspective

At first glance, CHERI and FIL-C may look like academic exercises or long-term research projects. But their relevance becomes much clearer when viewed through the lens of ELISA.

ELISA exists for a very specific reason: Linux is increasingly used in safety-critical systems.

That includes:

• automotive controllers
• industrial automation
• medical devices
• aerospace and avionics
• robotics and energy infrastructure

And Linux, for all its strengths, is still fundamentally:

A large C codebase running on hardware that does not enforce memory safety.

The Core Tension ELISA Faces

ELISA’s mission is not to redesign Linux.

It is to:

• make Linux usable in safety-critical contexts
• support certification efforts (ISO 26262, IEC 61508, etc.)
• improve predictability, traceability, and robustness
• do this without rewriting the kernel

That creates a fundamental tension:

• Linux is written in C
• C assumes unsafe pointers
• Safety standards assume bounded, analyzable behavior

Most current ELISA work focuses on:

• process isolation
• static analysis
• restricted subsets of C
• coding guidelines
• runtime monitoring
• testing and verification

All valuable. All necessary.

But none of them change the underlying truth:

The C memory model is still unsafe by construction.

Why Cheri and FIL-C Enter the Conversation

CHERI and FIL-C do not propose rewriting Linux.

They propose something more subtle:

Making the existing C code mean something safer.

This matters because they address a layer below where most safety work happens today.

Instead of asking:

• “Did the developer write correct code?”

They ask:

• “Can the machine even express an invalid access?”

That’s a fundamentally different approach.

CHERI in the ELISA Context

CHERI is interesting to safety engineers because:

• It enforces memory safety in hardware
• Violations become deterministic faults, not undefined behavior
• It supports fine-grained compartmentalization
• It aligns well with safety certification principles

But CHERI is also realistic about its scope:

• It requires new hardware
• It requires a new ABI
• It is not something you “turn on” in existing systems

Which means:

Cheri is not a short-term solution for ELISA, but it is a reference model for what correct looks like.

It provides a concrete answer to the question: “What would a memory-safe Linux look like if we could redesign the hardware?”

FIL-C in the ELISA Context

FIL-C sits at the opposite end of the spectrum.

It:

• runs on existing hardware
• keeps the C language
• enforces safety at runtime
• integrates with current toolchains

This makes it immediately relevant as:

• a verification tool
• a debugging platform
• a reference implementation of memory safety
• a way to experiment with safety properties on real code

But it also comes with trade-offs:

• performance overhead
• increased memory usage
• reliance on runtime checks

So again, not a drop-in replacement, but a valuable experimental lens.

The Direction Is Clear (Even If the Path Is Long)

The real contribution of CHERI and FIL-C is not that they make C safer by rewriting it.

It is that they show memory safety can be improved by changing the semantics of pointers, while leaving existing code largely untouched.

This distinction matters.

Large systems like Linux cannot realistically be rewritten. Their value lies in the fact that they already exist, have been validated in the field, and continue to evolve. Any approach that requires wholesale code changes, new languages, or a redesigned programming model is unlikely to be adopted.

CHERI and FIL-C take a different approach. They act below the source level:

• redefining what a pointer is allowed to represent
• enforcing additional semantics outside application logic
• turning undefined behavior into deterministic failure

In doing so, they demonstrate that memory safety can be introduced beneath existing software, rather than imposed on top of it.

That insight is more important than either implementation.

It shows that the path forward for Linux safety does not necessarily run through rewriting code, but through reintroducing explicit authority, bounds, and permissions into the way memory is accessed, even if this is done incrementally and imperfectly.

Looking Forward

Neither CHERI nor FIL-C is something Linux will adopt tomorrow.

CHERI depends on hardware that is not yet widely available and will inevitably influence ABIs, compilers, and toolchains. FIL-C works on current hardware, but with overheads that limit its use to specific contexts.

What they offer is not an immediate solution, but a reference direction.

They suggest that meaningful improvements to Linux safety are possible if we focus on:

• enforcing memory permissions more precisely
• narrowing the authority granted by pointers
• moving checks closer to the hardware or runtime
• minimizing required changes to existing code

This leaves room for intermediate approaches: solutions that do not redefine the language, but instead use existing mechanisms, such as the MMU, permission models, and controlled changes to pointer usage, to incrementally reduce the scope of memory errors.

In that sense, CHERI and FIL-C are less about what to deploy next and more about what properties future solutions must have.

They help clarify the goal: make memory access explicit, bounded, and enforceable… without rewriting Linux to get there.