Hetzner is a price-competitive and conceptually simpler alternative to AWS and the other hyperscalers for the small orgs and teams that I tend to work with.

NixOS is a declarative, reproducible operating system that turns 'works on my machine' into 'works on every machine' - infrastructure-as-code that's finally achievable for the lean teams that I tend to work with.

But Hetzner doesn't ship NixOS images, just the standard debian, ubuntu, and rhel clones.

Most folks resort to using nixos-infect or nixos-anywhere to transmogrify a Debian or Ubuntu instance into NixOS. The more ambitious reach for Packer and rescue mode, which requires munging around manually with Hetzner's rescue mode.

All three share the same fundamental ritual: provision a VM, SSH in, and overwrite its soul with NixOS. These approaches work, but you pay the conversion tax every time you spin up a new VM - and the demons don't work for free nor are they particularly fast.

But the winds have shifted and three things have fallen into place:

First, hcloud-upload-image was released in 2024. It's a simple Go tool that handles the soul conversion for you. hcloud-upload-image takes a disk image as input and side effects Hetzner with enough conviction that a Snapshot materializes.

hcloud-upload-image upload \
--image-path  result/nixos-image-25.11.x86_64-linux.img \
  --architecture x86
... long wait while dark arts are performed...
Uploaded Image: 123467

Second, Stéphan Kochen (gh) opened nixpkgs PR #375551 to bring native Hetzner Cloud image building into nixpkgs. The PR packages hcloud-upload-image, adds a NixOS image builder config, and includes his own systemd-network-generator-hcloud tool for IPv6 autoconfiguration.

Third, FlakeHub Cache, available since late 2024, changes how you deploy NixOS configurations. Normally, deploying a NixOS config means evaluating the entire flake on the target machine - slow, memory-hungry, and painful on a cheap VPS. FlakeHub pre-computes resolved store paths when you publish your flake, so fh apply nixos skips evaluation entirely. Push your config from CI, deploy to your server with one command, and the configuration arrives in seconds rather than minutes.

Separately, these tools solve pieces of the puzzle. Together, they enable a workflow that didn't exist before: Building reusable NixOS images for Hetzner Cloud!

I made a flake that packages these pieces into a ready-to-use solution for Hetzner Cloud images. I wrote the glue; the people above wrote the magic. The code is at outskirtslabs/nixos-hetzner.

As of writing, PR #375551 is still open. My flake vendors the relevant pieces so you don't need to wait for merge in the meanwhile.

HCLOUD_TOKEN=...your hcloud token...
ARCH=x86_64-linux # or aarch64-linux
HCLOUD_ARCH=x86   # or arm

nix build "github:outskirtslabs/nixos-hetzner#diskImages.$ARCH.hetzner" --print-build-logs

# inspect the image
ls result/*
IMAGE_PATH=$(ls result/*.img 2>/dev/null | head -1)

# upload to hetzner cloud
hcloud-upload-image upload \
    --image-path="$IMAGE_PATH" \
  --architecture="$HCLOUD_ARCH" \
  --description="nixos-hetzner image"

With that you have a NixOS Snapshot in your Hetzner Cloud console that you can clickops into a fresh VM. Once you've booted a VM from the image, you can authenticate with FlakeHub and use fh apply to deploy configurations directly.

FlakeHub Cache makes this fast - really fast. Instead of rebuilding or waiting for slow binary cache downloads, your configurations deploy in seconds.

You'll need to bring your own paid Hetzner and FlakeHub accounts of course.

For a more complete example, with Terraform/Opentofu and Github Actions, check out the outskirtslabs/nixos-hetzner-demo. It builds on nixos-hetzner and showcases a full continuous deployment methodology with NixOS.

Reply via: Email · Bluesky

Getting the IP address of a client making a request to your web application sounds like it should be embarrassingly simple. It's not.

If you think it's just a matter of checking :remote-addr in your request map or pulling a value out of the X-Forwarded-For header, you've just opened yourself up to IP spoofing vulnerabilities that would make a pentester giggle with delight.

There is no easy, or simple, solution. The problem is inextricably tied with your specific deployment environment: whether you're behind a reverse proxy, a CDN (or both), which headers your infrastructure uses, and crucially, which ones you can actually trust. None of this information lives in the HTTP request itself. You need out-of-band knowledge about your infrastructure topology to correctly parse the maze of forwarding headers without letting an attacker spoof their way past your IP-based security controls.

Armed with that knowledge you still need to correctly parse a tangled mess of headers, deal with IPv6 addresses with ports, know which header to trust when multiple ones exist, and implement all the validation logic to prevent spoofing attacks.

I've been wrestling with this problem for years across various client projects. After implementing the same buggy patterns over and over (and watching other developers do the same), I finally decided to solve this properly.

ol.client-ip is a Ring middleware (with zero deps!) that handles all this complexity so you don't have to. It's essentially a Clojure port of the excellent realclientip-go implementation, which itself was born from the collective frustration of developers who were tired of getting this wrong.

Who cares?

Why do we need IP-based security controls?

Well you often don't! And yes, in today's landscape of easy VPNs, CGNAT, the client's ip address is not a bullet-proof identifier or security mechanism.

But sometimes it is very helpful for rate limiting, geographic compliance requirements, abuse prevention, audit trails, or fraud detection. All of these are made easier when you can, with some confidence, know the actual IP address making the request.

Please don't reach for IP-based controls first thing, they are problematic. But if you do, understand how to protect your self from trivial IP spoofing.

Why This Is Actually Hard

Here's the thing about modern web infrastructure: your application almost never talks directly to the actual client. There's usually a reverse proxy, or three. Possibly a load-balancer. Maybe a CDN. All of the above?

Each of these helpful intermediaries likes to "help" by adding headers telling you about the original client. The problem? Any client can also set these headers. Watch this:

curl -H "X-Forwarded-For: 1.2.3.4" https://your-app.com

Does your app use http-kit as a web server and then look at :remote-addr? Boom. I just tricked your app in beliving I'm from IP 1.2.3.4. If you're using that IP for rate limiting, geolocation, or (heaven forbid) authentication decisions, you're in trouble.

(My PR for this is here here by the way, I hope it gets merged soon. It doesn't "solve" the problem completely it just makes it harder to shoot your self in the foot if you twiddle a non-default option.)

Often you hear advice like "just use the leftmost IP in X-Forwarded-For." This is terrible advice. It's trivially spoofable.

The slightly better advice is "use the rightmost IP that's not from your infrastructure." Getting warmer, but now you need to know what "your infrastructure" means, and that changes depending on your deployment.

Enter ol.client-ip

The library takes a strategy-based approach because (and this is my whole schtick!) there is no one-size-fits-all solution. Your network topology determines the correct strategy. The library just makes it easy to implement that strategy correctly.

(ns myapp.core
  (:require [ol.client-ip.core :as client-ip]
            [ol.client-ip.strategy :as strategy]))

;; Behind Cloudflare? They provide a trustworthy header
(def app
  (-> handler
      (client-ip/wrap-client-ip
        {:strategy (strategy/single-ip-header-strategy "cf-connecting-ip")})))

;; Behind exactly 2 proxies? Count backwards
(def app
  (-> handler
      (client-ip/wrap-client-ip
        {:strategy (strategy/rightmost-trusted-count-strategy "x-forwarded-for" 2)})))

;; Many more strategies available see usage documentation.

;; Your handler now has access to the "real" † client IP
(defn handler [request]
  (let [client-ip (:ol/client-ip request)]
    {:status 200
     :body (str "I actually know your IP: " client-ip)}))

† "Real" is doing a lot of work here. The epistemological problem isn't just technical. We're trying to identify "the client" through layers of network abstraction, where each proxy and NAT boundary forces us to accept increasingly indirect evidence of the originating request. What client-ip provides is the earliest source address in the chain that you've declared trustworthy through your configuration. It's not knowledge of the "true" client (whatever that means in a world of shared connections and VPNs), but rather the best available proxy for client identity given your position in the network topology.

For better or worse, this approach forces you to think about your network topology. You can't just slap it in and hope for the best. You have to make a conscious decision about which strategy matches your setup.

The strategies that Actually Matter

After implementing this for various clients, I've found that 90% of use cases fall into three categories:

Single trusted header: You're behind Cloudflare, Fly.io, or a single properly configured nginx. These services provide a header with the socket-level client IP that can't be spoofed (as long as clients can't bypass the proxy). This is the golden path if you have it.

Rightmost non-private: You have proxies in your private network (10.x.x.x, 192.168.x.x, etc.) and they all append to X-Forwarded-For. The rightmost non-private IP is your client. This works great until you put a proxy with a public IP in the chain, at which point you need...

Rightmost trusted count: You know exactly how many proxies are between the internet and your app. Count backwards that many IPs in the X-Forwarded-For chain. Simple, effective, but requires you to update the count if your infrastructure changes.

The library supports more strategies: trusted IP ranges, chain strategies for multiple paths, even the dangerous leftmost strategy for when you need to know what the client claims their IP is. But honestly? Start with these three.

Let's shave the yak

This library needs to parse ip addresses. It needs to do comparisons like is the ip address "192.168.1.22" in the trusted subnet "192.168.0/24".

Seems easy enough? But, like, what is an ip address? What can show up there?

Consider these valid IP addresses that might show up in your headers:

• 192.168.1.1 - Easy, IPv4
• 2001:db8::8a2e:370:7334 - IPv6, still manageable
• [2001:db8::1]:8080 - IPv6 with port notation
• fe80::1%eth0 - IPv6 with zone identifier (yes, that % is supposed to be there)
• ::ffff:192.0.2.1 - IPv4-mapped IPv6 address
• [fe80::1%25eth0]:8080 - URL-encoded zone identifier with port

And that's before malicious actors start sending you garbage like definitely-not-an-ip.com or 192.168.1.1.example.com hoping to trigger interesting behaviors in your parser.

I originally had hoped to just lean on Java's InetAddress.getByName(), which can do ip address parsing if you pass it raw ip addresses. Seems innocent enough, right?

Wrong. That method will happily perform DNS lookups (it is kind of what it is supposed to do) which will block your request thread. Very bad for throughput.

So I had to implement an IP address parser to guarantee there were no side effects. It's in the namespace [ol.client-ip.ip][ip-parse]. It might come in handy elsewhere..

Should You Use This?

If you're running a Clojure web app and you need to know client IPs (for analytics, rate limiting, geolocation, whatever), then, yea probably. At least read the documentation and existing literature do understand the problem space.

The library has zero dependencies beyond Clojure itself. It's about 1k SLOC (incl docstrings), thoroughly tested, and boring in all the right ways. It won't revolutionize your application, but it will prevent that awkward moment when you realize you've been rate-limiting your CDN instead of actual clients.

You can obtain ol.client-ip from Clojars or via a gitlib dep from the repo at github.com/outskirtslabs/client-ip.

Reply via: Email · Bluesky

I've been working with Datomic for years, and while I love its immutable, time-aware data model, deploying it has always been... an adventure. Unlike most modern databases that come with ready-to-use container images or OS packages, Datomic Pro arrives as a bare JAR file with some configuration examples.

If you are running it on bare metal or a standard Linux VM, then a custom systemd service file is all you need, but if you need to deploy it to a containerized environment you have a little work to do.

Datomic Pro is distributed as a Java application with a deployment model that requires manual assembly. It consists of a transactor (the server component), some storage backend (like a SQL db), and an optional console (web UI). Getting all these pieces working together requires:

1. JVM setup and configuration
2. Various storage backend configurations
3. Secret management
4. Classpath wrangling for custom drivers
5. Coordination between different components

And everyone ends up with custom shell scripts, Docker files, and deployment procedures that rapidly drift out of sync with the codebase. In other words, it's a perfect candidate for the reproducible deployment approach that NixOS offers.

Enter datomic-pro-flake, my attempt to bring sanity to Datomic Pro deployments using the power of Nix. If you've never heard of Nix, it's that declarative package manager that your one colleague won't shut up about. (Confession: I'm that colleague.)

What's in the Flake?

The datomic-pro-flake project provides three main components:

1. Nix Packages: Pure, reproducible builds of Datomic Pro components (transactor, console, and peer library)
2. NixOS Modules: Declarative configuration for running Datomic Pro on NixOS systems
3. Container Images: Run Datomic with your favorite container orchestrator (mine is docker compose), no nix required!

All three approaches are end-to-end tested in CI, which means you don't have to worry about whether they actually work. The tests exercise the package and module by booting a transactor, writing some datoms, and ensuring they can be read back. (I've spent enough late nights debugging non-functional database deployments for all of us.)

The Nix Package Approach

If you're already using Nix, this is probably what you're looking for. The flake provides:

# In your flake.nix
inputs.datomic-pro.url = "https://flakehub.com/f/Ramblurr/datomic-pro/$LATEST_TAG.tar.gz";

The packages are configurable through Nix's override pattern, allowing you to add custom Java libraries or native dependencies.

This is particularly handy if you need specific JDBC drivers or want to integrate with exotic storage systems.

What makes this approach special? The entire packaging and deployment becomes declarative, reproducible, and easily version-controlled.

One particularly neat feature is the automatic JRE slimming. Since Datomic only needs specific JDK modules, we use jdeps to analyze exactly what's needed and create a minimal runtime environment. This reduces the package size by hundreds of megabytes.

NixOS Modules: Set It and Forget It (But Don't Actually Forget It—This Is a Database After All)

For those running NixOS systems, the modules provide a simple approach:

# In your configuration.nix
services.datomic-pro = {
  enable = true;
  secretsFile = "/path/to/secrets";
  settings = {
    protocol = "sql";
    host = "0.0.0.0";
    port = 4334;
    # And any other Datomic settings... see README
  };
};

The module handles all the hardest parts:

• Systemd service configuration
• Runtime property generation
• Data directory setup
• Classpath configuration
• Ready to use with sops-nix, agenix, or whatever hand-rolled secrets management tool you have.

And there's a companion datomic-console module for running the web UI with similarly straightforward configuration.

Container Images: For Everyone Else

Not everyone is ready to drink the Nix Kool-Aid (though you should—it's delicious), so the flake also produces ready-to-use container images. Yes, good ol' Docker-compatible images that work exactly where you'd expect them to:

services:
  datomic-transactor:
    image: ghcr.io/ramblurr/datomic-pro:unstable
    environment:
      DATOMIC_PROTOCOL: sql
      DATOMIC_SQL_URL: jdbc:sqlite:/data/datomic-sqlite.db
      # ... more ....
    volumes:
      - ./data:/data
    ports:
      - 127.0.0.1:4334:4334

These images support both environment variables and file-based configuration, making them suitable for Kubernetes, Docker Compose, or other container orchestration systems. The README includes examples for deploying with various storage backends, including Postgres and SQLite.

Real-World Usage

I've had variations of this project in production for awhile, though it wasn't until relatively recently when Datomic Pro became totally free (as in beer) and the binaries released under the Apache 2.0 license that I felt comfortable making this public.

With the client projects I've used this on, it has significantly simplified deployments. The container image in particular "just works" and is grokkable by non-clojure non-jvm familiar operations folk.

FWIW the practical deployment patterns I've found useful:

1. Local Development: Use the container with a SQLite backend for quick local development
2. Production (Single-Node): Deploy with SQLite for simple single-node projects
3. Production (Multi-Node): Use PostgreSQL for scalable multi-node deployments

(For testing you're using datomic in-mem dbs, right?!)

Each approach is documented in the README with concrete examples you can start from.

Future Directions

The project is currently in a "stable but evolving" state. Before hitting 1.0, I'm planning to add:

• Better version pinning to prevent surprise database upgrades
• Out of the box example for NixOS modules w/ Postgres
• More storage backend examples and configurations
• Integration with secrets management tools like sops-nix
• Performance optimization for specific cloud environments

Why You Should Try It

If you're using Datomic Pro, the NixOS module or container image might save you hours of configuration headaches and provide a more reliable deployment process.

Even if you're just curious about Datomic, it offers the easiest way to get started without wrestling with configuration files.

Interested? Check out the GitHub repository.

And if you have questions or suggestions, feel free to open an issue or reach out to me on the Clojurians Slack (@Ramblurr).

As with any database deployment, remember: test thoroughly before pointing it at production data. Your future self will thank you.

Happy datom accretion!

Reply via: Email · Bluesky

...because someone had to bridge this particularly niche technological divide between JVM developer and the modern Linux desktop.

wayland-java's a set of Java bindings for libwayland and wayland-protocols that lets you create Wayland client (and server!) applications in pure Java - without writing a single line of C code.

If you've ever needed to call into native libraries from Java, you've likely experienced the special kind of frustration that is JNI. It's verbose, error-prone, and feels like you're working in two entirely different languages at once... because you are. Since JDK 1.1, released in the late 90s, this has been the inevitable tax paid by JVM-language developers wanting to interface with native code.

With the release of JDK 22, we got the finalized Foreign Memory & Memory API (formerly Project Panama). This is a serious upgrade for Java developers needing to interact with native code. The new FFM APIs mean: no more JNI ceremony, faster native interop, and no more brittle glue code in C, just a cleaner interface to the systems we need to talk to.

wayland-java leverages this new API to provide a proper Java interface to the Wayland protocol, making it possible to create graphical applications that run directly on Wayland compositors, while writing code in only Java (or other JVM hosted languages). The code reads like actual Java, because it is!

The codebase for this library isn't entirely new. It's a fork of Erik De Rijcke's 2015-era effort, which I've completely rewritten to use Project Panama for FFI. The original project was impressive in its own right, but the new FFM API makes this approach significantly cleaner and more maintainable.

However the project isn't production-ready, and probably won't be without more community or commercial interest. A client [funded me][ol] to create a proof-of-concept application using Wayland on Linux desktop through late 2024. The work was just R&D and won't be moving forward (for reasons unrelated to the tech!)

While I cannot opensource the full PoC project, I was able to extract out the wayland bindings into this little library.

If you're curious, the project provides several artifacts: client stubs, server stubs, shared stubs, a scanner tool that generates bindings from Wayland protocol XML descriptions, and a pre-packaged selection of protocols. You can generate your own protocols too, if the included ones don't meet your needs. The library isn't particularly high-level, you'll still need to understand wayland deeply.

I'm not expecting this library to take the world by storm. It's much more a case of "this exists, and if you need it someday, you won't have to suffer through building it yourself" The code is Apache-licensed, properly documented, and the build process won't make you question your career choices as a JVM developer. You'll need JDK 22+, Linux with libc, and libwayland available at runtime. (If you use nix, the included devshell will get it working out of the box)

Will the intersection of JVM developers and Wayland enthusiasts ever grow beyond dozens? Probably not. But for those few, perhaps this makes something possible that wasn't before.

And that's reason enough to build it.

Reply via: Email · Bluesky

It's strange working in tech and being a parent. I sell software as a solution, but in so many ways software is the cause of strife and problems in our lives. And specifically I mean family life.

YouTube, Spotifiy, and friends have no place near my pre-school aged children. It's not that I'm anti-technology — obviously — but I've seen too many children zombie-scrolling to want that for my own.

When I was a child we had cassette tapes. Boxes and boxes of cassete tapes. My sister and I each had a cheap little Sony cassette player/boom box and a pair of those cheap over-ear headphones that have the thinnest black foam as ear pads.

We spent hundreds, possibly thousands, of hours listening to stories and music. Sometimes replaying the same track or story 50 times in a row. Annoying for parents yes, but repetitive listening has been shown to be very important for language development.

I want to encourage an interest in stories and music while letting my children have control of their media. In 2020s, how do I give a similar experience to my children without a sketchy algorithm or screens?

Commercial Off the Shelf: Toniebox

If you're not familiar with the Toniebox concept, it's brilliant in its simplicity: a colorful cube that plays audio when children place special figurines (called "Tonies") on top. No screens, no complicated interfaces — just place a character and hear a story. The figures use RFID chips that the box recognizes, triggering specific content that is cached on the box itself.

It really is a great product, however cost is a problem. The Toniebox itself plus a Tonie figure costs around €100, and every Tonie thereafter costs €18-25.

And that's just for German language Tonies. My children are being raised bilinguaglly in English and German, with German naturally dominant given where we live.

Importing English language Tonies from the UK bumps the cost of a Tonie up to €25-30 (thanks Brexit!), and those are British-English. I'm not trying to be snobbish here, but we speak American English and home and I don't think it is unreasonable to want to make the same dialect available to my kids in their audio diet.

Importing Tonies from the USA? Now you are looking at €35-40 per Tonie. Yikes.

My goals were simple:

1. Create a device my child could use independently
2. Keep it screen-free (for the child, anyway)
3. Use open source tech
4. Make it robust enough to survive a preschooler

Clojure, LibVLC, and the Event-Driven Rabbit Hole

I decided to build the project in Clojure, partly because I love the language, but mostly because its concurrency model with core.async channels makes building an event-driven system a joy rather than a nightmare. I'd never actually used core.async as the central abstraction in a project before. It works (and might be elegant?), but it can turn into a bit of a tangled mess if you're not careful with your channels.

;; The pub/sub at the heart of fairybox
(defn emit! [bus topic value]
  (let [pub-ch (:publisher bus)]
    (async/put! pub-ch {:topic topic :value value})))

;; And how to subscribe to events
(defn subscribe [bus topic]
  (let [sub-ch (async/chan 10)
        pub (:publication bus)]
    (async/sub pub topic sub-ch)
    sub-ch))

;; Example usage in a component
(defn start-button-listener [bus button-gpio action]
  (let [events (subscribe bus :buttons)]
    (go-loop []
      (when-let [event (<! events)]
        (when (= (:gpio event) button-gpio)
          (emit! bus :player/commands {:action action}))
        (recur)))))

Every component of the system — the RFID reader, the buttons, the LEDs, the audio player — communicates through this event system. When your toddler places an RFID card on the reader, it sends an event that gets picked up by the audio component, which then plays the associated content. Simple, elegant, robust.

One decision I'm particularly proud of was keeping the media player in-process using LibVLC rather than shelling out to a separate player. The Java bindings for LibVLC are surprisingly comprehensive and work well, even though they're excruciatingly Java-y with their AbstractFactoryBuilderImplementationFactoryImpl naming conventions. But hey, at least I didn't have to write the JNI bindings myself.

;; Using LibVLC to play media without spawning external processes
(defn play! [player url]
  (-> player
      (.mediaPlayer)
      (.media)
      (.play url nil)))

The Web UI: HTMX and Websockets for the Win

While the device itself is deliberately screen-free for my child, I still needed a way for parents to configure it. Enter HTMX and websockets — a surprisingly powerful combination that let me build a responsive web interface with minimal JavaScript.

The UI lets me assign RFID cards to specific audiobooks or playlists, control volume, and see what my kid is listening to. It's accessible from any device on our home network, which means I can easily change settings or help troubleshoot from my phone while my child is using the device. The interface updates in real-time when buttons are pressed on the physical device, making the whole thing feel cohesive (which, considering my partner and I are the only users, is probably an excessive level of polish, but whatever).

;; A taste of the HTMX magic
(defn audio-controls [req]
  [:div#audio-controls.audio-controls
   {:hx-ext "ws" 
    :ws-connect "/ws/audio-controls"}
   [:div.audio-title 
    [:span#title "Ready to play"]]
   [:div.controls-row
    [:button.control-button 
     {:hx-ws "send" :hx-vals {:action "prev"}}
     (icon/prev)]]])

The websocket connection keeps the UI in sync with the device state. When my daughter hits the physical "next track" button, the web UI updates instantly to show the new track. It's a small touch, but it makes the entire experience feel seamless.

Hardware Woes: The NixOS Experiment That Broke My Spirit

I've been using NixOS for years on my servers and Linux boxes, not just as a development environment. I thought, "Wouldn't it be elegant to deploy the Fairybox on NixOS too?" This, friends, is what we call technological hubris of the highest order.

What followed was a weeks-long battle with device tree overlays, GPIO permissions, and the peculiarities of the Raspberry Pi 4. It turns out that while NixOS runs beautifully on the Pi, accessing GPIO pins and using device tree overlays is... let me put it this way: if you ever want to feel truly humbled by your own incompetence, try getting a NixOS RPi4 to talk to an RFID reader over SPI while also driving PWM LEDs.

# A painful snippet from my failed NixOS experiment
hardware.deviceTree = {
  enable = true;
  filter = lib.mkForce "bcm2711-rpi-4-b.dtb";
  overlays = [
    {
      name = "spi0-1cs-overlay";
      dtsText = builtins.readFile ./overlays/spi0-1cs-overlay.dts;
    }
  ];
};

I banged my head against this wall for what seemed like eternity. I'd get one component working, only to find that another stopped. After exhausting every forum post, GitHub issue, and Discord chat I could find, I admitted defeat. I returned to a more conventional Raspberry Pi OS setup and deployed everything with a simple Ansible playbook. Sometimes the boring solution is the right one.

Hardware Iterations and Soldering Disasters

Let me tell you something about toddlers: they're walking chaos engines. No matter how well you think you've secured your hardware connections, a determined 3-year-old will find a way to jostle something loose.

The Fairybox went through several hardware iterations, especially the power solution. The first version used a standard USB power bank, but the Raspberry Pi would occasionally brown out when playing audio at high volumes. The second version used a LiPo battery with a simple boost converter, which lasted longer but still had stability issues.

For the final version, I went full overkill and got a KWeld spot welder (a whole other DIY project) to build a custom battery pack, paired with the AmpRipper 4000 charge controller/boost converter. This gives me a stable 5V 3A power supply that can run for hours and recharge quickly.

Despite years of practice, my soldering skills remain firmly in the "functional but ugly" category. Every time I open up the Fairybox to fix something, I'm confronted with blobs of solder that look like they were applied by someone wearing boxing gloves. But hey, they conduct electricity, and that's what counts, right?

The Final Product: A Box Full of Joy

The finished Fairybox sits in my daughter's room (when she's not carrying it around the apartment), a simple wooden box with colorful LED buttons. She places an RFID card on top, the lights dance in acknowledgment, and her favorite audiobook begins playing. She can press the big, friendly buttons to pause, skip tracks, or adjust volume.

What amazes me is how quickly she mastered it. There's something wonderfully intuitive about physical interfaces — no menus to navigate, no apps to open, just tangible cause and effect. (Though I should admit "intuitive" is stretching it a bit — it took her a couple weeks of placing cards and repeatedly asking "What does this button do again?" before she fully got the hang of it.)

Now my youngest is starting to get jealous, giving me the perfect excuse to build version 2.0. I've already started gathering components and thinking about improvements. I'm slightly terrified at the prospect of having two audio devices wandering around our not-very-large apartment, potentially blaring different stories at maximum volume simultaneously. I may need to include some kind of proximity sensor that lowers the volume when they get too close to each other.

Beyond the Build: Technology That Respects Childhood

This project has reinforced my belief that technology doesn't have to mean screens and apps. We can build digital tools that respect the developmental needs of children — tools that encourage imagination rather than passive consumption.

As someone who builds technology for a living, I feel a responsibility to be thoughtful about the digital environments I create, especially for the most vulnerable users. My clients are human rights organizations trying to make the world better. Shouldn't I apply the same principles at home?

The Fairybox isn't just a toy — it's a statement about the kind of technology I want in my children's lives: respectful, empowering, and imagination-enhancing rather than attention-extracting.

And if you're wondering if all this effort was worth it when I could have just bought something off the shelf — the look on my daughter's face when she first used something Dad built answers that question perfectly.

Now I just need to finish the second one before my youngest figures out how to stage a toddler coup.

Reply via: Email · Bluesky

If you've found yourself here through an old link to elusivetruth.net or binaryelysium.com, I should probably explain: those blogs no longer exist.

I've taken them down, redirected the domains, and generally done some digital housekeeping. Apologies for dead-end.

For those wondering (there can't be many of you), binaryelysium.com and elusivetruth.net were my open-source dev and travel blogs from my early-to-mid twenties. Binary Elysium started with my first excursions into open source around 2006 and chronicled my days as a KDE contributor.

When I began traveling full-time I found an audience interested in my travelogue, so Elusive Truth was spun out as a standalone blog. Both blogs drifted for while I also drifted along with a laptop, doing contract work while experimenting with various forms of travel.

This period of time culminated in my traversal of Europe from the North Sea to the Black Sea using nothing but human power: pedaling my bike and paddling my packraft.

I completed that journey in 2014, and had my sights set further east, but as is our way, settled down, found a partner, and started a family.

The internet has changed dramatically since those early blogging days. What was once a more unfiltered, personal, and decidedly less commercial space has evolved into something sanitized, algorithmic, monetized, and focused on "content."

My old posts feel like artifacts from a different era. Not just an era of of my life, but of the web itself. An era when people wrote purely for expression, no algorithms or brands in sight. I did at least (though my contributions were objectively mediocre).

My trans-European journey was partly inspired by Patrick Leigh Fermor's book "A Time of Gifts," a memoir of hiking across pre-WWII Europe in the 1930s. A musty book despite being "only" from the '70s. The narrative is of a privileged white guy struggling with the self-imposed misery of traipsing through Europe with letters of invitations to high society in his pocket but holes in shoes.

Huh. 🤔 His story is similar to mine in most aspects. It lacks the pre-war intrigue. The letters of invitation are swapped for Couchsurfing/Warmshowers. But white and privileged, check check.

It is embarrassing reading your own writing from a decade past, especially when it's infused with the naivety characterized by that age.

Though they say the internet never forgets, I've found that's only partially true. If you're determined and know how to use the Wayback Machine, you could probably unearth some of my old writings and photos.

But honestly, I wasn't an exceptional writer, and there were (and are) far better blogs and books on the subject. I'm at peace with that realization now, in a way my younger self might have found disappointing.

Rather than read me, here are a few authors that influenced my deeply during those wandering years. Their work deserves to be revisited far more than mine, especially if you're someone looking to go on an adventure.

You can find them preserved at onvagrancy.com

My favorite piece remains "On Vagrancy" by Isabelle Eberhardt, a remarkable French iconoclast who defied every convention of her era - in her 20s she was dressing as a male Arab, converting to Islam, and living as a nomadic writer in the Maghreb Sahara.

Was it a coincidence that I was around the same age when her writing gripped me so completely? Probably not, but she tragically died at 27 years old in a flash flood. Meanwhile from nearly-40 I can see what she never got the chance to.

I can't let this reflection end without mentioning "The Gentle Art of Tramping" by Stephen Graham, a 1924 book which was my road bible, and I think has something for everyone.

"The tramp is a friend of society; he is a seeker, he pays his way if he can... Tramping is a way of approach, to Nature, to your fellowman, to a nation, to a foreign nation, to beauty, to life itself."

The digital detritus of our past selves accumulates so quickly now. Perhaps there's value in the occasional pruning—making space for the work that continues to resonate. The words of Eberhardt and Graham still do. My early blogs don't.

Reply via: Email · Bluesky

There's nothing quite like the panic that sets in when your home internet connection goes down right before an important client call. After one too many of these moments, I decided it was time for a proper backup solution.

But being the engineer and self-hoster I am, I couldn't just buy something off the shelf. No, I needed to overcomplicate things with a DIY approach involving a Raspberry Pi, a USB 4G dongle, and a few hours of tinkering.

I've built this setup with a Unifi Security Gateway (USG), but the Raspberry Pi part would work with most routers that support a secondary WAN connection. Fair warning: what follows is decidedly not a polished product.

The goal: turn a Raspberry Pi into a mini-router that connects to your cellular network via a 4G USB dongle and presents itself as a standard ethernet WAN connection to your USG. When your primary internet fails, the USG automatically fails over to the Pi's cellular connection, and your work continues uninterrupted—albeit potentially more expensive if your cellular plan charges by the gigabyte.

I opted for TinyCore's piCore Linux for the Pi because it's very lightweight, boots fast, and, well, I wanted to take it for a spin (I'm all about those alternative ARM Linux distros).

The setup involves configuring multiple network interfaces: the USB dongle interface (usb0), the ethernet port (eth0) that connects to the USG, and optionally WiFi (wlan0) for out-of-band management. It's really not much more than a tidy bit of Linux networking and a sprinkling of udev rules.

Of course, no DIY project is complete without a maddening workaround. The ZTE MF823 dongle I'm using creates its own default subnet (192.168.0.1/24), and despite my best telnet-hacking attempts to change it, it stubbornly resets after each power cycle. So if your home network uses the common 192.168.0.0/24 subnet, you'll need to adjust your network, use a different dongle model entirely, or, like, go get a real failover product.

What about performance? The theoretical LTE speeds sound impressive, but the reality is that LTE through a USB port on a Raspberry Pi 3 is not. The Pi 3's ethernet and USB ports share the same USB 2.0 bus with a theoretical maximum of 480 Mbps, creating a bottleneck that limits real-world throughput to around 15-30 Mbps. But this project is about failover and continuity, not performance. Just having connectivity at all is the primary goal. Pulling your fat Docker containers or the latest JavaScript framework can wait until your primary connection is restored.

If you want to replicate this setup, I've included instructions below.

Table of Contents

• Hardware
• Software
• Address Space
• Pre-setup - Test LTE dongle is working
• Setup of PI
• Unifi USG WAN failover configuration
• Resources

Hardware

• Unifi USG
• An extra switch (unmanaged, 100mb or 1gbit depending on if your PI supports gbit or not)
• Raspberry Pi
• ZTE MF823 LTE USB Dongle

Software

• Unifi Controller
• TinyCoreLinux PiCore download (v11 at time of writing)

Address Space

• 192.168.0.1/24 for usb0 - this is the default subnet on the MF823, your home LAN mustn't overlap with this.
• 192.168.73.1/24 for usb0 - we will change the MF823 to use this subnet editing the dongle's settings doesn't stick across reboots
• 192.168.12.1/24 for eth0 - for USG<->PI

Pre-setup - Test LTE dongle is working

Flash PiCore onto SD card

Boot PiCore, plug into ethernet

sudo /sbin/udhcp -v -i eth0 -x hostname:wan2 -p /var/run/udhcp.eth0.pid
ping 1.1.1.1

Insert dongle .. wait for it to settle ~60 seconds

Test

lsusb

While the dongle is booting it will show a red light and you will see

ID 19d2:1225

After it is ready the light will turn green and it will change to

ID 19d2:1405

Load the kernel module

modprobe cdc_ether
ifconfig -a

You should see usb0

Setup usb0

sudo /sbin/udhcp -v -i usb0 -x hostname:wan2 -p /var/run/udhcp.usb0.pid
ping 192.168.0.1

Use Socks proxy from your workstation to access MF823's webui

# on workstation on same LAN as the pi
ssh -D 1337 tc@192.168.1.146

(192.168.1.146 was the ip on my local lan the pi got for eth0)

Use browser's socks settings to set socks 5 proxy 192.168.1.146 port 1337

Browse to 192.168.0.1 in browser, confirm the web ui is loading. If you have a sim card in, you should see LTE network connection status.

Setup of PI

Ok it's working. Time to set up the router on the pi.

First, let's set up wifi on the pi. eth0 will become the LAN port for the pi router, but we need a headless/oob management channel, this will be over wifi.

SSH into the pi

Add ssh config and passwd to persistent config

mkdir ~/.ssh
vi ~/.ssh/authorized_keys
# paste your ssh key
sudo echo '/usr/local/etc/ssh' >> /opt/.filetool.lst 
sudo echo '/etc/shadow' >> /opt/.filetool.lst
sudo echo '/home/tc/.ssh/' >> /opt/.filetool.lst
filetool.sh -b

Download the wifi extension and reboot to load the module

tce-load -wi firmware-rpi-wifi.tcz
tce-load -wi wifi.tcz
sudo reboot

SSH again (using the key this time) and check for wlan0

iwconfig

Connect to your AP

sudo /usr/local/bin/wifi.sh 

Check wlan0 connection

ifconfig wlan0

Configure auto wlan connect on system boot

sudo echo '/usr/local/bin/wifi.sh -a 2>&1 > /tmp/wifi.log' >> /opt/bootlocal.sh
filetool.sh -b

Reboot to test wlan0 auto config

sudo reboot

Quickly SSH in over the eth0 interface, get the wlan0 ip address and then ssh back in over wifi.

From here on out I assume you are managing the pi over wlan0, as we will be making changes to eth0.

NOTE: The following section is included, but does not seem to actually work. Every time the dongle is rebooted, the settings are reverted.

Next, let's change the subnet used by the ZTE MF823 router so it doesn't use the common 192.168.0.1 subnet.

Ssh into the Pi, then telnet into the router

telnet 192.168.0.1
# user: root
# password: zte9x15

Edit the file at /usr/zte/zte_conf/config/userseting_nvconfig.txt

Change the values:

dhcpStart
dhcpEnd
lan_ipaddr
lan_ipaddr_for_current

I assume in the rest of this that you are using the subnet 192.168.73.0/24

Actually, we continue with 192.168.0.1, since the above does not stick.

Create /opt/eth0.sh

#!/bin/sh

sleep .5

sleep 1
if [ -f /var/run/udhcpc.eth0.pid ]; then
kill `cat /var/run/udhcpc.eth0.pid`
sleep 0.1
fi

ifconfig eth0 192.168.12.1 netmask 255.255.255.0 broadcast 192.168.12.255 up

sleep .1
sudo udhcpd /etc/eth0_udhcpd.conf &

Make it executable

chmod 775 /etc/eth0.sh

Create DHCP config for eth0 in /etc/eth0_udhcpd.conf

start 192.168.12.100
end 192.168.12.200
interface eth0
option subnet 255.255.255.0
option router 192.168.12.1
option lease 43200
option dns 192.168.12.1
option domain wanfailover

Start and test dhcp server. You should see it listening on port udp 67.

sudo udhcpd /etc/eth0_udhcpd.conf
ps -ef | grep udhcpd
sudo netstat -anp | grep udhcpd

Create init script to manage usb0 in /etc/init.d/dhcp-usb0.sh

Get file contents here dhcp-usb0.sh

Make it executable

chmod 766 /etc/init.d/dhcp-usb0.sh

Create udev rule to auto connect to the usb0 network in /etc/udev/rules.d/15-zte-mf823.rules

SUBSYSTEM=="usb", ATTR{idProduct}=="1405", ATTR{idVendor}=="19d2", RUN+="/etc/init.d/dhcp-usb0.sh restart"

Reload udev rules

sudo udevadm control --reload-rules 

Unplug USB device, wait a few seconds, plug it back in. Check that usb0 has an ip in the 192.168.0.0/24 subnet.

Persist the config

sudo echo '/opt/eth0.sh' >> /opt/.filetool.lst
sudo echo '/etc/eth0_udhcpd.conf' >> /opt/.filetool.lst
sudo echo '/etc/init.d/dhcp-usb0.sh' >> /opt/.filetool.lst
sudo echo '/etc/udev/rules.d/15-zte-mf823.rules' >> /opt/.filetool.lst
sudo echo '/opt/eth0.sh &' >> /opt/bootlocal.sh
filetool.sh -b 

Reboot to test. You should see eth0 with an ip address of 192.168.12.1, and usb0 should be configured.

Enable ipv4 forwarding

sudo sysctl -w net.ipv4.ip_forward=1
sudo echo 'sysctl -w net.ipv4.ip_forward=1' >> /opt/bootlocal.sh
filetool.sh -b

Install dnsmasq and iptables

tce-load -wi dnsmasq
tce-load -wi iptables

Enable NAT

sudo iptables -t nat -A POSTROUTING -o usb0 -j MASQUERADE

Make it persistent

sudo echo 'iptables -t nat -A POSTROUTING -o usb0 -j MASQUERADE' >> /opt/bootlocal.sh
sudo echo 'dnsmasq' >> /opt/bootlocal.sh

Finally, add a little script to remove the wifi default gateway. Without this, the wifi script will take over the default gateway. The actual default gateway is set by our usb udev script.

/opt/fix-gw.sh

#!/bin/sh

gw=$(route -n|grep "UG"|grep -v "UGH"|cut -f 10 -d " ")

if [ ! -z "$gw" ]; then
  route del default gw "$gw"
fi

Save it

chmod 777 /opt/fix-gw.sh
sudo echo '/opt/fix-gw.sh' >> /opt/.filetool.lst
sudo echo '/opt/fix-gw.sh' >> /opt/bootlocal.sh
filetool.sh -b

Do a final reboot. Connect your pi to an empty switch, connect your laptop to the switch. You should have internet via the LTE dongle, verify with

curl https://iconfig.co/json | jq

You should see your LTE provider's info.

Unifi USG WAN failover configuration

This is a small cluster-f*** depending on your Controller version and whether you have the old or new settings interface. In short, any docs you read about setting the "Port Remapping" feature are out of date since at least 2019.

You must not be using the WAN2 port for LAN traffic.

Assuming the old, non beta (as of Oct 2020) settings UI you can follow what is below.

Are you from the future where the new beta UI is no longer beta, and the old UI is gone? Good luck.

1. Create a WAN2 network

   Settings -> Networks -> [ + Create New Network ]
   Purpose: WAN
   Network Group: WAN2
   Load Balancing: dropdown, choose
       "Failover Only" to use the WAN2 port only if WAN has failed
   

2. Assign the USG's port to the WAN2 network

   Devices -> USG -> Ports tab -> [ Configure interfaces ]
   
   Port WAN2/LAN2 Network: WAN2
   
   Apply
   

Wait for the USG to re-provision.

Test it by sshing into the USG and execute:

ip addr # check eth2
show load-balance status
show load-balance watchdog

That's it. Unplug your WAN1, watch it failover to WAN2. Plug WAN1 back in and see WAN2 recover.

In case you're wondering: you do get email alerts and alerts in the Controller UI whenever a WAN transition happens.

Resources

• Hacking MF 823's UI
• ArchLinux Modem Device info
• Tinycore ip router setup

Reply via: Email · Bluesky

Everyone's talking about Kubernetes. At every conference, in every DevOps Slack channel, in the Orange Pages, the message is clear: if you're not running Kubernetes, you're doing containers wrong. Well, I'm here to tell you that for most small and medium businesses, Kubernetes is like using a battleship for your daily commute.

Don't get me wrong, containers are fantastic. They've revolutionized how we deploy software (especially for runtimes like ruby, node, and python, less for the jar/war community). But somewhere along the way, we confused "using containers" with "needing Google-scale orchestration."

I consult with SMBs and nonprofits on their technical infrastructure. Here's what their container deployments actually look like:

• 5 to 20 containers running their core applications.
• A <your favorite>SQL database.
• Maybe Redis for caching.
• They don't have a dedicated ops team. They have a Sarah who knows Linux and Docker pretty well and Jim who's really good with databases.

That's it.

But then they give into the FOMO (after all they don't want to be the dinosaur still using "just Docker"). Now here's what happens when they deploy Kubernetes: suddenly they're running 20+ containers just for the infrastructure! The Kubernetes control plane itself, ingress controllers, storage plugins, network policies, monitoring sidecars... the service mesh shudder. When your infrastructure containers outnumber your actual workloads by 2-to-1, you've got to ask yourself: what problem are we solving here?

The teams I work with typically have 0-5 developers who also handle operations in devops style. They don't have dedicated SREs. They push code during business hours and schedule maintenance windows for updates. Their uptime requirements? "Don't break during the workday, and please let us sleep through the night."

These aren't the problems Kubernetes was designed to solve. Kubernetes solves Google problems: thousands of services, millions of requests, teams distributed across the globe. Most businesses, even successful and growing ones, will never have Google problems.

The Hidden Tax of Complexity

Matt Rogish from ReactiveOps recently argued that "Kubernetes has low accidental complexity and high essential complexity". I appreciate this argument, in that simple doesn't mean easy, but the "essential complexity" of Kubernetes is still astronomical for most organizations. Essential complexity isn't inherently virtuous. It's only valuable when it maps to essential problems you actually have. He dismisses CTOs who say "I just have a Rails application and plain old EC2 VMs will give me what I need" as being short-sighted.

The "essential complexity" of Kubernetes includes:

• Understanding pods, deployments, services, and ingresses
• Grokking the networking model (ClusterIP vs NodePort vs LoadBalancer)
• Learning YAML templating with Helm or Kustomize (and the absolute mess that managing that is)
• Debugging why your pod is stuck in CrashLoopBackOff
• Figuring out why your PersistentVolumeClaim won't bind
• Understanding RBAC and service accounts
• Keeping up with the regular Kubernetes releases and deprecations

This isn't accidental complexity. These are fundamental concepts in the Kubernetes model. But for a team running 10 containers, this essential complexity is solving problems they don't have while creating new ones they didn't ask for.

The real cost is opportunity cost.

While you're wrestling with pod network policies and figuring out why your persistent volumes won't mount, your competitors are shipping features using boring, simple technology that just works.

A battleship is still a battleship even if you are renting it

I can see the k8s proponotes frothing at the mouth now, "what about managed Kubernetes?" In the last year Google's launched GKE for a while, Azure AKS last year, and this year AWS has launched EKS. True, these services remove some operational burden. You don't have to manage the control plane or worry about etcd backups. But here's the thing: even with managed Kubernetes, you're still on the hook for a lot.

You still need to understand pods, services, deployments, ingress controllers, persistent volumes, and the rest of the Kubernetes abstraction layer. When your app won't deploy, the error messages assume you speak fluent Kubernetes. When performance tanks, you're debugging through layers of network policies and service meshes. You're still managing node pools, scaling policies, resource quotas, network policies, and RBAC configurations. You're still debugging why your pods are stuck in "ImagePullBackOff" or why your PersistentVolumeClaim won't bind. Managed or not, you still need to know what a CNI plugin is and why yours is misbehaving.

Rogish admits that most companies don't need Kubernetes for scaling, the one thing it's probably great at.

Rogish's argument is that EC2 instances can't automatically restart crashed applications. "Apps running on regular EC2 instances have no automatic restart if your Rails application runs out of memory," he writes.

Simple Alternatives That Actually Work

Here's the thing: you probably already have everything you need. Docker plus systemd can handle most single-host deployments beautifully. Write a systemd unit file, enable it, and you're done. Need to update? docker pull, systemd restart. It's so simple it feels like cheating.

For multi-container applications, docker-compose gives you 80% of what Kubernetes offers with about 10% of the complexity. Define your services in a YAML file, run docker-compose up, and watch your stack come alive. Need horizontal scaling? Run haproxy or nginx on a box as a load balancer, run several other boxes running Docker. Your favorite monitoring package. It's not fancy, but it works.

And sometimes, a bash script that pulls a new container and restarts it is all you need. I've seen this "architecture" quietly mint millions in revenue without breaking a sweat. It's understandable, debuggable, and maintainable by anyone who knows basic Linux.

The Boring Path Forward

Dan McKinley's "Choose Boring Technology" remains my north star for tech-stack and infrastructure decisions. Every company gets about three innovation tokens. Spend them wisely. If your core business is e-commerce, why spend an innovation token on orchestration? Use boring tools for infrastructure so you can be innovative where it matters: your actual product.

Start with the simplest thing that could possibly work. Measure actual pain points before adding complexity. Feel the pain of manual deployments before automating. Hit scaling limits before building for infinite scale. You might be surprised how far simple solutions can take you.

If you eventually need Kubernetes, it'll be obvious. You'll have specific problems that simpler tools can't solve. Your team will have grown. You'll have budget for dedicated operations. When that day comes, yea sure, start with a managed service like GKE or AKS. The migration will make sense because it solves real problems, not theoretical ones.

Reply via: Email · Bluesky

In the aftermath of the Arab Spring, I consulted on a project developing a mobile application for citizen journalists. The app was technically sophisticated. It helped people record and edit video on their smartphones. It had built-in lessons with training on framing, narrative structure, and journalism topics like citing and protecting sources. The Valley-based developers were excited about "empowering everyday people" to document historical events with journalistic rigor. Keep in mind this was before the rise of TikTok and YouTube Shorts. It was back when traditional media was used and respected.

In principle, it worked perfectly. In practice? The app failed.

It generated high-quality video files ranging from 500MB to 3GB per story package—impossible to transmit over the congested 3G networks and intermittent connections available in regions experiencing political upheaval. And that's before considering the prohibitive mobile bandwidth costs. Citizen journalists reverted to posting short, unedited clips directly to social media platforms that compressed the videos. The app's sophisticated storytelling features went unused, while important documentation happened through the simplest channels that could actually handle infrastructure bottlenecks.

This wasn't a one-off failure. In human rights tech, this sort of failure is common.

The Great Disconnect

A vast reality gap separates the people who build human rights technology from those who need to use it. We've created a situation where tools are designed in Silicon Valley, London, or Berlin for use in entirely different contexts; where infrastructure, digital literacy, and everyday realities are vastly different.

While human rights defenders prioritize simplicity, familiarity, and reliable offline functionality, developers often focus on advanced features, perfect security models, and sophisticated workflows that assume stable infrastructure.

What we're left with is a graveyard of well-intentioned technologies that look impressive in demos but fail in the field.

In my decade of consulting with human rights organizations, I've seen this story play out repeatedly. Applications require constant internet connectivity in regions where connections are spotty at best. Tools generate large files without considering how users will transmit or store them. Software gets designed for high-end devices in contexts where users have older phones with limited processing power. Security updates require bandwidth many users simply can't access.

Some years ago, I worked with a cocoa importer who wanted to improve their sustainability monitoring across their West African supply chain. Their team understood the reality on the ground. Mobile signals in rural areas were either non-existent or prohibitively expensive. Instead of forcing a digital-first approach, their field inspectors gathered information using paper forms. The software solution processed these paper forms once inspectors returned to their offices, automatically extracting and aggregating sustainability metrics from the scanned documents. The system generated the quantitative data they needed for annual reports and year-over-year tracking.

The inspectors could focus on their work in the field without worrying about connectivity, while the importer got the digital sustainability data they needed. Not flashy, but it worked because it respected local constraints from the start

What happens when these tools fail? Human rights defenders don't just abandon their work, they find creative workarounds. Activist collectives use shared email accounts as makeshift archives. WhatsApp groups become documentation databases (yes, sigh)

They're smart, they're adaptable! When your solution fails them (because you didn't understand their local constraints), they come up with a better one.

Better, because it actually works.

Reply via: Email · Bluesky

An NGO documenting human rights abuses was encouraged by security experts to switch from Windows to a free software operating system. The reasoning was sound - better security, less surveillance, more control. But there was a problem.

Their printer wouldn't work with the new system.

Staff started saving sensitive documents to USB sticks and printing them at local internet cafes. The USB sticks went missing, exposing far more sensitive data than the theoretical risks they'd been trying to avoid.

I call this the "security paradox" - when well-intentioned security advice actually creates new, often more dangerous vulnerabilities.

Why Does This Keep Happening?

The root cause isn't complicated: there's a massive disconnect between security experts (often in Europe or the US) and the daily realities of human rights defenders working in challenging contexts.

Valeria Umaña, who works with groups in Nicaragua, puts it bluntly in the Technology Tools in Human Rights report:

"For people in the countryside, the more apps they have, the more problems they can have because they often don't know how to use them."

The pursuit of perfect security ignores imperfect realities, and actively endangers the people it's meant to protect.

When security recommendations fail, vulnerable populations bear the risk. It's not the security consultant who faces danger when documentation leaks - it's the victims and witnesses who provided testimony.

Trust evaporates after a security recommendation backfires, making organizations resistant to all security advice, even the good stuff.

For chronically underfunded human rights organizations, investing precious time and money into systems that ultimately fail is devastating.

I've witnessed organizations abandon critical documentation work altogether after particularly traumatic security failures. That's not just a technical problem - it's a human rights catastrophe.

A Better Way Forward

The good news? There are approaches that actually work. The conclusion of the Engine Room study has some stellar advice.

I also would add these three additional techniques I have found useful:

1. Start with risk assessment, not tool selection. Understand the specific threats an organization faces before recommending solutions. A women's collective documenting domestic violence faces different risks than journalists exposing government corruption.

2. Value simplicity and familiarity. Sometimes a slightly less secure tool that people will actually use correctly is better than a theoretically secure one they'll work around. As Rory Byrne says in the report, "People like to stick with what they know."

3. Implement changes incrementally. Rather than radical overhauls, focus on gradual improvements with continuous feedback. When I work with human rights groups, we start with one small change, perfect it, then move to the next.

The Reality Check We Need

I believe deeply in the right to privacy and the importance of security for human rights work. But I've learned to be humble about technology solutions. The most secure system isn't the one that looks best on paper, it's the one that actually protects people in real-world conditions.

Technology doesn't make change. People make change. They need security approaches that recognize their humanity, constraints, and contexts.

So the next time you receive security advice, remember the NGO that had printer trouble and ended up with USB sticks in internet cafes. They weren't careless - they were trying to be more secure.

Don't let the latest security solution become the next problem.

Reply via: Email · Bluesky

Want the yubikey+google 2-factor authentication solution? Skip to the good stuff.

Passwords rule our lives on the Internet; they are the foundation of identity management. When a website or service wants to know who you are, you prove you are you with your username and password. However, passwords aren't the fundamental piece of this identity management system.

What happens when you lose your password? We've all been through this jig before. The website usually sends an email to you with a link or instructions on how to reset your password. By proving you have access to your email, you are effectively proving you are you.

Then, your email account is the lowest common denominator; with access to your email account (nearly) all your other accounts can be accessed. If you use a completely unique passwords for each service (and store them in a password manager like Keepass or Lastpass), then access to your email account is even more attractive. Therefore, the importance of securing your email account cannot be overstated.

When it comes to securing your email Google's 2-factor authentication is pretty awesome. Even though there are still some important flaws one should be aware of, it can significantly increase the security of your Google or Google Apps account.

For me there is one major drawback to Google's 2-factor offering: it requires a cellphone to be useful. This is a drawback for several reasons.

First, your smartphone isn't as secure as we would like. Mobile malware is on the rise--particularly if you have an Android phone--and if I was a malware writer I would be targeting 2-factor authentication apps like Google's.

The second drawback most people won't identify with: I don't want to carry a smartphone with me! I travel. A lot. In fact, I travel by bike. When traveling by bike, minimizing weight is important followed closely by minimizing the value of my equipment (shiny stuff gets broken or stolen), and smartphones are heavy and expensive. So, I carry a tiny and cheap $30 cellphone and swap sim cards as I enter new countries.

How then can I retain the benefit of Google's 2-factor authentication, while ditching the phone? I could generate OTPs through Google and write them down, which I have been doing for awhile, but that is a huge PITA.

Enter the Yubikey. The Yubikey is a tiny usb device that produces One-Time Passwords and appears to the OS as a USB keyboard making it work on all platforms. The Yubikey can hold two identities that can be configured according to four different options (Yubico OTP, OATH, static, challenge-response).

A Yubikey seems like the perfect lightweight, secure replacement for OTP generation, how then could I use it with Google's 2-factor authentication?

Finding common ground: OATH-TOTP

Originally, I imagined some system that stored your Google 2-factor auth secret, and allowed you to auth with your Yubico OTP. Such a system would not be ideal, because wherever that system lived so would live your google secret. We want to be sure wherever the secret is stored is secure.

As it turns out Google's 2-factor authenticator is an implementation of the OATH-TOTP protocol, a system for generating one-time passwords via a HMAC-SHA1 hash using the current time as input.

The Yubikey also happens to support the OATH-HOTP protocol (of which TOTP is a variant), so we should be able to configure the Yubikey to generate OATH-HOTP OTPs somehow. Unfortunately, the Yubikey is battery-less, so it is unable to store the current time.

All is not lost, for the Yubikey's fourth configuration option is a challenge-response configuration. This allows a client-side application to send a challenge to the Yubikey, which the Yubikey uses as input to generate a HMAC-SHA1 hash that becomes the response. This is exactly the cryptographic hashed used by OATH-TOTP and hence Google's 2-factor auth.

Around the same time I figured all this out, Yubico posted the same explanation I just gave, along with a Windows client-side application that used the challenge-response method described to enable Google authentication with a Yubikey. Huzzah!

YubiTOTP for Linux

It took awhile, but a friend and I eventually got around to implementing a similar client-side helper application for Linux.

The implementation is fairly simple (if not pretty). A challenge is generated based on the current time, sent to the yubikey using the ykchalresp utility, and then the HMAC-SHA1 hash is mangled according to the HOTP specification to produce a 6 digit code.

Before you can use the tool, you must configure your Yubikey, but then generating OTPs from your Yubikey is as simple as: $ ./yubi_goog.py.

The tool can also be used to generate OTPs without a Yubikey (using the --generate flag), but you must enter the secret on every invocation.

Grab the tool and instructions over at the github repo.

Not Perfect

We now have a way to generate OTPs for Google's 2-factor authentication without a phone; however, this isn't the perfect solution. Generating a TOTP requires the current time, so the Yubikey must be told the current time which stipulates the use of a client-side helper application.

So, using this method you can only use your Yubikey where you are able to run my helper app (or the windows version from Yubico). I often find myself in Internet cafes or other public terminals, where running a python script isn't feasible.

As of yet I do not have a working solution. It would be fantastic if Google would natively support the Yubikey, but in the meantime we'll have to be satsfied with innovative hacks.

Reply via: Email · Bluesky

Play-by-mail gaming

Assuming you know what it is, I suspect that phrase produces two different reactions in the minds of those who read it. Either it conjures up fond memories of a special era in gaming, an era in which you spent hours with piles of papers haphazardly spread about, lost in the universe of your imagination, and then hours more in anticipation waiting for that darned post man to arrive at your mailbox with the results of your dastardly scheming or the outcome of a space battle of truly epic proportions; or that phrase incites an altogether different thought, one of a bygone era of rotary telephones or a hand written letters, which is to say thoughts of the recently antiquated.

The postal system as a medium for gaming has certainly declined in recent decades, and unfortunately the number of games in the style that PBM (play-by-mail) promoted has declined as well. In the 90's e-mail developed, and many PBMs became PBEMs (play-by-email) with much success; however the transition to the Internet did not fair as well. The way most people use and view the Internet is directly contradictory to the important core mechanic of anticipation (i.e., lack of instant feedback). After all, the Internet enabled real-time mechanics that were previously impossible (not to mention the advances in computers that allow for 3d graphics).

This PBM style can be generally summed up in this laundry list:

• Asynchronous, turn based play -- players didn't have to gather and play simultaneously
• Multiplayer -- player interaction was a driving force
• Massive scale -- a position consisted of running an empire or a a group of characters
• Depth -- large number of possible player actions
• Use of anticipation and suspense to keep players interested
• Imagination based -- as opposed to graphic visuals

Imagination has been replaced with intense 3D visuals. The feeling of anticipation and suspense was replaced by instant gratification made possible by real-time feedback systems. Turn based, asynchronous multiplayer was replaced by real-time synchronous multiplayer. As a result of these replacements, the depth and scale of modern games has suffered. For example, in a real-time strategy game you can't have both massive scale and extreme depth, unless you want games to last for days, a big drawback for multiplayer games. These supplanted qualities thrived in a postal medium, where long and careful planning was favored over instant feedback.

 ****************************
 |                          |
 |        _       __        |
 |       / \     |  \       |
 |      /   \    |   \      |
 |     /         |    \     |
 |    /          |     \    |
 |   /           |      \   |
 |  (ONQUEST and |ESTINY )  |
 |   \           |      /   |
 |    \          |     /    |
 |     \         |    /     |
 |      \   /    |   /      |
 |       \_/     |__/       |
 |                          |
 |                          |
 ****************************

Many people don't fully understand the depth and scale of these old PBM games. These games were not limited to your familiar chess, scrabble, and diplomacy games. They were the first true massively multiplayer games. For example, one such game was Conquest and Destiny, an open-ended, civilization building, role playing game that ran in the early 90s in which every player commanded his/her own race of custom designed beings. The game galaxy had over 7 million stars and planets to explore, colonize, and conquer. While it boasted players in the hundreds or thousands as opposed to EVE Online's 300,000+ or World of Warcraft's 10 million+, it was definitely a massive game for its time. Conquest and Destiny lived and died before the Internet, consequently all we have left from this massive text-based universe is a rulebook and advertisement rescued from the bowels of USENET. Undoubtedly there were many fascinating inter-player narratives of the kind EVE is famous for, but we'll never read them.

Another example of a massive PBM is Hyborian War, a game of imperial conquest in the age of Conan. It began in the 1980s and is still running strong today! Check out an example turn report for a Hyborian War position, courtesy of GrimFinger's Hyborian War site. There were hundreds of these games once upon a time (a by all means non-exhaustive list can be found here).

Internet powered real-time, graphical games are not the logical conclusion of modern gaming. That is, I do not believe real-time is better than turn-based simply because it is newer, for proof of this look no further than the wildly successful Civilization franchise.

Bringing PBM into the second decade of the 21st century faces a major hurdle: the modern generation of gamers might not have the patience or interest in playing a game where feedback between turns is measured in days or more. I suspect this isn't an immutable fact, rather gamers simply need to be (re) introduced to the style and their imaginations reactivated.

The turn-based, long-term, and imaginative play-style is certainly still possible, and in today's fast-paced media intensive life it might be a welcome respite to many gamers.

Reply via: Email · Bluesky

As a coder I'm always on the lookout for interesting technologies to complement my toolset. This is not a post about one of those technologies.

There is a class of programming languages classified as 'esoteric'. While programming languages designed for production use concern themselves with practical features such as readability, performance, and flexibility, esoteric languages explore the boundaries of language design with rampant disregard for all the aforementioned features. A few famous examples of esoteric programming languages (if it even makes sense to refer to an esoteric something as famous) are Brainfuck, LOLCODE, and Malbolge.

In this post I want to talk about an esoteric language that attempts to breakout of a mold every other programming language has taken for granted. This language is Piet and was created by David Morgan-Mar.

From Piet's specification:

Piet is a programming language in which programs look like abstract paintings. The language is named after Piet Mondrian, who pioneered the field of geometric abstract art.

Piet differs from nearly every other language because it is expressed not as text but as colored blocks. Importantly, Piet does not use a simple one-to-one mapping of operations onto colors such as "add is green and divide is red" for that would be trivial and uninteresting. Rather operations are represented as changes in hue and lightness. It is a stack oriented language with low-level operations on par with many Assembly languages. To the left is a sample Piet program, created by Thomas Schoch. It prints "Piet!", and is designed to look like a painting of Piet Mondrian. The image is literally the program. If you are interested in how the language works, I suggest you take a peek at the specification, because there is much more to it than I presented here.

Often a piece of code is referred to as beautiful or ugly; however, what the speaker usually means is that the algorithm, idea, or strategy behind the code is beautiful or elegant. Rarely can you present a piece of code and speak of the code itself as possessing the aesthetic qualities we associate with things of real beauty.

This is a figure from Karl Fogel's chapter in Orielly's book Beautiful Code on Subversion's Delta Editor (full chapter here). Is the beauty of that tree diagram not immediately apparent to you? Don't worry, it wasn't obvious to the author either.

I cannot claim that the beauty of this interface was immediately obvious to me. - Karl Fogel

Piet is an attempt to provide a functional language that can be used to create beautiful programs in the style of modern abstract painters. The sample programs page has quite a few examples of programs written in Piet. Some are beautiful and others are downright ugly.

Take these two programs:

They both print the string "Hello World", yet one is obviously more appealing than the other. Furthermore, the aesthetic properties of each program can be judged by any layman.

Intrigued by this concept of beautiful programs, I set out to create such a program in Piet, but I soon realized that in breaking out of the confines of the text editor Piet had nowhere else to land. That is, conventional image editors (such as the GIMP) are not suited to creating Piet programs. This is because operations in Piet are defined as changes between hue and lightness, which means to know which operation one particular pixel represents you must know the operations of every preceding pixel. Without assistance from the editor creating a program of any non-trivial length is extremely difficult. You are unlikely to understand even a non-trivial program after putting it down for several days.

There are a couple development tools for Piet out there, but most are incomplete or are suffering from bit rot. Given the graphical nature of Piet it only makes sense that the IDE and debugger should be graphical as well. With this in mind and convinced that Piet could actually be a usable langauge given the right environment, I have developed an IDE and debugger.

Piet Creator lives on github. It is written in C++ w/ Qt, so should compile on Windows, Mac, and Linux. It has only been tested on Linux.

The application is named Piet Creator and the source is up on github released under the GPL v3. It is written in C++ with Qt, so it should run on Linux, Mac, and Windows platforms, but I have only tested in on Linux. For the backend interpreter it uses a slightly modified version of the fantastic Piet interpreter npiet written by Erik Schoenfelder.

Practically speaking I realize Piet and Piet Creator are probably useless. Though, I am convinced many programmers take themselves too seriously, so Piet Creator is a serious exercise in not being serious.

I have grand (if silly) visions for Piet and Piet Creator: sub-procedures, standard libraries, and arbitrary color sets. Imagine a world where programmers aren't viewed as digital carpenters or engineers pushing bits around to some functional end, but intriguing artists slinging color across a digital canvas creating functional art appreciable by all.

Reply via: Email · Bluesky

Internet technology is seeing an overall trend towards increased connectivity (always on), information sharing (openness), and most importantly, data existing in "the Cloud" (anywhere access). This is a fascinating prospect, because no one can tell what the results will be.

Consider: The Amazon Kindle. At the core this device does something that many devices have done before, that is, it offers always on transparent connectivity, but it took this commonplace technology and applied it to an entirely different market.

Technology migration between markets is common enough, so why is this case special? Because the Kindle did not just take an existing technology. It took a technology and gave users access to the Cloud.

What is this Cloud thing anyway?

Many people are already accustomed to the idea of a global village facilitated through the internet. In fact, Herbert Marshall McLuhan coined "global village" in his 1962 work The Gutenberg Galaxy, describing how electronic media dissolves traditional communication barriers between humans. This creates a world where there are virtually no limits, and people can exchange ideas, share knowledge, and provide services on a global scale. The internet has enabled all of these feats, yet our digital interactions remain constrained by physical boundaries.

Playing off the village metaphor, this physical limit could be analogized as personal houses in the cyber-village. At your metaphorical house is everything important to you. You have

• a closet full of email
• stacks of MP3s and movies
• filing cabinets full of documents from your first 9th grade paper
• your finances
• your latest business model

Not to mention the room dedicated to your photo albums and the basement/workshop with toolboxes containing:

• word processors
• media players
• text editors
• photo editing tools

In real life, of course, this house is your computer. Rather, it is probably a couple of computers: at work, home, or in your pocket. The thought of losing your virtual house to any number of possible disasters (e.g., theft, crackers, disk corruption) is devastating. The dread is a consequence of the technology shift. An information technology shift from the analog to the digital, from physical to abstract. We're approaching an era where this digital house dissolves entirely. Where your information exists everywhere and nowhere, accessible from any device, anywhere.

The Kindle represents an early glimpse of this transformation. With the Kindle you have access to books, newspapers, blogs, Wikipedia, and the internet in general, all without connecting to a computer.

With the Kindle you have access to books, newspapers, blogs, Wikipedia, and the internet in general, all without connecting to a computer. You can buy books through the Kindle store and while they are sent to your device Amazon keeps a copy of everything you purchase (subscriptions, books, magazines) in the Cloud. Not only that but the Kindle automatically sends your notes, annotations, clippings, and bookmarks in the Cloud.

But this post isn't supposed to be Kindle advertisement. What the Kindle lacks is openness. Being wrapped in layers of DRM forces the Kindle into not sharing nicely with others.

Openness will be crucial to life in the Cloud. As we move our data and applications into this new distributed model, we need assurance that our information won't be locked into proprietary silos controlled by a single company. The free and open source software movement has already demonstrated the power of collaborative development and shared standards. These are principles that become even more vital when our digital lives exist across multiple platforms and services.

Reply via: Email · Bluesky