ongoing by Tim Bray

ongoing fragmented essay by Tim Bray

False Creek Friend 27 Jul 2022, 7:00 pm

I want to share a project I’ve been helping out with for the last couple of years; the False Creek Friends Society. I haven’t wanted to write about it before now because it was just big-ideas talk. But there’s some science starting up and if you’re nearby you might want to get involved.

False Creek

Courtesy of OpenStreetMap.

What’s a “False Creek”?

Wikipedia has the facts. It’s a little piece of the Pacific stabbing into the belly of Vancouver, surrounded by condos, marinas, a cement maker, museums, festivals, Granville Island, and a really nice seawall. I’m writing this in our family boat, my home office several days a week, tied up at one of those marinas. I bicycle on the seawall. It’s a unique, special place and I care a lot about it.

View of Eastern False creek

Near the east end. There are human and legal stories to be told about all those tied-up boats, in another entry.


I feel a pretty deep connection with False Creek and can’t help noticing a few real problems:

  1. The quality of the water; there are regular no-swim notices from the Health people. There also may be industrial-chemical pollutants left from when it was surrounded by sawmills and factories. But, we don’t actually know much about the nature of the problems. The science just hasn’t been done.

  2. False Creek contains several locations that were central to the lives of the peoples of the Squamish, Musqueam, and Tsleil-Waututh nations that our ancestors stole the territory from. But there are few manifestations of Indigenous culture to be seen.

  3. It’s not a place, it’s a hole in the map with lots of interesting stuff around it.

Foggy day on False Creek

A foggy day under the Burrard bridge.

What are Friends for?

I was contacted in late 2020 by Zaida Schneider, retired journalist and mariner, who mostly lives in his lovely tugboat at another of the marinas, and like me cares about the place. Since then, we’ve talked to a whole lot of people and registered a nonprofit and assembled a seven-strong Board of Directors (I’m one) and launched collaborations. What, exactly, are we trying to achieve?

Well, we have a tactical To-Do List. I strongly suggest that if you are in the neighborhood, you read it. But, we have bigger ideas.

Spring flowers beside False Creek

A spring morning.

How about a park? Canada has National Parks, Vancouver has lots of urban parks, and the province of BC has marine parks. We even have a National Urban Park.

Those Marine Parks are just fabulous, lovely places, but they’re (by design) a long way from where everybody is and to enjoy them, you have to be well-off enough to access a boat.

Why shouldn’t Canada have a National Urban Marine Park, and why shouldn’t it be False Creek, where a half-million people can walk to it and it’s an easy trip for another 2½ million?

Canada has Indigenous Protected and Conserved Areas, which tend to be remote and fairly unspoiled. You’ll never unspoil False Creek, but we could protect it from further damage and heal it where it’s diseased, and prove that a city can live in better harmony with the earth that upholds it and the water that surrounds it. Vancouver has a large Indigenous population; why shouldn’t they lead the protection and conservation?

I would totally love for this to be a hot-spot for Indigenous culture and employment. And, after all, Vancouver is a tourist town, we could educate and delight not only the locals but people from everywhere.

There’s a little problem in that apparently there was never actually an Indigenous-language name for the whole of False Creek. But, you know, that could be fixed.

View of False Creek looking east

Big clouds over the condo towers.

Latest news

Starting in spring, we worked with the Hakai Institute on their Light-Trap program, gathering data on what crustaceans are living under the not-terribly-clear waters of False Creek. We think Dungeness crab may be coming back.

In September, there’s a BioBlitz, sort of the Marine-Biologist equivalent of a hackathon, where in the course of a few days scientists try to build a snapshot inventory of what’s living here.

Come on down!

Lots more projects are brewing. If you’re one of the very many people who either live in sight of False Creek or visit regularly, and our dream of making it something special resonates with you, come out for one of these projects and maybe join the Society!

And anyhow, if you’re in a place where you can visit False Creek, you should do that. And if you’re planning a visit to Vancouver, do drop by. It’s just a really good place. But it could be so much better.

Dark Matter Fun 18 Jul 2022, 7:00 pm

Anyone who’s read one or two of my Long Links curations knows that I’m fascinated by the ongoing conundra in Astrophysics and Cosmology on the subject of Dark Matter. There are a couple of recent developments that are worth noting.


Astrophysicists were kind of forced into concluding that Dark Matter exists for several reasons, but the most obvious is that galaxies spin and the stars on the outer edges are orbiting too fast.

M33 rotation curve

Too fast, that is, given two assumptions:

  1. The only force that matters at galactic scale is gravity.

  2. The action of gravity is (classically, not relativistically) described by Isaac Newton’s Greatest Hit:

    F = G m 1 m 2 r 2

If you’re using a browser that supports MathML (e.g. Safari or Firefox)
you’ll be seeing the famous F=G(m1m2/r2) looking like real math.
Chrome doesn’t think the Web needs math.

Most experts in the field now believe that galaxies are surrounded by huge halos of Dark Matter; arrange it right and then ol’ Isaac’s formula has the stars moving just the way we observe them. It seems like regular matter is a fifth or less of the total amount of gravitational mass in the universe. Dark Matter usefully explains other things about the structure of the Cosmic Microwave Background and the layout of matter in space generally.

The fact that we can’t see it through telescopes or with particle detectors probably means it must be a new form of matter. Which makes physicists’ eyes light up, and they’ve been trying to detect the stuff for decades. And failing, and failing, and failing. They’ve had multiple perfectly reasonable theories for what kind of particles might make up Dark Matter and devised insanely clever experiments to catch them, but no luck.

Eventually someone said, “Uh, about those assumptions…” In particular the second, which is that Newton’s law applies universally, both in the relatively matter-dense neighborhood that I and you, Dear Reader, inhabit, and out in the millions-of-times-less-crowded space at galaxies’ edges.

Thus, Modified Newtonian Dynamics (“MOND” for short). That Wikipedia link has a nice little introduction and the math in the overview isn’t too scary. It introduces a new fundamental constant a0, the (very, very low) level of acceleration where gravity maybe starts to act differently.

MOND isn’t an actual physical theory, it’s just a hypothesis that predicts the observed behavior of most galaxies very nicely without the help of Dark Matter. As noted above, most physicists still think the Dark Matter is out there, we just haven’t figured out how to detect it. But not all physicists.

The Elephant in the Universe by Govert Schilling

What’s new?

Two things. First, there’s a new book giving an overview of the problem: The Elephant in the Universe: Our Hundred-Year Search for Dark Matter, by Govert Schilling, who seems an accomplished science writer. It was given a mostly-positive review in the WSJ by Katie Mack, one of my heroes, so of course I bought it. [Warning: If you follow that link and buy it too, not only will you be sending money to Amazon, I get a cut too.]

It’s a big subject but it’s not a big book; I plowed through it in a single evening. He introduces you to the people leading the Dark-Matter charge and they’re interesting people. It’s even got pictures, and he keeps the math under control. I don’t pretend to deep understanding of Real Astrophysics Math but the author did a good job (in my opinion as the holder of a little-used decades-old math degree) of giving the essential flavors of the problem.

Amusingly, his chapter on the folk interested in MOND is entitled “The Heretics”.


This is currently the largest and most advanced working dark-matter detection experiment, situated in an old mine a little over a mile under the soil of South Dakota. It has just been spun up and the results of its first sixty days’ real science were released on July 7 as a published paper and a presentation: slides and video.

At the center of the detector is seven tonnes of liquid xenon, surrounded by really a whole lot of other stuff. The flavor of Dark Matter they’re looking for consists of Weakly Interacting Massive Particles, which physicists call WIMPs. (Another theorized but not observed flavor of Dark Matter particles is referred to as MACHOs, for Massive Compact Halo Objects.)

Inside the LZ detector

Inside the LZ detector. The real action is in the white central cylinder behind the enclosing transparent one.

I watched the video on YouTube and enjoyed it immensely. The detector is an absolutely fabulous construct, full of raw steampunk beauty, and their description of the problems they’ve had to solve to get the thing to work was absolutely compelling.

So, did they find any WIMPs? They did not, in sixty days’ worth of data. But they plan to run for thousands, and then we’ll see. As with previous Dark-Matter-hunting experiments, they did lower the ceiling on the hypothetical WIMP’s cross section to 6x10-48 cm2. There are two other experiments of a comparable scale spinning up, and the research groups have already got together, decided they need a still-bigger detector, and founded the XLZD consortium to drive that.

Will they find their WIMP?

Nobody knows; that’s the point. But they’re routing millions and millions in research funding at the problem, and literally thousands of researcher-years of effort.

The people who find MOND interesting are dubious. One example would be Stacy McGaugh, a professor at Case Western University, who has a well-constructed and highly readable blog called Triton Station. He discusses the LUX-ZEPLN results in LZ: another non-detection.

He is polite but fairly cutting: “As remarkable a technological accomplishment as experiments like LZ are, they are becoming the definition of insanity: repeating the same action but expecting a different result.”

For their part, the (large majority) who think the Dark Matter is out there and we just have to measure harder are serenely confident.

Isn’t it fun?

Living With Models 13 Jul 2022, 7:00 pm

More people, more often, are finding themselves dealing with the output of Machine-Learning (ML) software. With even a little practice you can spot when it’s happening. It makes me wonder what the future of human/ML interaction looks like. Herewith a few personal experiences.

Gmail spam

One of the world’s largest software systems with an ML component? Spam used to be such a major irritant but now mostly it isn’t except when it is. Obviously, Gmail judges itself by how well it avoids errors both positive (branding legit email as spam) and negative (letting it through to get in my face).

It veers back and forth as they evolve the model. Occasionally some class of spamster figures out how to fool it — I remember when it was translators, most recently it’s subject “Invoice” with a Geek-squad logo in the message body.

But the positive errors are more damaging, because I don’t wanna visit the spam folder ever really, but I feel like sometimes I have to. And when the model goes off the rails in the positive direction, it’s baffling: Posts to mailing lists where I often speak up, notes from teachers about a child’s school issues, things that I really care about.

Protip: Type “/in:spam” into Gmail.

As I write this, Gmail has been running smoothly down the right side of the track in recent weeks, which to say admitting a few spooky spams but filtering out nothing of any value. I bet it doesn’t last.

And I confess to enjoying certain rare classes of spam: Residential real-estate in Mozambique and Mongolia, used Heidelberg presses for sale, lurid love offerings in hilariously broken English.

Jaguar wipers

I mean the ones on my car’s windshield. On an extended drive in variable precipitation, they get dialed in and are amazingly great, flipping the blades back and forth at just the right speed for conditions between drizzle and downpour.

But boy, do they get confused when they’re initially trying to get the picture, flailing furiously at an occasional speckle on the glass, or steadfastly refusing to move when I’m looking through considerable snowflake impact.

Protip: Hit the wiper-fluid button. Which apparently causes it to flush all its caches and begin learning from scratch; almost always yields a sensible state.

Streaming smarts

Back to Google, which is probably OK because they’ve exposed as many humans to ML-model output as any other organization on the planet.

It’s taken literally years, but I have successfully battered YouTube Music’s Your Supermix stream into sanity. I signed up when Google Music kicked the bucket but offered a promise, on its dying breath, to pass along my personal library of twelve thousand or so songs.

It asked me, while getting started, to name some musicians I liked, but then literally crashed when I picked eighteen or so from its alphabetical list. Maybe the problem was they represented too many unique genres?

Anyhow, it eventually got the idea that I liked soft, dreamy stuff and besieged me with Bohren & der Club of Gore and Skinshape and Emancipator which, well OK, all of them, but could we rock out or hear a pop tune occasionally? Yeah, now we can. Here’s a random sampling of my personal “Supermix” on this Wednesday evening in July 2022:

Tim Bray’s YouTube Music “Supermix”

A lot of these are good.

The nice thing is that on another evening it’ll bulk up on Coltrane and Muddy Waters and Brenda Lee. Also, it seems to be sensitive, in a good way, to time of day.

It’s not perfect. In particular, it’s completely fucking useless on classical music.

Protip: Be aggressive with the Thumbs Up/Down buttons.

This model is unlike any other in my experience in that adapts to input stimuli over a time-frame measured in months-to-years.

The future is full of silent struggle

The above are just anecdata. But just as thirtysomethings are the first generation to have grown up never without Internet, today’s very young will be those who will grow up never not interacting with ML models. And remember, the ML model is never on your side; its primary agenda is the agenda of whoever paid to have it built.

I think people younger than I will become skilled at influencing ML models (a skill they mostly won’t even notice they have) and that this skill will increase more or less in parallel with the models’ builders’ skill in meeting the builders’ undisclosed internal business goals.

I retain some shreds of optimism that the humans will stay ahead, at least for a while.

Long Links 30 Jun 2022, 7:00 pm

Dear Readers, once again a selection of links to long-form pieces which I, as a semi-retired person, have time for, but people with normal lives probably don’t. The hope is that one or two will enrich your life. Housekeeping note: I have said “Yes” too often recently and Long-Linking opportunities will probably shrink. Featured this time out: American roots music, BC beer, how wealth vanishes, and the Apple M2.

Bill Strings band with Molly Tuttle

Let’s start with the music. My streamer has noticed that I really like Sierra Ferrell and has been feeding me increasing doses of acoustic American flavors. While I’m not giving giving up on Bach or alt-metal, I’ve enjoyed it. I particularly recommend Billy Strings and Molly Tuttle charging full-throttle through Little Maggie, and AJ Lee and the Brothers Comatose gracefully covering Neil Young’s Harvest Moon.

This one is specific to my part of the world, but hey, we’re a tourist destination and if you’re going to visit us, I think a close study would reward your vacation planning. Local CBC reportor Justin McElroy, who has found a lovely niche in ranking things (neighborhoods, parks, welcome signs) has ranked all the breweries in the bottom left corner of Canada. A few of these were already among my faves and I surely plan to visit more, particularly those within bicycle range.

For your enjoyment, a piece of excellent propaganda. Recently I accidentally studied the Demerara Slave Rebellion of 1823 (even wrote about it). This is a big link in the chain of events and activities that eventually led to the British abolition of slavery. One of the activists’ tools were the 19th-century equivalent of blogs, namely leaflets. Courtesy of the Internet Archive, here is Cushoo: A Dialogue.

Cushoo: An anti-slavery pamphlet

It’s short and snappy (and racist of course, in the manner of its time) but well-done. We know this was effective because the colonial plantation-owners lobbied ferociously to get it banned and, above all, to keep it from circulating among their own slaves. They failed.

There’s no geek-out like a linguistics geek-out. The Economist covers the strange reverse-transatlantic journey of the subjunctive case in modern English. Fun!

Way back when the Net was young, Kevin Kelly suggested that you could make a living with one thousand true fans. Which turned out to mostly not work. But, maybe it’s starting to?

Humans rely on tools. Here’s a convincing curation of nice ones.

OK, this has been a pretty cheery long-link curation so far. But it’s 2022 and There Are Politics and They Are Not Good. Elephant in the Room addresses something that’s going on but is really hard to talk about: The fact that many progressive organizations have been ineffective in recent years because they’ve been tearing themselves apart in an effort to live the progressive ideas they exist to promote. Empirically, we are really bad both at avoiding institutionalized racist/sexist/homophobic culture and at constructively working to get out from under that shadow. This piece is long and (I think) balanced and (I think) important. I have experienced, indirectly, some of the heat and churn being discussed here. We really need to get better at this stuff.

It is no secret that the Big Techs actively try to “manage out” their least effective employees. It’s easy to splash a broad brush of condemnation on this but on the other hand, when an organization is growing by 25%+/year, you’re going to screw up some of that hiring; what to do about it? I sure don’t know. Anyhow, the conversation continues, and I thought “Standing Up For Us Plebs.” Amazon Leaders Reject Policy To Push Employees Out, about an Amazon division going rogue on this issue, sheds illuminating light.

Eric Alterman is a really good writer on politics and popular music. There are a lot of terrible things in the world, and the continuing bloodshed and brutality in Israel/Palestine has been one of them for pretty well my entire life. Alterman surveys the (terrible) state of affairs in 2022 in Israel and Palestine and the Absence of a Solution.

There’s a lot of financial doom-and-gloom going on, you hear stories about how a trillion dollars worth of cryptocurrency value vanished, or how Bezos or Musk or whoever “lost billions”. It’s reasonable to ask “Well, where did that money go?” Noah Smith explains: Where does the wealth go when asset prices go down? It vanishes into nothingness. While I’m as anti-billionaire as the next progressive, I was sort of annoyed, in the recent stock-market run-up, at the feverish headlines about how Bezos was making a million dollars a minute. In fact, wealth due to ownership of financial instruments was being created out of thin air by equity-market machinations; and now it’s being un-created. Poor Jeff has probably lost more money in 2022 than any individual human in history has ever lost before. Not feeling sorry for him. And anyhow it’ll probably all come back. Or maybe not. Whatever. Oh, and the basic politico-financial infrastructure that lets this happen is broken, obviously.

This wouldn’t be a Long Links without an anti-cryptocurrency screed. These days, the stinkiness has become so obvious that it’s kind of hard to find anything new to say. Kyle E. Mitchell, whose blog is called /dev/lawyer, has a new angle in The Work, the Tech, and the Crime. As you’d expect, the viewpoint is that of a working lawyer. Let me pull a paragraph for flavor: “I don’t support cracking down on blockchain people just for being blockchain people. A scene being riven with datajackers, confidence men, and self-taught, emoji-adept bucket shop jockeys does not condemn others not so involved by abstract association, be they merely fools or holdout true believers. But neither do I support special accommodations for those in denial or indifference to the unwelcome company they keep.” Good stuff.

Let’s finish up with a hard-core tech geek-out. (Civilians can close the tab now, my feelings won’t be hurt.) Those who actually care about the nuts and bolts of CPUs and how they fit into the computers in your pocket and on your desk will enjoy Apple M2 Die Shot and Architecture Analysis – Big Cost Increase And A15 Based IP. I’m particularly impressed by the M2 design’s apparent focus on RAM bandwidth and latency; all CPUs wait for memory at the same speed.

Until next time, whenever that is.

Small Tables 25 Jun 2022, 7:00 pm

Computer programs organize bits and bytes into “data structures”. In software of any import, the data structures are usually more interesting than the code around them. This part of the Quamina Diary takes a close look at a very simple data structure that I have greatly enjoyed using to build finite automata, and which I think has lessons to teach; it’s called smallTable.

The problem

As described in a (pretty short) previous episode of this Diary, Quamina matches Patterns to Events, which are flattened into field-name/value pairs for the purpose. We try to match field names and values from Events to those offered in a Pattern. Matching names is easy, they’re immutable strings both in Events and Patterns, and thus the following suffices.

transitions map[string]*valueMatcher

Now, inside that valueMatcher is a harder problem: How to match the value of a field, because patterns can include incantations like *.jpg and "anything-but": ["foo"] and so on. Clearly a map keyed by a string isn’t going to do the job. In fact, we need another flavor of finite automaton that works at the level of characters within fields.

Characters or bytes?

If you want to run an automaton over Unicode text, you have to decide whether you’re going to transition on code points, of which there are potentially 1,114,112 (although only 144,697 are in use as I write this), or on UTF-8 bytes, for which there are only 245 distinct values. Yes, a byte has eight bits, but the values 0XF5-0XFF can’t appear in correctly constructed UTF-8. Which is better, code points or bytes?

On the one hand, here are generally fewer code points than UTF-8 bytes in a string, which means fewer steps to get through your automaton. On the other, the Events coming in over the wire are almost certainly already in UTF-8, and anyhow, a lot of things you’re going to match will be ASCII (for example, numbers), where the number of bytes and characters is the same. For Quamina’s predecessor, which I wrote while at AWS, I picked bytes and never regretted it. And given how easy Go makes it to work with []byte, I wasn’t tempted to do anything else.

Goals and constraints

Let’s say we have a step type that represents a transition between automaton states in response to a UTF-8 byte. Here are two simple Go idioms that represent that kind of state table: map[byte]step and [245]step. Neither of these made me happy, based on prior experience.

First of all, I know that size will be a problem; adding a million Patterns to a Quamina instance is something that people will want to do, then they’ll complain about memory burn.

The second lesson came from gathering statistics on state machines in the wild. Many of the UTF-8 states have only one or two branches, and even in big complicated automata, the average number of out-transitions from a state remains well below 10.

Third, when you’re building automata, it’s common to want to express “transition on a range of byte values.” For example, to match a wildcard construct like *a you transition to a new state when you see a, and loop back on any other value. Similarly, you might be interested in transitioning on anything that’s a decimal digit while matching numbers, or on hex digits while matching IPv6 addresses.

Both the Go map and array fragments above look like they’re going to use way more memory than needed, and in particular Go’s map construct is not simple and not free to initialize. It’s complicated enough that, even after considerable searching, I couldn’t figure out how it actually implements map[byte]. And neither simple option is very appealing for storing byte ranges. So…

Meet smallTable

You can look on GitHub, but here it is:

type smallTable[S comparable] struct {
    ceilings []byte
    steps    []S

It’s generic and the S stands for “step”; the steps are different for deterministic and nondeterministic automata, but the logic for processing them is the same.

First, here’s the code for “find the step that corresponds to a UTF-8 byte value”.

// step finds the member of "steps" in the smallTable that corresponds to a UTF-8 byte value. It may return nil.
func (t *smallTable[S]) step(utf8Byte byte) S {
    for index, ceiling := range t.ceilings {
        if utf8Byte < ceiling {
            return t.steps[index]
    panic("Malformed smallTable")

“Wait,” you say, “that’s going to run off the end of the array!” Here’s why it doesn’t.

const byteCeiling int = 0xf6

// newSmallTable mostly exists to enforce the constraint that every smallTable has a byteCeiling entry at
// the end, which smallTable.step totally depends on.
func newSmallTable[S comparable]() *smallTable[S] {
    var sNil S // declared but not assigned, thus serves as nil
    return &smallTable[S]{
        ceilings: []byte{byte(byteCeiling)},
        steps:    []S{sNil},

Here’s an explanation-by-example of how smallTable works. Suppose we want to model a table where byte values 3 and 4 map to ss1 and 0x34 maps to ss2. Then the smallTable would look like:


Suppose we wanted the same thing, but to map all values other than 3, 4, and 0x34 to ss3. Then we’d have:


Invariant: The last element of ceilings is always byteCeiling. [The attentive reader may be wondering why byteCeiling is 0xf6 rather than 0xf5. We’ll get to that in a subsequent article.]


Everybody knows that dereferencing a hash table (as in map) is O(1), so why would I want to search sequentially? Bear in mind that I’m pulling in a byte array, so I get eight or more ceiling values for each memory fetch, and we know that the size of ceilings is usually less then ten. So I figure that I can run (on average) halfway through the byte array by the time the hashtable code has done a couple of memory fetches and computed a hash. (All computers wait for memory at the same speed.) That’s why ceilings and steps are separate arrays, as opposed to an array of ceiling/step structures. It also helps that the code itself is very compact and should be cache-friendly.

Speculation aside, it used to be a hash table and it didn’t slow down when I switched in smallTable. Plus, it used way less memory. Plus, it represents matches to ranges of bytes very nicely.

It’s not perfect

There’s a lot to like about smallTable. And, one big problem: It’s hard to update. A couple of times I’ve tried to write the code to change an arbitrary byte mapping in an existing smallTable and pretty soon my head started to hurt. So what the code’s doing now is unpacking the smallTable into a unpackedTable:

// unpackedTable replicates the data in the smallTable ceilings and steps arrays.  It's quite hard to
// update the list structure in a smallTable, but trivial in an unpackedTable.  The idea is that to update
// a smallDfaTable you unpack it, update, then re-pack it.  Not gonna be the most efficient thing so at some future point…
// TODO: Figure out how to update a smallDfaTable in place
type unpackedTable[S comparable] [byteCeiling]S

The support routines look like this; the body of code isn’t that interesting:

func unpackTable[S comparable](t *smallTable[S]) *unpackedTable[S]
func (t *smallTable[S]) pack(u *unpackedTable[S])

The profiler shows that in situations where you’re adding a lot of patterns to a Quamina instance, this approach is expensive, especially that pack call. I’m pretty sure that with more head-scratching and a bit of inspiration, I’ll find a faster way to update these things.

smallTable makes me happy. I think that if you’re building an automaton driven by a range of small numbers with modest state fanout, you probably need something much like it.

Demerara Rebellion 12 Jun 2022, 7:00 pm

I accidentally ended up learning a lot about the Demerara Slave Rebellion of 1823. It’s a fascinating set of stories, worth sharing, worth learning about.


I’d always liked the name “Demerara” partly because my Dad loved molasses (on pancakes, toast, whatever) and so my childhood memories are rich with images of a big sticky black pot of Demerara Molasses on the breakfast table. I’ve recently released a software project I’ve been working on for a few months and thought I’d use that name. But I did a web search for “demerara” and thus discovered the 1823 Rebellion (shame on me for not having known about it). Anyhow, I quickly ran across multiple statements that the Rebellion’s leader was Quamina, and loved that name so much that I stole it for my project. Here it is on this blog and on GitHub.

Apologies and triggers

I regard myself as reasonably historically literate but somehow had never learned much about slavery: Its culture or economics or the experience of the enslaved. Again, shame on me.

Now, trigger warnings, and I mean all of them: This story is full of appalling racism and sexual abuse and sordid death and sickening hypocrisy. Blecch.

The sources

There’s been less published on this than you might think. I recommend starting, as I did, with Account of an insurrection of the negro slaves in the colony of Demerara by Joshua Bryant, who was an accountant in Georgetown, the colony capital, and a member of the militia that battled the rebels. The link above is to the Internet Archive, where you can read this for free. It was published the year after the rebellion and is thus very fresh. It’s much more fair than one might expect. Pretty well just the facts and, while the whole thing is implicitly totally racist, I have trouble recollecting anything explicitly racist or mean-spirited from Bryant.

The typography is terrible and the whole book devolves into a series of footnotes that drive the supposed main text into a pathetic couple of lines across the top of the page. But the story is strong and the images are cool.

Retreat of Lt Brady

“Retreat of Lt Brady” from Bryant’s Account.

Plate 9

“Plate 9” (Marine Battalion at Cummingsburgh) from Bryant’s Account. Bryant was a member of this Battalion.

The Gallows

“The Gallows; Five of the culprits in chains, as they appeared on the 20th of September, 1823”. From Bryant’s Account.

The second book is Crowns of Glory, Tears of Blood: The Demerara Slave Rebellion of 1823 by Emília Viotti da Costa, published in 1994. It’s a serious modern work of history, competently if not beautifully written, impeccably sourced, usefully illustrated. I may read more on the Rebellion but, after Bryant and da Costa, I don’t feel I really need to to get the picture.

Below is a picture of a page from da Costa’s book, which I borrowed from the public library. More people should take books out of libraries. You should too.

A page of da Costa’s “Crowns of Glory, Tears of Blood”

For books that I want to take notes on and write about, I’ve become very accustomed to Kindle, which makes this easy. But I was sitting up late working on this one with the help of sticky notes and enjoying the penciled annotations of a previous reader whose identity I’ll never know, and thinking “maybe there’s something to this physical-book idea.”

The story

Demerara produced cotton, coffee, and sugar on plantations which were long and narrow, one of the narrow ends up against the sea or a huge river. The work was labor-intensive, and the labor was slave labor. The slaves were hideously oppressed. Some of the English and Dutch owners were onsite but many were absentees, represented by a local slave-driver (the first occasion I’ve ever had to use that term literally).

Around 1800 the London Missionary Society started a well-organized and well-funded effort to bring Christianity to the slaves; teach them to read, to know Scripture, and to join the church. Many of the slaves took to this eagerly and the congregations were large. The plantation-owner class hated the missionaries, resisted their arrival, and tried in every way to undermine and circumscribe them. They believed (correctly) that slaves who could read, and who regarded their souls as the equals of free people’s, would be less tractable.

The slave trade was outlawed in 1808 (not slavery, just the extraction of human flesh and blood from Africa). This, and the missionary outreach, were part of a growing anti-slavery movement in Britain, led by Wilberforce.

The movement had not been successful in outlawing slavery, but in 1823 they managed to get an Order In Council which forbade the flogging of female slaves and limited working hours to nine per day.

The colonial administration and the planters hated this and dragged their feet on implementing or even announcing it. But word got out among the slaves, many of whom came to believe they had been given their freedom by the King and that the local administration was keeping it from them.

The revolt broke out on the 18th of August, 1823, and was joined by over ten thousand slaves. The level of violence was low; few of the white planters and administrators were killed or even injured. The Governor declared martial law, supplemented his existing military forces by conscripting basically every white male, and sent them out to fight, well-armed and reasonably well-organized.

It was all over in less than a week. Many of the slaves who were perceived as leaders were unceremoniously hanged on the spot, and then there was a series of ludicrously unfair show trials at which basically every slave brought forth was briskly condemned to death. The executions were public and conducted with high ceremony, as a propaganda exercise.

While the Rebellion failed, it may have changed the course of history. But before I come back to that, let’s meet the people involved.

Dramatis personae

The central figure is Jack Gladstone, the slave who actually led the rebellion. He was Quamina’s son, born a slave. As a cooper, a skilled worker, he had considerable scope for travel among the plantations. He was tall and charismatic and passionate, convinced that the slave-masters were withholding the King’s freedom. I’m going to call him just Jack because “Gladstone” was the name of the absentee plantation owner, which slaves inherited by default.

He lost his wife Susanna when she was taken as mistress by a white plantation manager.

Jack was captured a few days into the rebellion and, weeks later, given an unusually extended and ceremonious trial with, unlike the other slaves, assistance from an attorney. Many witnesses, both colonialists and slaves, were called. Bryant’s book includes a lot of material from this trial. The testimony made it clear that once the Rebellion was ablaze, Jack had taken the lead on multiple occasions in protecting the whites from slaves that had extremely well-founded motivations for inflicting extreme violence on them.

Nonetheless, he was sentenced to hang. Not wanting to create a martyr, the colonial Governor applied to the King for mercy for Jack and a few other convicted slaves, on the grounds of his protection of the whites. The mercy was granted, Jack was sold to another slave-owner in Santa Lucia, sent there, and vanishes from history.

John Wray was one of the first missionaries in Demerara, arriving in January 1809. He was doggedly determined in the face of the planters’ resistance, and got considerable support from the administration back in Britain. Eventually he was successful in reaching out to the slaves and built large and thriving congregations across multiple plantations.

His message was not revolutionary, and heavily emphasized the “Christian duty” of obedience and loyalty to the masters. Whatever he said, the message of the equality of souls was implicit in the Scripture the slaves had learned to read, and they apparently drew their own highly reasonable conclusions about justice and morality. Furthermore, Wray himself treated the slaves with respect as humans, and several of them took up roles as church deacons and catechism instructors. He was proactive in trying to mediate disputes between slaves and masters.

He also preached Christian sexual morality, which was being violently contravened every day by the plantation owners, who flagrantly feasted sexually on the infinitely-vulnerable Black women they owned. Once again, the slaves could and did draw the obvious conclusion.

Finally, he fiercely defended the Sabbath, and the right of slaves to attend Sunday service. This was another irritant to the slave-owners, who felt entitled to seven days of work from humans they saw as property.

In 1816, under pressure from the colonists and following on the death of the pious plantation owner who had supported his arrival and residence, John Wray accepted another position in a nearby colony.

Rev. John Smith

His replacement John Smith arrived in early 1817. Smith took up where Wray left off, enlarging still further the slaves’ enthusiasm for and their practice of religion. He is a truly interesting character; a poor simple Englishman who perhaps joined the Missionary cause mostly in search of a living, a man who became possessed by his mission, a man who was a living lesson that Black slaves could and should be treated like humans, a man who became (in private) revolted at the existence of slavery.

Smith may have known that the Rebellion was about to break out, but was apparently consistent in advising the slaves against it. On the other hand, he angrily refused to join the white militia that was putting the Rebellion down.

Finally, Quamina. He was born in West Africa and kidnapped away to Demerara with his mother while very young. He was a skilled worker, a carpenter. An early and enthusiastic adopter of Wray’s missionary message, he became literate, was baptized in 1808, and was elected Deacon in Wray’s church. Later, he worked closely with John Smith and his wife. Both Wray and Smith regarded him highly, praising him in their diaries

Statue of Quamina i Georgetown

Statue of Quamina in Georgetown, Guyana. It’s unlikely the artist had any reliable sources for what he looked like, but the cross and the cutlass capture an important part of the message.

When Quamina’s long-time partner Peggy became ill, his slavemaster refused him time off to care for her; one day after work he found her dead.

Quamina must have known about the planned rebellion. But when it broke out, he strongly advised the slaves to wait, wait for the King’s freedom to be proclaimed. They didn’t.

We don’t know if he participated in the Rebellion, although the colonial government repeatedly proclaimed he was the ringleader. We do know that he swore he would never be taken alive by the whites, and that on September 16, he was tracked down by a colonial military party, ignored their orders to stop, and was shot dead.

The aftermath

Jack Gladstone’s trial was not the main legal event. John Smith was arrested and charged with having promoted “discontent and dissatisfaction, in the minds of the negro slaves toward their lawful masters, managers, and overseers” and various other charges. He was brought to trial before a court martial in October 1823.

Smith represented himself and consciously turned the trial into a piece of political theater. It’s a hell of a story, nicely told by da Costa.

The court martial found Smith guilty and issued the usual death sentence, but an appeal for mercy was sent to the King across the sea. Mercy was granted, but Smith died of consumption in prison before the mercy arrived.

Which led to a huge scandal back in Britain, where Smith was presented as a saint martyred by the cruel, grasping plantation owners. The narrative had the advantage of being pretty well the truth.

While you can argue about details and might-have-beens, historians seem to feel that that the white-hot scandal around the trial significantly advanced the anti-slavery cause, leading inexorably to the Slavery Abolition Act ten years later. So Quamina and Wray and Smith and especially Jack may not have won their battle, but pretty clearly they helped their side win the war.

Things I learned

Reading old books is good! But aside from that…

  • The abolition of the slave trade (well before that of slavery) was a big deal economically because, for a slave-owner, it was much cheaper to buy a slave off the boat than it was to raise a baby slave into adult servitude.

    Also, a lot of the female slaves were unenthusiastic about bringing children into their life of slavery. So without a fresh crop of victims from Africa, the labor force tended to decline.

    So, of course, there was plenty of illegal slave smuggling.

  • Manumission — the freeing of slaves — was not terribly uncommon, but became much rarer after the abolition of the slave trade. Still, it’s unsurprising that the population of free Black people in Demerara grew vigorously. But “free” didn’t work with the plantation economy.

  • You probably already have a bad opinion of colonial plantation slave-owners. The more you learn about them, they more you’ll loathe them. I absolutely cannot begin to understand how people could live in that role and look at their own faces in the mirror. The abuse, violence, exploitation, and rape was continuous and egregious and probably worse than a hyperoverprivileged person like me can imagine.

  • The white-hot screeds against the missionaries and the evils that they would inflict by teaching humans to read, and exposing them to Christian scripture, are astonishing. I quote: “Our colonies are now inundated with canting hypocritical tailors, carpenters, tinkers, cobblers, etc. who, too lazy to work for an honest livelihood in the Mother Country, and charmed with the idea of living in ease and luxury abroad, found it very convenient to become converts to the new light and volunteer to teach the Gospel without the ability to spell one of its verses.” Sounds like a bottom-feeding GOP politician snarling at Wokeness.

  • The planters made a lot of money… for a while. Then they became victim to falling commodity prices, financial chicanery, and the fact that the money and power were in London, held by people who didn’t care about them. Predictably, when their businesses turned sour, the suffering went mostly to the slaves.

  • Working in the plantation fields wasn’t the worst place a slave could find themself. Their owners regularly hired their slaves out in “task gangs” to other colonials. This was generally a terrible experience because the task-gang customer had no incentives to have a good relationship with the rented people, and typically worked them unmercifully.

  • The slaves were not completely without rights; there was an official called the “fiscal” whose job it was to investigate mistreatment and illegal oppression. Having nothing to lose, the slaves were extremely litigious, bombarding the fiscal with an endless stream of complaints. Most, of course, were quickly dismissed, and often the complainers were punished for speaking up. But people in desperate places will do desperate things in the pursuit of justice.

  • The slaves had an internal economy; they had allotments of land to grow produce to farm and to sell in the Georgetown market. These plots of land were called “provision grounds” and there were regular disputes when the slaves’ owners did not grant them the time to cultivate to which they felt entitled or, worse, seized the provision-ground land to grow their own crops on. I find that term “provision grounds”, to be evocative and sad. It haunts me somewhat.

  • The women among the slaves had it way worse than the men. There was a culture of casual brutality against them, the slave-owners telling each other that the women were trouble-makers, recalcitrant, needed “firm discipline”. I can’t even bring myself to write about the sexual predation and the treatment of these women’s children; it’s beyond horrible.

  • Demerara was a long way from London, and its rulers found it convenient to ignore many of the edicts from the Mother Country urging better treatment of the slaves. Those rulers, however, were unsuccessful in trying to prevent the word about these edicts getting out. This pattern of behavior was a key factor in provoking the sense of grievance that led to the Rebellion.

  • The slaves came from several different African ethnic groups, and obviously (for at least the first few generations) retained some of the African languages and cultural patterns. There is evidence that the leadership of the Rebellion was mostly slaves coming from the Akan people, called “Coromantee” by the colonialists. But of these patterns and structures among the slaves, we know almost nothing, because their owners didn’t care and they had no incentive to share. The missionaries, who were eager to tell the slaves’ side of the story, ignored those stories. Sadly, they seem to have been lost.

    After my reading, I feel I know Wray and Smith pretty well. I have much less insight into the real protagonists, Quamina and Jack.

  • The British post-Rebellion political struggles focused around the person of John Smith. The slaves were given no agency by either side. It is shocking but true that it took the imprisonment and death of one not-very-distinguished white man to launch the forces into motion that, in the end, led to a good outcome.

It’s a dark, terrible story. But there are rays of light in it. I am profoundly grateful that I live in an age where my children are at risk neither of becoming slaves nor slavers. Obviously, in the twenty-first century, we are still trying to find ways to heal the wounds inflicted on our society and culture by slavery. Anti-racism is the price of entry to the community of people trying to heal the wounds. But we’re still not very good at it.

Making Code Faster 10 Jun 2022, 7:00 pm

I’ve enjoyed writing software for 40+ years now. Lots of activities fall into that “writing software” basket, and here’s my favorite: When you have a body of code with a decent unit-test suite and you need to make it go faster. This part of the Quamina diary is a case study of making a piece of the library faster.

How to think about it

When you’re sitting down to create software, you probably start with preconceptions about whether and where there are likely to be performance issues. If you’re writing Ruby code to turn your blog drafts into HTML, it doesn’t much matter if republishing takes a few extra seconds. So, write the first cut of that software in the simplest and clearest possible way, without sweating the milliseconds.

But for code that’s on the critical path of a service back-end running on a big sharded fleet with lots of zeroes in the “per second” numbers, it makes sense to think hard about performance at design time.

At this point, I want to call out to Nelson Elhage’s Reflections on software performance. Nelson approvingly quotes the following pieces of conventional wisdom:

  • premature optimization is the root of all evil.

  • Make it work, then make it right, then make it fast.

  • CPU time is always cheaper than an engineer’s time.

I approve too! But… Sometimes it just has to be fast, and sometimes that means the performance has to be designed in. Nelson agrees and has smart things to say on the subject.

And then there’s the background against which I wrote the Quamina code: It’s a successor to similar library I wrote v1.0 of at AWS, and which is in production in multiple high-volume services, running hot. So I didn’t have to speculate about the kinds of stress that would be applied, nor about what kinds of design goofs would produce bottlenecks.

On profiling

Nobody is smart enough figure out what’s making a piece of code slow by looking at its inputs, outputs, and structure. That’s why we have profilers. Fortunately for me, Quamina is written in Go, which means the profiler is built-in, helpful, and doesn’t even slow your benchmarks down much. The other two languages I’ve used mostly in recent decades are Java and Ruby and the profiler situation is for both those languages is kind of shitty. I had to pay real money to get the Java profiler I used at AWS and while it worked, it was klunky, not fun to use.

Whatever language you’re in, have a look around and find a good profiler you can live with and keep it handy. And if you’re in Go-land, go read up on the profiler (avoid the official docs Google turns up, they’re outdated and useless).

The Go profiler has a very decent command line that will show you readouts in a local HTTP server or textual tables or exported as PDF. I find the latter easiest to use but YMMV. But what I actually use is the built-in profiler module in GoLand, JetBrains’ Go IDE. There’s a “run with profiler” button and when you hit that the first thing you get is an IDE pane full of flame graph. Which takes a little while to figure out but once you do it’s brilliant. It might not always show you what your problem is but it sure as hell shows you where to look. Here’s one of those.

GoLand flame-graph output from the Go profiler

The middle bump is loading the test data, the right third is profiling overhead (I think), the app logic is in the left third.
It suggests we have a close look at
storeArrayElementField and oh, we will.

The GoLand profiler module lets you walk around the call graph and seek out hot methods and so on; I never wanted it to do anything it couldn’t.

I’m not going do a how-to on the profiler; it’s not hard to learn, and you should learn it. Anyhow, most of you kids are in VS Code these days, which I assume has good profiler support. And having said all that, the graphics below are the PDF output from the Go profiler, no IDE involved.

Gotta have those tests

So, you understand what the performance problem is, and you have a profiler setup that you know how to use. But just like it says in the first paragraph, good unit tests are totally essential. Because the profiler’s going to show you a problem, and you’re going to refactor and rewrite to fix the problem. You can do that fearlessly if you have a high-coverage unit-test suite that you can run with a single key combo and comes back in ten seconds or less. Otherwise, sucks to be you.

Because when you refactor and rewrite, you will break things! Guaranteed. But if you’ve got that unit-test backstop, you don’t need to worry, because you’ll find out right away, and you can iterate till it passes and find out if your work helped.

If you need to speed up code and it doesn’t have decent unit-test coverage… well, then, writing the unit tests is the price of admission. Which, at the end of the day, is just another reason to never let anyone check code in that doesn’t have those tests in good shape.

Benchmark choices

With a profiler and unit tests, you have two legs of the tripod. The third is a nice data-driven repeatable benchmark. I use the San Francisco Parcels JSON file, a couple of hundred thousand JSON objects comprising 190MB or so. Big enough to get meaningful measurement; small enough that Quamina can run pattern sets on it in a second and change.

The San Francisco JSON objects have several large arrays full of 19-digit floating-point numbers, which is pretty well a worst-case scenario for the flattener, so most people will see better performance than we do while we’re testing. Finding a good benchmark data set is a good investment of your time.

Back Story

V1.0 of Quamina’s ancestor at AWS (I’ve been calling it “The Matcher”) was about fast enough for EventBridge, so it didn’t really get any optimization love. Then when other services started picking it up, we started hearing the occasional gripe. In particular some team down in the bowels of CloudWatch wanted to use it for something and benchmarked it against a few other options that looked slow to me. The Matcher won, but not by very much, at which point it occurred to me to profile it. Did I ever get a surprise.

As discussed elsewhere in this Diary, matching Patterns to Events is a two-step process: First you have to “flatten” the event into a list of pathname/value fields, then you run the matching automaton. When I fired up the profiler, I found that he Matcher was spending like 90% of its time flattening events. Ouch!

I’d been using a popular JSON library to turn the event into a tree structure, then running around the tree structure to peel out the fields. Which requires a whole lot of memory allocation. BZZZZT! Wrong! Anyone with significant optimization experience will tell you to keep memory management off your critical path; some of my biggest optimization wins over the years have come from pre-allocating buffers. At AWS, I juiced up the flattening code but can’t remember most details. Except for, I used a streaming JSON parser.

Cheap JSON

Well, I wasn’t going to do the same dumb thing twice, so my first cut at Quamina’s JSON flattener used Go’s json package’s Tokenizer streaming API. Thus I was pretty disappointed the first time I profiled a big Quamina run and saw the flattener burning 90% of the time, and and then again when the method burning most of the time was some internal thing with Malloc in its name.

A couple of minutes’ research revealed that yeah, that API is known to be slow and yeah, it’s because it gobbles memory like a hungry teenager. There are a couple of alternate community-built tokenizers; I looked at them but for some reason nothing really appealed. Also I’d like for Quamina to have the minimum in dependencies, ideally zero, just because that makes it easier to adopt.

Also, I realized, I had a couple of advantages over someone writing a general-purpose tokenizer. As I’ve mentioned before, one of Quamina’s designed-in optimizations is that once you’ve added a bunch of Patterns to an instance, you know which fields you need to look at while matching and, crucially, which you can ignore. So while you’re flattening, you can skip over lots of fields and for that, the memory cost should be zero.

Second, to match field values, Quamina uses a byte-driven automaton to support things like * and numerics. So there’s no need to turn the field values into strings or numeric data types, everything can be a []byte. And since a message coming from outside is also a big []byte slice, there’s really no need to allocate anything except for sub-slices, and that only for the fields you care about.

Thus was born flattenJSON, a complete (but not general-purpose) JSON tokenizer in 750 lines of Go. It’s a collection of state-machine mini-parsers; one at the top level, then sub-parsers for objects, arrays, numbers, string values, and so on. As of now, it still takes more time to flatten than match an event, but it’s close. Which means the whole thing sped up by a factor of about five.

What next?

Time to declare victory? Well, let’s break out that profiler again. Here are reports on CPU and memory utilization.

CPU profile of the Quamina flattenerMemory allocation profile of the Quamina flattener.

Above: CPU profile. Below: Memory profile.
You can enlarge them, but if you want a quality view
there are PDFs for CPU and memory.

For people who are learning the Go profiler, this command produced files named and with the profile data:

go test -cpuprofile -memprofile  -v -run=TestFlatteningProfile

And here are the command lines that generated the PDFs:

go tool pprof -pdf -show_from flattenJSON
go tool pprof -pdf -show_from flattenJSON

So, what is the profiler trying to tell us? The CPU profile makes it pretty clear that the flattener is spending the bulk of its time managing memory, names like memmove and refillAllocCache are a giveaway. In an earlier revision, the heavy-hitting methods had GC in their names, so at least I’m now avoiding the garbage collector.

If we look what called the heavy hitters, or alternatively just glance at the memory profile, it’s pretty obvious that storeArrayElementField is our prime suspect. Let’s have a look.

// storeArrayElementField adds a field to be returned to the Flatten caller, straightforward except for the field needs its
//  own snapshot of the array-trail data, because it'll be different for each array element
//  NOTE: The profiler says this is the most expensive function in the whole matchesForJSONEvent universe, presumably
//   because of the necessity to construct a new arrayTrail for each element.
func (fj *flattenJSON) storeArrayElementField(path []byte, val []byte) {
	f := Field{Path: path, ArrayTrail: make([]ArrayPos, len(fj.arrayTrail)), Val: val}
	copy(f.ArrayTrail, fj.arrayTrail)
	fj.fields = append(fj.fields, f)

Yep, it makes a new Field struct and then gives it a copy of that array of ArrayPos structs, and then appends the new Field to a list called fields.

It’s that array-copy that’s getting us. Can you think of anything that might reduce the pain here?

I can, but only because I know what this thing is all about. To match correctly, you need to know about the arrays in the element you’re matching so that so that you don’t match fields from different array elements. For example, if you have an array of objects representing the Beatles, you don’t want a match of
{"FirstName":["Paul"], "LastName":["Lennon"]}
to work.

Anyhow, as you run through a long array, it turns out that each Field only needs an ArrayTrail copy in case it has child nodes that will append more stuff to the trail. So if you made the ArrayTrail segmented and Fields could share segments, you could probably re-use the path-so-far segment and avoid a lot of the copying. Or you could think about of having an array of *ArrayPos rather than ArrayPos, so you could share steps and only copy pointers.

Both options would introduce complexity and the second might not even be a big performance win, because later in the matching process you have to run through array and look at each element and you’ll lose a lot of memory locality as you fetch them.

Will I go after storeArrayElementField at some future point? Maybe, if I can think of something simpler. Got any good ideas? Quamina’s already pretty fast.


Test. Benchmark. Refactor. Iterate. It’s not fancy. It’s fun. I have been known to whoop out loud with glee when some little move knocks the runtime down significantly. How often do you do that at work?

Dangerous Gift 2 Jun 2022, 7:00 pm

A friend of mine pinged me online, said “Hey, remember that thing we were talking about doing a couple years ago? I still have one of the domain names, gonna let it go unless you want it.”

I said “sure, I’ll take it”.

Him: “Let me look up how to transfer it.”

Me: “I’ll look up how to receive it. There’s always some bureaucracy and ceremony. But all my domains are at R these days and they’re pretty good at making easy stuff easy.” (“R” refers to a well-known top-25 registrar, whose name I’m withholding to protect the guilty.)

Him: “WAIT. STOP.”

Me: “?”

Him: “I’m at R too. Hold on… this looks easy.” [A minute of silence…] “OK, look and see if you got it.”

So I sign into R and list all my domains. “Holy crap, there it is. That was easy.”

So I told him thanks. But that evening, the transaction kept rattling around in my mind, and I was getting less and less comfortable.

Because I was thinking, maybe a bad actor could use this to SWAT me. Suppose the bad actor has an account at R, held by some anonymous tax-shell company in a remote jurisdiction, and they own plenty of domain names, maybe innocuous, maybe horrific, suggesting torture, suffering children, revenge video, death camps… Suppose they posted truly horrific (and violently illegal) stuff at some IP address on a “bullet-proof” overseas server, pointed one of their names at it, transferred the name to me, and then tipped off law enforcement about this horrific abuse being hosted by some guy named Tim. How long till my front door gets broken down?

That evening, I mentioned it to my spouse who is also my business partner and she said “Oh yeah, I wondered what that was about, I got an email from R saying your buddy had transferred a domain name to you.”

I inquired if they’d asked her to do anything to accept the transfer and she said “No, but it did have a number to call if this wasn’t kosher.”

Which might help avert the nightmare SWAT scenario, assuming you are the kind of person who diligently keeps up on your email inbasket and promptly reads bureaucratic-sounding emails from domain shops.

And anyhow it seemed too obvious; surely there must be some policy or regulation in place to keep this kind of awfulness from happening?

Well, I hang around the IETF (I’m currently co-chairing a very minor working group.) And in the IETF are people who know people who Really Know Their Stuff about how domain names work and are regulated, in practice. So I found one of those people.

I told him the story and asked “Is what happened there legal, and could a bad actor make it look like I operated a bad domain?”

Him: “Dunno about ‘legal’ because I learned what IANAL means about 35 years ago. But, yeah, a bad actor could make you look bad because when the police look at the WHOIS data for the domain name, your info would be there, and it would be assured by your registrar, who is also the registrar for other domains that you own.”

Me: “So, are there any regulations or policies or, you know, laws, that apply here?”

Him: “There are definitely contractual agreements between ICANN and every registrar. Looking at ICANN’s official transfer policy, what happened seems to fall well outside of Section II.C.1.2, which clearly says that R must ‘Obtain confirmation of the Change of Registrant request from the New Registrant, or a Designated Agent of the New Registrant’ before R ‘process[es] a Change of Registrant from the Prior Registrant to a New Registrant’. Sending a message afterwards doesn’t seem to pass the sniff test for that, at least to this non-lawyer.”

Me: “Practically speaking, what do you think might happen if the bad guy made this move and tipped off the right law-enforcement agency?”

Him: “It really depends on how savvy that LEA is. These days, one would hope LEA officers would at least look at who owns the domain name, but you just said that the registrar transferred it to you and changed the WHOIS data to use your full name and address. I don’t see how they could distinguish this from you registering it yourself unless they notice that the name had transferred recently.”

Me: “So, what should we do about this?”

Him: “Tell ICANN. They’ve got a compliance department who deals with registrars and registries who don’t follow the contracts. You could instead just tell R, but I can’t really imagine a scenario where even a great tech support person would both understand the problem and be able to get it to the right people on their legal team in an reliable fashion.”

“Me: OK, will do. It seems likely that if R is doing this, some of the other thousands of registrars are too. Hey, there’s a blog piece in this, and maybe another when it gets resolved.”


This is actually co-authored by me and one of the friends who appears in the conversations above, who prefers to remain un-named.

Field-Value Automata 19 May 2022, 7:00 pm

When I introduced Quamina, I described the core trick: You prepare an arbitrary JSON blob for automaton-based matching by “Flattening” it into a list of name/value pairs, then sorting them in name order. Today, a closer look at how you work with the flattened data.

This is one of a series of essays on programming topics motivated by my work on eventing services at AWS and, since then, on Quamina, a content-filtering library implemented in Go (GitHub).

By example

Consider the following Pattern:

{"x": ["a"], "y": [1, 2]}

It should match any JSON blob containing a top-level x field whose value is the string "a" and a top-level y field whose value is 1 or 2.

Since we sort the fields of incoming events in order of name, we know how to build the automaton. There are two things that aren’t obvious:

  1. Matching each field is a two-step process: First, see if the field name matches, then the value.

  2. This pattern doesn’t care about any fields except for the top-level x and y. So the automaton has to bypass all the others.

Given that, here’s the little automaton that could:

simple finite automaton


There are two kinds of things in the picture: field matchers and value matchers, labeled in the diagram with names beginning fm and vm. In Go code, they’re implemented by types named fieldMatcher and valueMatcher.

The matchers transition straightforwardly, looking for an x field with value "a" and then a y field with value 1 or 2. But, each of these matchers have a * representing “anything else” that loops back to the field-matcher state. This is how fields that don’t appear in the pattern are ignored. The automaton loops in fm0 until it sees an x field, then moves along to fm1 if the field’s value is "a", otherwise looping back to fm0. If it does get to fm1, it’ll loop there forever until it sees a y field with value 1 or 2, transitioning to fm2 or fm3 respectively and bypassing any other fields we don’t care about.

The big red !T on fm2 and fm3 says reaching either state means you’ve matched the pattern T that was added in the code sample above.


I’d used finite automata more than once before this, but on every other occasion I was parsing a programming language or Internet data format, which have none of this “skip any fields you’re not looking for” crap. Which meant that the state machine I drew on the whiteboard in my AWS office looked like the one just above, but without those loopbacks labeled *. Yeah, even though they’re implicit in the design. What can I say, I was in a hurry.

So when I sat down to write the code to traverse the automaton, I just couldn’t figure out how to make it work. I felt, as I often feel, that I’d gotten in over my head.

Since I wasn’t smart enough to write down the correct automaton, I had to figure out how to code around the one I had. Sitting on the living-room couch in my Mom’s over-full house at Christmas 2014, I came up with this. Go check it out if you like reading code, but let me try to explain it…

In English

You have an automaton that looks like the picture above, only without the * back-links. It has a start state. You don’t know which fields in the event (if any) are going to match the pattern. You do know that any match has to begin with the start state.

So, you make a little type called a Proposal, which says, state S might match field F. And you have a pool of Proposals to work on. To start with, that pool contains one proposal for each field, suggesting that the start state might match it.

Then you turn a loop loose that runs as long as there are any proposals in the pool. It reads a proposal and tries matching its state to its field. If it works, which means the field name/value combo transitions to another state, you toss proposals for that state and all the following fields back into the pool. Let’s work a quick example

Suppose you have 3 fields (F0, F1, F2), so you load the pool with proposals for F0/Start-State, F1/Start-State, and F2/Start-State. Let’s say that neither F1 nor F2 matches the Start-State, but F0 does, transitioning to State-X. So you toss proposals for F1/State-X and F2/State-X into the pool. F1 doesn’t match State-X but F2 does, transitioning to State-Y. State-Y has an annotation that you’ve matched a pattern, so you have something to return to your caller. The pool is now empty and there are no more fields after F2 to build new proposals, you’re done.

The fact that the fields are sorted by name really matters here; as you work your way through the automaton, you never have to worry about transferring back to a previous state.

It’s all in less than fifty lines of code (once again, starting here), which I’m not going to try to squeeze into this skinny blog column. If you’re looking at that code, please ignore (for the moment) the static about “exists:false matches” and “Array conflict”; but both are maybe interesting enough to get a write-up later in this series.

I don’t know if this approach to traversing automata (a) has been investigated, (b) has a name, or (c) is any good. I do know that it worked really well in practice, handling many millions of events per second in multiple AWS services.

I’ve done a bit of pen-and-paper analysis and don’t think the amount of work is meaningfully different from a conventional traversal. But it did occur to me that in principle this approach could be made multithreaded; you could process multiple proposals in parallel on different cores. But anyhow, the profiler says this part of traversing the automaton is hardly visible as part of the total compute. So I left it this way in Quamina for sentimental reasons.


In the first cut, the field matcher was just map[string]*valueMatcher and the value-matcher was map[string]*fieldMatcher. It worked OK and the fieldMatcher is still like that but, for reasons I’ll write about later, that’s a bad choice for the valueMatcher.

Smarter than me

At some level, Quamina is. One symptom is places like this in the code, distinguished by extended verbose comments. These are where I got stuck and bashed my head across the wall until I got something that worked, and knew I’d have no chance of understanding it later (nor would any subsequent visitor) unless I could squeeze out a coherent English explanation. As Prof. Feynman said, if you can’t explain something in simple language, you don’t really understand it. The observation that computer programmers can build executable abstractions that work but they then have trouble understanding is not new and not surprising. Lots of our code is smarter than we are.

The code is also smart because at AWS I had extremely talented collaborators who added things that I didn’t think were possible until they worked. The proportion of the useful ideas in here that are actually mine is probably less than 50% now.

Finally, we had the insane luxury of running this in production against millions-per-second event flows and watching what broke. And of hearing from other teams using it about what they had managed to break. I guarantee: Nobody is smart enough to predict the behavior of software under this kind of stress without experiencing it.

So if you are face-to-face with a piece of production software that’s new to you, and find yourself feeling stupid, don’t worry about it. That software represents a collection of flashes of peak-energy inspiration from people more or less as smart as you, and there were probably quite a few of those people, and it’s been enhanced with the wisdom that comes only from being used at length for real work.

If course it’s smarter than you. That’s OK, you can still improve it.


As of late May, Quamina has picked up a couple of collaborators with way more GitHub expertise than me, and its repo is growing all sorts of bells and whistles, mostly on the CI/CD front. Which means that I hope to do a release next month and see if anyone actually wants to use this. Also, more stuff to write about!

Golang Diaries: Generics 14 May 2022, 7:00 pm

My coding time this year has been invested in Quamina, an event-filtering library implemented in Go. Just in the last couple of weeks I spotted an opportunity to bring Go’s shiny new generics to bear, and not just anywhere, but for a central data structure. I got good results, but the process was sort of painful; I kept trying things that looked like they ought to work but didn’t. I’m sharing this tour through that experience because I’m a reasonably competent programmer with mainstream tastes and if I hit these bumps in the road others probably will too.

[The title of this piece suggests it’s a continuation of two Golang Diaries entries I wrote back in 2013 when I was first discovering the language. Why not? I have at least one more in mind.]

Why generics?

Quamina is all about building and running finite automata. To do this you need a table-like data structure that represents states and the transitions between them. Quamina uses both deterministic and non-deterministic finite automata, DFAs and NFAs for short. The only real difference is that a transition from a DFA state is always to one other state; in an NFA you can have transitions to multiple others, in practice to a list of states.

I noticed that the NFA and DFA tables were almost identical, as was the code that that built them and stepped through them while matching. I had recently read When To Use Generics on the Go Blog, which has sections entitled:

  • When using language-defined container types,

  • General purpose data structures, and

  • Implementing a common method.

Check, check, and check. Then in the final section, One simple guideline, it says “If you find yourself writing the exact same code multiple times, where the only difference between the copies is that the code uses different types, consider whether you can use a type parameter.” Check. So, off I went. Later that week it was all working. But along the way I yelled at the computer more than I’d have expected.

I don’t recommend diving into the Quamina source at this point, there’s too much else to explain. So…

Back to basics

To help illustrate my bumpy road, I made a toy GitHub project with a file called typing.go. It defines a genericized container type:

type container[S comparable] struct {
  label string
  list  []S

And one associated function:

func (c *container[S]) size() int {
  return len(c.list)

Now I’m going to define a boring struct type:

type fish struct {
  species string

Strong typing oops

Here’s where things started going off the rails. I offered the following and neither the IDE nor the compiler complained:

type fishTank container[fish]

“Wow”, I told myself, “now I have a strongly-typed container, so I can write nice clean code without any generics cruft!” On that basis, I defined another function:

func (t *fishTank) fishCount() string {
  return fmt.Sprintf("How many fishies? %d!", t.size())

Ouch! The Goland IDE said Unresolved reference 'size' and the compiler (via go build) said t.size undefined (type *fishTank has no field or method size).

It seems to me like fishTank should have a size() method? After all, it’s a container, and the compiler seemed to be OK with me saying so, and container has size().

Note: Dear reader, as I proceed through this narrative, if you are Go-erudite it will doubtless occur to you that:

  1. There’s a better (and supported) way to do what I’m trying to do.

  2. I shouldn’t want to do what I’m trying to do.

I welcome the first type of remark; one goal of writing this piece is to provoke them. As for the second, I really don’t care what what you think I should want. As I said above, my hypothesis is that enough others will also want to do this type of thing that the issue’s worth addressing.

Update, the morning after (see #1 above)

So there is a way to accomplish what I want, like so:

type fishTank struct {

Thanks to the commenters below and over at YCombinator for the guidance. It’s not obvious to me that this is “better”, though. Um, would it be rude to say that it’s counter-intuitive and requires understanding things that ordinary folks in the code mines shouldn’t have to? But OK, it works, and I’m going to go back and make some pieces of Quamina a little more readable.

Fat fish

Let’s enrich that fish type a bit:

type dataRichFish struct {
  species string
  datum   [8]float64

Now, suppose this: First of all, not all fish have that datum material available. Second, this app goes into production and we start to track a whole lot of fish, millions and millions. And people start yelling at us to use less memory. Well, Go’s interface mechanism should make this easy:

type flexibleFish interface {
  getSpecies() string
  getDatum()   [8]float64
func (f *dataRichFish) getSpecies() string {
  return f.species
func (f *dataRichFish) getDatum() [8]float64 {
  return f.datum

var dataNotProvided [8]float64
type stingyFish struct {
  species string
func (f *stingyFish) getSpecies() string {
  return f.species  
func (f *stingyFish) getDatum() [8]float64 {
  return dataNotProvided

See how it works? A flexibleFish can either have an attached datum or not and if it doesn’t, it signals the absence by returning the static global dataNotProvided.

Now let’s put those flexibleFish in the tank…

type flexibleTank container[flexibleFish]

Ouch! Quoth the compiler: flexibleFish does not implement comparable.

So interfaces aren’t comparable? Hmm, but I can make(map[interface{}]int) and don’t you have to be comparable to be a map key? I guess an interface could be implemented by something whose type was []int, which isn’t comparable. OK, the shallowness of my understanding is showing here.

But, suppose there was a way to promise that a type was comparable. It looks like there is, at least in Go 1.18. Let’s try it.

type comparableFlexibleFish interface {
  getSpecies() string
  getDatum()   [8]float64

The compiler seems to be OK with that. So, one last step…

type comparableFlexibleFishTank container[comparableFlexibleFish]

Ouch! Goland: interface includes constraint elements, can only be used in type parameters. Compiler: interface is (or embeds) comparable. I understand them about equally well. [Narrator: He’s equally baffled.]

I guess I am skating pretty far outside the lines here? But I think the intent of that comparable in the type declaration is perfectly clear. This thing implements the listed methods and promises to be comparable. So if I try to claim something is a comparableFlexibleFish and it’s implemented the right methods but isn’t comparable, the compiler can stop me with a helpful error.

Once again, maybe I shouldn’t want to do these things, but I do.


The performance was great, no detectable slowdown. [Note this is not always true and if you care, you should really follow that link. But get a large cup of coffee first.]

And I threw away a whole bunch of duplicative code, which made me happy.

When generics arrived in Java, I hated them. I thought the ratio of pain removed to complexity added was way too low. I don’t hate Go’s generics, which is actually pretty strong testimony given that I spent the best part of a couple of days fighting all the stuff described above.

But, my guess is the Go generics story is isn’t finished yet.

Meet Quamina 12 May 2022, 7:00 pm

No parent will name a favorite among their children. But I do have one among my brainchildren, my software contributions over the decades: The event-streaming code I helped build at AWS. After rage-quitting I missed it so much that over the last few months, I wrote a library (in Go) called Quamina (GitHub) that does some of the same things. This is about that.

Quamina offers an API to a construct called a “Matcher”. You add one or a hundred or a million “Patterns” to a Matcher then feed “Events” (data objects with possibly-nested fields and values) to it, and it will return you an array (possibly empty) of the Patterns each Event matches. Both Patterns and Events are represented by JSON objects (but it should be easy to support other Event encodings).

Quamina (and here I beg pardon for a bit of chest-pounding) is really freaking fast. But what’s more interesting is that its speed doesn’t depend much on the number of Patterns that have been added. (Not strictly speaking O(1), but pretty close.) Concretely: On my 2019 Intel MacBook Pro, it can pattern-match several hundred thousand Events per second without much regard for how many Patterns are loaded into the Matcher. The benchmark I use most for optimization has maliciously-constructed rules that force the Matcher to look at basically all the fields in 1KB-ish Events, where the values are arrays of long floating-point numbers. I try to keep this worst-case running at 150K/second or better.


Back in 2014-15 I wrote v1.0 of a conceptually similar library operating inside AWS, which now implements pattern-matching capabilities for EventBridge and quite a few other teams around Amazon, filtering millions and millions of events per second so they can be accepted, rejected, or routed. Lots of super-talented collaborators emerged over the years and I bet that these days, my code isn’t even a majority of the thing.

I’ve been lobbying AWS to open-source it and that conversation continues, it might happen. But I missed working on it so much that I built Quamina. Also, the AWS version isn’t in Go and I thought a Go library might be nice.

I plan to write a few pieces about Quamina. Whether or not its features interest you, it illustrates lessons in dealing with data in motion at a scale that not very many people are lucky enough to be given a chance to code for.


Quamina is based on finite state machines a.k.a. finite automata. There is a rich body of academic investigation into the discrete-math theory of automata, almost none of which I’ve studied. But once I got the basic idea, state machines have been the first tool I’ve reached for across the decades when faced with hard programming problems. I’m not going to introduce the concept here because anyone who’s either got a CS degree or written much software knows the basics.

I like using automata because they have very, very predictable performance. If you’re processing a row of data items you only have to look at each item once, which is a pretty strong constraint on the time you’re going to spend. Also, the typical unit of work in stepping through an automaton is small. Thus, fast and predictable. Assuming I write more on Quamina, there’ll be lots of digging into details of how to make automata useful.

Quamina by example

This is going to be easier if we have an example to work with. Here’s a sample Event:

  "Image": {
    "Width":  800,
    "Height": 600,
    "Title":  "View from 15th Floor",
    "Thumbnail": {
      "Url":    "",
      "Height": 125,
      "Width":  100
    "Animated" : false,
    "IDs": [116, 943, 234, 38793]

(This is actually the canonical example of a JSON object from RFC8259.)

Here are two Patterns that would match the Event:

{"Image": {"Width": [800]}}
  "Image": {
    "Animated": [ false ],
    "Thumbnail": {
      "Height": [ 125 ]
    "IDs": [ 943, 811 ]

For more examples and explanation, see the README.

This doesn’t look much like a typical query or pattern-matching language; the Pattern echoes the “shape” of the Event you’re trying to match. This turns out to have pleased a few developers, one or two of whom have said nice things to me.

Unfortunately I have no recollection of the thought processes that got us there in the white heat of the early design sprints for what became EventBridge. I remember a meeting in Seattle where I got up and sketched a few of these on the whiteboard and people said “It’s not SQL should it be?” and then eventually “Yeah, looks OK”, then we moved on. There must have been some discussion before that?

The trick

The idea of compiling together a bunch of automata and using them to match data is not exactly new; I think the “|” operator was introduced to grep while I was still in high school. The idea of using it on data that takes the form of JSON objects is a little less obvious. Here’s how it’s done:

  1. First, you flatten the Event fields into name/value pairs. Here are a few of the fields in flattened form:

    Image.Width: 800
    Image.Thumbnail.Width: 100
    Image.IDs: [116, 943]
  2. Then you sort them by pathname. This is perfectly OK, because the order of fields in a JSON object doesn’t matter.

    Image.IDs: [116, 943]
    Image.Thumbnail.Width: 100
    Image.Width: 800
  3. Then you do the same trick with the Pattern. For example, let’s use the second one above:

    Image.Animated: false
    Image.IDs: 943
    Image.Thumbnail.Height: 125
  4. Then you make a state machine based on the flattened sorted Pattern and run it against the flattened sorted Event, and away you go.

What next?

Quamina has enough features to be useful and the unit test coverage isn’t terrible. I’d like people to look at it and see if solves any problems they have, or alternately if they see a reason why it’s All Wrong. Bear in mind that there’ll probably be another (more feature-rich) version in another popular programming language coming from AWS later this summer.

It isn’t at 1.0 yet and has no CI/CD, which means I reserve the right to change the APIs. In particular, should the MatchesForJsonEvent() be made a Go generic? (Quamina already has generics internally). I couldn’t stop anyone putting it into production, but I probably wouldn’t if I were you. But if that idea even crosses your mind, please get in touch.

Long Links 6 May 2022, 7:00 pm

Once again, a selection of long-form pieces for those who (unlike me) have jobs and won’t have time for all of, but might benefit from ingesting one or two. This month we have lots of musical offerings, along with politics, advice on how to walk, and 5G.

Let’s start weird. I first ran across the writing of John Michael Greer when his site was called The Archdruid Report because Greer was in fact an Archdruid, which is to say the leader of a niche sect with “Druid” in its name. He’s irritating as hell because his writing is bathed in an atmosphere of “I’m so smart, I can see all sorts of things nobody else can and I’m doing you the favor of sharing my wisdom.” But I still occasionally read him because once you get past that, he does say lots of interesting things. In Whispers of the Fall, he takes up the argument that we are members of a falling civilization. Grim stuff, but good reading.

Hania Rani

These L-L aggregations are always going to have some geeky software stuff. But before I scare away the tech civilians, let’s take our first dip into music, with the amazingly lovely work of Hania Rani, a Polish composer, keyboardist, and singer. I’d put her somewhere on a musical spectrum with Phil Glass and Enya at the ends. A good place to start is Live from Studio S2, but type her name into YouTube and you’ll see lots of great stuff. Several of her performances are collaborations with cellist Dobrawa Czocher, and they’re exquisite.

I’m a little nervous about saying this, but: I just can’t take my eyes off the screen when Ms Rani is on it. She is neither glamorous nor conventionally pretty, but (particularly in that Studio S2 video) she somehow, when performing, radiates effortless beauty so extreme that (for me at least) it lifts up the music.

Now let’s read about fast software. Nelson Elhage’s Reflections on software performance takes up the vexed question about how and when we should think about making software performant. He acknowledges the following three guidelines, which I generally believe in and have intoned at younger programmers more than once:

  1. Premature optimization is the root of all evil.

  2. Make it work, then make it right, then make it fast.

  3. CPU time is always cheaper than an engineer’s time.

And yet, and yet… as Elhage explains eloquently, performance is important, and sometimes you just can’t add it in later. This strikes close to my heart because during my years at AWS my primary coding project was an event-filtering library that was obsessively concerned with performance. I have hope that AWS will open-source it, but in recent months I’ve put in a lot of hours rebuilding it in another programming language, and with this thing, performance is sort of the point. So I can see both sides of this one.

While we’re doing tech, let’s have a look at 5G, about which I recently wrote a highly skeptical blog piece. I remain unconvinced that the huge capital outlay 5G requires will buy much that’s useful. Anyhow, AWS has a product called “Wavelength” that is supposed to deliver 5G’s benefits by locating compute out in the cellular-network relays. AWS just announced New AWS Wavelength Zone in Toronto – The First in Canada. That link is to a blog about putting 5G to work, which I think is valuable because it really leans into the practicalities.

What’s the application that’s going to benefit from 5G wonderfulness? A little knee-high rolling robot used to make small food deliveries, for example a single cup of coffee, within a 2km radius. The benefit is a factor of ten cost reduction, which is nice I guess. But, uh…

Now let’s veer gently away from technology into politics by way of Josh Marshall’s On The Elon Musk Razzmatazz. I’ve been viscerally unhappy about the Musk Twitter takeover, but unable to sort out why. Mr Marshall did. Here’s a quote:

That’s what this is and where we are: an extremely powerful and wealthy jackass on an ego trip. You can take the bro out of the frat house but you can’t take the frat house out of the bro. In fact when you’re worth hundreds of billions of dollars (for now…) you don’t even have to leave the frat house. You can bring it with you. This is a guy whose ideas about speech and also the construction of syllogisms apparently culminated two joints in at a dorm room bull session in sophomore year…

It’s mostly not quite that sharp-edged; in fact, a carefully-considered piece that anyone who cares about Twitter, or about the quality of online discourse, should read.

EROS, by Boris and Endon

Back to music, namely a brand-new release on Bandcamp: BORIS + ENDON / EROS. Boris is a Japanese experimental-metal band that I’ve blogged about twice in Photos of Wata of Boris and Live Metal Is Better. This is a collaboration with Japanese even-more-experimental metallistas Endon, whom I once saw live opening for Boris and sort of dissed in that second blog piece. But here we have 28 minutes or so of thoughtful and well-executed metal which is intermittently quite beautiful. If you like it, you can buy it off Bandcamp and a noticeable proportion of that money goes straight to the artists, so go ahead and do that.

Back to civic policy now, and an issue that’s white-hot where I live and I think in most big interesting cities: Is it OK to build big fancy high-rises in the middle of un-fancy neighborhoods? Density is good, right, but gentrification is, um, mostly bad I guess? Anyhow, the NYTimes has numbers to report: A Luxury Apartment Rises in a Poor Neighborhood. What Happens Next? The numbers aren’t that satisfying, but the take-away is that it’s not obvious that the high-rise does any particular damage. Which isn’t going to make this issue any less vexed, but if you read it you’ll at least have seen some data being thought about.

NYT on guitar soloing

While we’re visiting the Times, let’s veer back into music via a piece that’s pretty well pure fun: Why We Can’t Quit the Guitar Solo. It’s a triumph of fancy Web presentation technology, with cool guitar riffs bursting out at you as you scroll, many of which you will have heard. Only a fool would want a musical life entirely without shredding, so the target here is pretty soft, but the author and Web geeks here have great fun skewering it, and you’ll probably have fun scrolling through it.

Now let’s step sideways by way of Interview: Ramez Naam, futurist, author, and investor. I loathe the term “futurist” so, despite this having been recommended by a Smart Person, went in hostile. But I was won over. I certainly didn’t agree with everything Mr Naam said, but I was never bored, and felt like I’d learned one or two useful things. Perhaps I was drawn in by his arguments for something I profoundly believe: Assuming we don’t all die in the climate catastrophe, there is going to be a freaking huge amount of money made building out the low-carbon energy services as we transfer from now to the future. So if you have some money, finding a way to invest in this project might help save a couple billion lives and also pay back pretty well.

In the last Long Links I enthused about the writing of Chris Arnade. He goes for long walks in interesting parts of the world, and then in parts that most people wouldn’t think interesting, and takes pictures, and writes them up, and it’s never not interesting. Recently he published How to Walk (12 miles a day). Walking for extended distances is, I believe, one of the core competences of Homo sapiens and if you don’t believe me go read Bruce Chatwin’s The Songlines then come back to argue the point; but you won’t. We are almost all city livers now, and thus it’s hard to overvalue good practical advice on how to excel at this essential human behavior in the context that we actually inhabit.

There is, however, a point that is troublesome: Arnade points out that in many long-walk contexts, you just can’t beat the comfort and practicality of socks-under-sandals (good socks and good sandals, of course). But I have been advised more than once by the members of the demographic that I would like to have sex with that I can forget it if I ever appear in that combination. So I don’t, which doesn’t mean that he’s wrong.

Long Time Coming Sierra Ferrell

And now for our musical grand finale: Sierra Ferrell. I listen to her music more or less every day and caught her latest tour during a Covid downcurve. I have rarely been so sure about anything as I am in predicting: Gonna be a big star.

What happened was, she ran away from a broken West-Virginia home to be a hobo — no really, hopping trains and living in camps. Then a few people noticed the talent, then she got onto the tiny-venue acoustic circuit, then a few of her YouTubes started getting millions of views, then the tour I attended (still in progress as I write) is selling out most of its dates.

So, why? Probably most important, the songs are wonderful. Pretty well a pure country aesthetic, but with melodies that you’ll wake up humming and lyrics that go to surprising but truthful places. Then, the voice; she is the kind of singer that arrives only every other generation, if we’re lucky.

Then there’s that beauty thing that I mentioned reluctantly above. I don’t feel in the slightest reluctant to talk about this because she works so hard at it. She never appears twice in the same output, and every outfit is fabulous. As is her facial jewelry, and tattoo, and hairstyle. The night we saw her she wore this floor-length dress that was pure white and covered with reflective sequins so that she sparkled like diamonds, as if there weren’t enough other reasons to watch her. If you drop by her Wikipedia entry, the only picture just now is by me and it’s not very good, only faintly suggesting the visual effect of That Dress. I have no idea if she’s actually conventionally pretty (although I’m pretty sure, to quote Blondie, that her hair is beautiful) because onstage, she has that Star Power that you just can’t look away from.

Ms Ferrell deeply understands how to do online video, so the best way to check her out is to drop by YouTube and type “Sierra Ferrell” into the search bar and watch whatever comes up. One hint: The stuff she’s been posting recently is quite a bit better than those millions-of-views videos from a few years ago.

No, wait, what did I just say? The best way to check her out is to see if the tour’s coming anywhere near you and if it is, drop everything, postpone grandma’s funeral if necessary, and go see the show. You’ll thank me.

4,000,000m Lessons 18 Apr 2022, 7:00 pm

The odometer on my e-bike clicked over to 4K and, rather than a general-purpose “e-bikes are great” rave, I thought I’d assemble a few concrete arguments for them, suitable for re-use with friends and loved ones in the (likely) case that you’re already convinced. With pictures.

4001km on the e-bike odometer

… is prime.

It’s good for you

The skeptics are prone to say that if you want fitness, you should ride a real bike, dammit. They are wrong. Let me end this argument right here and now with Health benefits of electrically-assisted cycling: a systematic review and Riding Electric Bicycles Boon To Health And Not Cheating, Confirms Literature Review and The effect of cycling on cognitive function and well-being in older adults and Metabolic and Cardiovascular Responses to a Simulated Commute on an E-Bike. Or if you want non-academic, there’s a nice NYTimes piece: E-Bikes Can Provide a Good Workout.

It’s important to be clear that by “e-bike” I mean the ones with electric assist; i.e. no throttle, you have to pedal to go, and pedal harder to go faster. Also, you have to keep the boost set at a moderate level. The Bosch electric system that appears on a whole lot of popular e-bikes has four levels: Eco, Tour, Sport, and Turbo. I keep it on Eco and switch to Tour for challenging hills. I’ve never really used the top two levels, nor felt the need.

It’s fast

I find that my average e-bike cruise is something like 25km/h, and I get around town astonishingly fast (note also the numbers in that NYTimes piece above). It helps a lot that Vancouver has a pretty decent bike-route network. And obviously, so does no looking for parking.

Less obviously, when your bike path runs along the side of a road with cars on it that has stop lights and intersections, you realize how agonizingly slow these things are. The endless halts behind someone waiting to turn left or for pedestrians to cross, and behind people inching along looking for parking, you don’t even notice them when you’re driving, but a lot of it you can just breeze through on a bike-path while staying perfectly safe.

For navigating across the city anywhere less than 5km or so, the bike is not going to get me there significantly later than my car. And, I’ll be able to park right in front of where I’m going. And, the parking is free.

It’s never boring

Closely related to the previous point, sitting in traffic look at another car’s ass end, or even worse, stuck behind a truck so you can’t even see ahead, is boring. On a bike, you’re always moving, you’re way more intimate with the scenery, and if you come to a nice view you can stop on impulse and take a picture.

Nice view along a Vancouver e-bike commute

It’s cheap

The bikes themselves are aren’t cheap. For the purposes of this piece, I poked around the landscape to pick out something that’s mid-range, well-reviewed, and from a manufacturer I have respect for: the Trek Verve+ 3. US$3,300 isn’t cheap for a bike but it’s insanely cheaper than anything with four wheels that you’d want to drive, and the running costs are really too low to be worth measuring. I’ve had my bike for three years and a bit and the service charges, including repairs after a pretty bad accident (see below) and a couple upgrades, add up to less than $1,000.

Trek Verve+ 3

It’s good for the city

Look, if you don’t see that a city with more bikes and less four-wheelers is a better city to live and work, there’s nothing I can say that’ll help you. But I will say this: Nobody wants to live in a house on a main road, but a house on one of our city’s main cycle paths would gain value.

It’s good for the planet

This is hardly in doubt, but I stumbled across a good quantitative write-up on the subject, from Britain: How green is cycling? Riding, walking, ebikes and driving ranked. I’m going to reproduce four of their summary bullet points, which widened my eyes, and encourage you to go read the whole piece.

  • Cycling has a carbon footprint of about 21g of CO2 per kilometre. That’s less than walking or getting the bus and less than a tenth the emissions of driving

  • About three-quarters of cycling’s greenhouse gas emissions occur when producing the extra food required to “fuel” cycling, while the rest comes from manufacturing the bicycle

  • Electric bikes have an even lower carbon footprint than conventional bikes because fewer calories are burned per kilometre, despite the emissions from battery manufacturing and electricity use

  • If cycling’s popularity in Britain increased six-fold (equivalent to returning to 1940s levels) and all this pedalling replaced driving, this could make a net reduction of 7.7-million tons of CO2 annually, equivalent to 6% of the UK’s transport emissions

Tell me a couple of those didn’t surprise you.

The infrastructure is human-scale

Nobody loves taking their car in to get fixed. The world of bikes is less monopolized, less exploitive, and generally nicer. By way of evidence I offer this:

Velostar coffee and bikes in Vancouver

At Velo Star, the coffee is good and the baristas are cute. And they replaced my chain for $70 Canadian, tax and labor in.

It’s practical

Once you have an e-bike, you really, really need to spend a little more and get a good pannier. They’ll try to sell you a pair, which I have but only use one. Because I have no trouble getting my laptop, my raingear, and the groceries I picked up on the way home in mine, which doesn’t even look that big.

Did I mention raingear? A decent raincoat and rainpants make all the difference in the world.

Now, let’s be honest. When you’re going to drive away, you just get in the car, fasten the seatbelt, and you’re off. On a bike, you have to get your helmet (and maybe raingear) on, unlock it, and stow your pannier before you’re in motion. When you’re just getting started biking, this feels onerous. But as with most things in life, practice makes perfect. A couple of months in and your muscle memory will have all this stuff — especially the lock/unlock dance — filed away and it’ll happen quickly without even thinking about it.

What about safety?

Yeah, it’s an issue, as you can see below. And I think e-bikes are possibly more dangerous than regular ones. That’s what the numbers say (at least they’re safer than e-scooters).

The author after a bike accident

It’s pretty simple: There are a lot of situations where an e-bike makes it easy to go way faster than is safe. So… don’t go that fast! For example, I got my face decorated, as you see above, when I cut the corner on a T-section in the bike-path, banking through a patch of muddy ground, and BANG! I was on my side on the ground, bleeding.

Before I go on, I want to point out that my helmet saved my life on that occasion. I recommend Bontrager helmets, another product of Trek. And let me confess: When I’m just going five blocks to pick up parmesan and red wine because we’re having pasta, I often don’t wear one. And then I don’t go fast at all.

But yeah, in general, slow down. In particular, I want to point out that bicycle brakes really, really suck on wet pavement, so if you’re going downhill on a rainy road, dial it way the hell back.

My take-away: If you’re careful and if you have a decent bike-route system, the benefits of e-biking overbalance any risk, particular if you don’t bomb along at maximum speed. Now, I do bomb along at maximum speed in places where it’s safe, and damn that’s fun.

E-biking is fun. And good for you. And your city. And your planet. Give it a try.

Three Perfect Tools 13 Apr 2022, 7:00 pm

We live in a world of things that have Product Managers and Design Teams and Documentation Architecture and often vast ambition. And yet, so many of them are so bad. And worse, some which are good then become bad for no good reason. There is a particular joy in a product that just does what you need done, in about the way you expect or (thrillingly) better, and isn’t hard to figure out, and doesn’t change unnecessarily. Here are three to learn from.


At our cabin, we have reasonably modern electric heat that keeps things comfy, but for our occasional winter visits we also have a nice efficient wood-stove. Because we are surrounded by 100+-year-old forest, some of which falls down (or, rarely, is cut if it threatens to fall in the wrong place), there’s plenty of wood to burn. But it comes in big pieces which have to be sawed down to short sections then split into pieces small enough to fit into the stove.

Normally, splitting wood is a joyous form of light exercise — hit the end of the fragment with a decent axe, put a little oomph in it, and the wood springs apart. Very satisfying. But certain pieces of wood are some combination of hard and knotty and twisted and big. Even a reasonably skilled wood-splitter can be driven to distraction.

Spiral splitting wedge

For these, what you need is this Spiral Splitting Wedge, which I bought from Lee Valley. You position it judiciously against the end of the fragment, and pound away with a hammer; I use a nice little one-hand 4lb sledge, which seems about right. You don’t have to hit it that hard, just enough for the blade to bite, then the combination of the wedge’s spread and its spiral twist inexorably rips apart the awfulest, gnarliest firewood. Not quite as satisfying as an axe on a clean straight piece, but then few things are. Just Works. Is cheap.

Then you curl up on the sofa facing the stove, feeling virtuous while the rain lashes the windows and metal roof, and watch the kids toast marshmallows.

I’m pretty sure the design of this product hasn’t changed in my lifetime. And I live without fear that it will change, because the one we have will definitely be working long after I’m gone.


Our cabin barbecue died as a consequence of being left out in the rain too much during the current renovation. It was at least a decade old, and we replaced it with the current equivalent model, a Weber Q 3200 (link is to the Canadian site) because it’s compact, easy to roll around, and can feed 8 people with a little cooking forethought and ingenuity.

Weber Q 3200 barbecue

Compared to the model it’d just replaced, the changes were subtle and thoughtful: A slightly slicker tank retainer, a better spark mechanism, and an elegant little light placed just right for those times you’re cooking after dark.

Why am I so excited about this decent but pedestrian product? Partly because it hasn’t changed; the design is pretty well correct and no hyperoverentitled Product Manager has been allowed to introduce any Bold New Paradigms.

But the great thing, the truly great thing, is the assembly directions. There are a quite a few pieces, but they all fit together in only one way, which is the right way, and they snap into place, and the directions illustrate each step in a way that no-one, no matter how clueless, can misinterpret. You go from flat-pack to working BBQ in an astonishingly short time with not a single good reason to swear.

This appeals to a lot of my core values.

And should you get an invitation to our cabin, I’ll demonstrate the machine’s excellence with a hearty serving of Eastern-Mediterranean chicken kebabs (Arabic: شيش طاووق; Hebrew: שישליק עוף; Turkish: tavuk şiş). They have reduced strong men to tears of joy.

Music writer

My music teacher sent me a Rumba score we were going to work on, which meant picking up the MuseScore app. I hadn’t used it in years, but as soon as I opened it I smiled, because it’s truly great software. Now, you have to know how to read music and the names of the glyphs and markers that appear in it. But if you do, you can teach yourself in almost no time to produce a pretty damn decent-looking score and do so quickly.

I don’t remember ever opening a tutorial or reading a HOWTO; the first time I wanted to capture some music, I just opened it up, dived in, and figured it out.

When was that first time? Last century when I was studying cello, my teacher, a couple times a year, held friends-and-family recitals that filled most of a medium-sized church; they were fun!

One time I performed a duet with my wife Lauren, who has a light, soaring soprano; an arrangement I made of Simon and Garfunkel’s Scarborough Fair/Canticle. For the first part I just played chordal pizzicato background to her voice. For the middle part, with the two voices, the cello took the second, with lots of legato. I think the voice/cello trade-off sounded intense and lovely, and we got a nice round of applause. Here’s a sample of the main and middle sections.

Scarborough Fair/Canticle for voice/cello; main partScarborough Fair/Canticle for voice/cello; middle part

In case anyone out there plays the cello (or any other alto-to-bass register instrument) and wants to take a run at this, here’s the music. It doesn’t include full lyrics or the vocal line but that’s OK because I bet most singers already know it and will be comfy in E minor.

At the event, I used a hand-written score, where by “written” I mean “scribbled”; my handwriting is terrible and that includes music. A couple of fellow students asked for copies so I downloaded MuseScore and within a couple of hours, I had it. When I fooled with that Rumba just recently, I found it similarly effortless.

But wait, there’s more!

The MuseScore app is open-source, free, standards-compliant, well-documented and seems to have a lively community around it. I suspect this is because whenever you’re in a roomful of serious software practitioners, the proportion that are competent amateur musicians is going to be way higher than in the general population. So the number of people who combine programming talent and music literacy isn’t small.

It’s 2022 and the world is terrible but still has good things in it. We need more like these, please.

Long Links 4 Apr 2022, 7:00 pm

Welcome to another Long Links, selected pointers to long-form pieces that have added value to my semi-retired days and one or two of which might reward you if you make time for them. A few of this month’s pieces aren’t free; it’s a bitter fact that the truth is paywalled but the lies are free. Sorry.

Let’s start with climate issues, which I’m cheerful enough to think might be at the center of the world’s attention were it not for Covid and Mr Putin. One of my favorite environmentalists is Tzeporah Berman, who’s been fighting the good fights, and winning a few, for decades. She writes Fear and Loathing in the Climate Era: 8 Thoughts on How We Win. It’s cheerful, practical, and wise.

There’s a certain faction of people who claim to share reasonable concerns about the onrushing climate catastrophe but scoff at us hippies always going on about renewables because, they say, Nuclear is what we really need. I can’t suppress a suspicion that it’s because there’s something manly about nuclear power. Except for, it’s way, way too expensive. The nukies reply that future technologies will fix that. The most plausible of which is Small Modular Reactors (everyone says SMRs). The Economist dives deep in Developers of small modular reactors hope their time has come. I’d be wide-open to them if they turn out to be affordable and deliverable. Per this article, they’re not obviously hopeless, but they’re nowhere near a slam-dunk.

While we’re on the carbon trail, my former employer now offers this new Customer Carbon Footprint Tool. The idea is that if you’re doing cloud computing you might still want to know how much you’re worsening the planet’s carbon lead. This is supposed to do that (haven’t tried it myself). It’s a big deal because for a lot of big enterprises who don’t actually bend metal or refine fluids, their IT might be where their carbon load is concentrated. A little bird told me that when, back in 2019, Amazon decided to sign The Climate Pledge and try to be a better climate citizen, one input was customers beating up on AWS leadership, saying “I need to know my carbon load, and that’s you.” Tip o’ the hat to AWS!

Speaking of not wasting energy, here’s another chance to expose the dangerous lunacy that is the cryptocurrency world. And absolutely nobody is better at that than Amy Castor. Read The gorilla in the room: Yuga Labs investors pretend to care about the planet. They don’t and you’ll probably find yourself agreeing with me about Amy. Her blog is reliably excellent on this stuff.

Let’s dip into politics. Conservative politics. Regular readers know that I’m pretty far out on the left wing, but I make a point of trying to track the other side, because I think we need intelligent conservatives. Which are and have been mostly absent in North America. But, things are getting weird.

There is starting to be a conservative faction saying that our business-dominated civic fabric is doing a really bad job of building a decent society; that monopoly is a problem, and working-class impoverishment, and exploitive health-care economics. Which is perfectly sensible; what’s weird is that this is often coupled with religious absolutism, activism against any sexual expression which is not straight outta the Fifties, and regular flirtation with racism; all packaged up under the banner of being “anti-Woke”. I personally think that anyone inveighing against “Woke”-ness probably has their head up their ass, but like I say, there are also voices saying interesting things.

Like for example Only the Economic Left Can Beat the Woke, by David Reiff. Ignore that “woke” in the title and give it a try. It’s intelligent, well-written and, yes, wrong in places. But it feels like someone you could talk to, unlike anything out of today’s conservative Trumpist mainstream. Sound-bites: “To put it brutally, you can’t fight a culture war when you really have never been interested in culture before.” And “The mainstream right simply doesn’t have the intellectual tools to fight a battle of ideas.”

When someone rants about the awfulness of the “woke”, you can bet that pretty soon, they’re gonna be inveighing against “cancel culture.” Most such rhetoric is slimy unserious partisanship, the hyperoverprivileged whining that they can no longer get away with being arrogantly offensive pricks in public. Can we conclude therefore that the whole cluster of issues is a nothingburger? I’d say “Yes, … mostly.” But damn, it’s a hard subject to talk about. Ken White offers Our Fundamental Right To Shame And Shun The New York Times – What the NYT Cancel Culture Editorial Got Wrong, And What It Got Right. I don’t agree with all of it but it’s careful and made me think about some things I hadn’t before. Out-take: “Similarly, if you denounce ‘cancel culture’ without citing specific examples and suggesting how people should act differently, you’re closer to chilling speech than fixing it.”

Writing this piece led me to a pleasant surprise. I was going to recommend a few pieces by Chris Arnade on the basis that “even conservatives can write smart, eloquent, important things.” I don’t know how I got the idea he’s one of those, because he isn’t. Anyhow, what he’s up to recently is going places and walking around them, taking pictures and notes. For example, Kyiv. And Lima. He also writes things that aren’t pure travel narratives, for example Among the Half-Masked – COVID theater is a sign of a healthy society.

OK, let’s retreat to the comfortable left side of the road for a heartfelt JWZ rant: Following the money. There are few places in the world more likely than San Francisco to drive thoughtful people to righteous rage; this is a fine piece of that.

Famous non-conservatives include Albert Einstein, who in 1949 wrote Why Socialism?, from which this nice little excerpt, with a link to the whole piece.

OK, enough politics. Let’s talk food, and address the question Does Adding Pasta Water Really Make a Difference? Turns out it does! (Also, fun piece.) Don’t say I don’t add value for my readers.

Come On In, by R.L. Burnside

This month’s musical Long Link is R.L. Burnside’s Come On In, which popped up in a random blues-fan stream I follow and oh my goodness, it’s a keeper. Although Wikipedia says R.L. did some innovating on this one, to me it just sounds like pure, minimal, warm-toned and warm-hearted electric blues. Something that the world always needs more of

I’m going to end up on a geeky note, so those of you who are tech-hostile or tech-oblivious can drop out now and my feelings won’t be hurt. Baby’s First AWS Deployment is the story of a first-timer building a little app on AWS from scratch. The author eventually decides to walk away from AWS, for reasons that obviously don’t apply to everyone, but I found instructive and you might too

Let’s close with what I think could become a very, very big story. It’s from The Information which is paywalled (and expensive), but you can see the first couple of paragraphs of Amazon’s Public Shareholders Show Their Hand. It contains lots of juicy insider information that I won’t reproduce here but I think the following is very significant:

“As of February, Bezos had the right to vote just 12.7% of Amazon stock, including shares owned by his ex-wife, MacKenzie Scott. A decade ago, Bezos could vote 19.5%. Back in 1998, shortly after Amazon went public, he owned 41%. Perhaps just as meaningful: This year the combined stakes of two big institutional holders, Vanguard and BlackRock, is almost equal to his. And remember, Amazon doesn’t have a class of supervoting shares guaranteeing its founder control regardless of his ownership, unlike firms such as Alphabet and Meta Platforms.

“What this means is that power at Amazon is gradually shifting to public shareholders…”

Big Tech was distinguished, throughout my career, by having founding leadership still setting its directions. Which in most cases meant a focus on customer success that is not exactly a usual feature of Twenty-First century capitalism. (OK, yeah, I know, Larry Ellison, but still.)

If that changes — if Big Tech becomes just another arena for financial engineers to maximize their extractive power, it’ll be a whole lot less fun. And by the way, an even more attractive target for legislators and regulators, which is a good thing. I suppose it had to happen someday.

5G Skeptic 26 Mar 2022, 7:00 pm

When I was working at AWS, around 2017 we started getting excited pitches from companies who wanted to be part of the 5G build-out, saying that obviously there’d be lots of opportunities for public-cloud providers. But I never walked away convinced. Either I didn’t believe the supposed customers really needed what 5G offered, or I didn’t believe the opportunity was anywhere near big enough to justify the trillion-dollar build-out investment. Six years later, I still don’t. This is a report on a little online survey I ran, looking for actual real-world 5G impact to see if I was wrong.

[Note: When I say “G” or “M” I’m talking about Gbits or Mbits/second.]

Why 5G?

Here’s why the 5G vendors say we need their product:

  1. Faster connections — there is talk of 10G!

  2. More bandwidth, so you can provide data in crowded places like sports stadiums.

  3. Lower latency, because you can put servers in base stations, which in particular could be useful for self-driven vehicles. AWS offers infrastructure: see AWS Wavelength.

Why not?

Disclosure: I’m affected by my personal experience. My home office is on our boat, anchored in central Vancouver. I pay my mobile provider more for extra data and do all my work via a hotspot on my aging Pixel 4. The phone calls what it sees “LTE+” (I don’t claim to understand what that means) which de facto gives me lots of tens of Mbits/sec, plenty enough for heavy Internet geeking with streamed background music, and watching ball games (remember, I’m semi-retired). Interestingly, the Marina also provides a WiFi signal which is pathetically slow and unreliable compared to the 4G data; the notion that WiFi is the gold standard for wireless Internet is pretty well over.

Our family has a cabin an hour’s boat-ride from Vancouver on the shores of Howe Sound. We have “Smart Hub Rural Internet”, which delivers a solid 15-25M down and 10+ up. Plenty enough for four people.

So, I’m having trouble seeing what problem I have that 5G will solve.


Granted: Like many people, at home we have “fiber” Internet which offers hundreds of M so that our family of four can all stream and game at the same time, no problem.

Question: How often do you need more than the 50M or so LTE offers in a situation where it’s cheaper to provide it with 5G than with a wired connection?


This is one that I can sort of believe in. In a football stadium or a big conference keynote, it is possible to provide decent WiFi coverage (I’ve experienced it). Is 5G a cheaper or better way to do that? I don’t know, but it doesn’t sound crazy.


Obviously, this would be a big deal for online PvP gaming, although rollback netcode is producing remarkably good results these days. And then there’s the autonomous-vehicle space. So, once again, doesn’t sound crazy.

The gaming angle sounds more plausible than self-driving, which (I think) would need reliable 5G along the whole route — nobody wants a self-driving feature that randomly cuts in and out as you zip down the road.

The whole low-latency thing is based on co-locating compute in the (many, many) 5G base stations. Note the reference to AWS Wavelength above; having a public-cloud provider run the compute makes all sorts of sense to me. But still, I do wonder about the economics; all that compute is going to add up.

The real question

5G started rolling out in the spring of 2019, so it’s had three years to make a difference. I don’t personally know anyone whose life has been changed by 5G. Yes, family members with newer phones occasionally report that the status bar says “5G”, but I don’t hear that they’re having a different experience.

But I don’t know that many people. So I asked the Internet.

Asking Twitter about 5G

Last time I checked that tweet had around 52,000 impressions, people who had a chance to pipe up and relate their personal or professional experience of 5G. Lots did, so I’ll use a few of the more clueful responses to support this narrative.

Also, note that my readership includes a whole lot of professional software developers. So, let’s see what they say about new kinds of applications and services that 5G has brought to the table.

Theme: 5G vs wired

Does it make sense to replace your existing wired Internet with 5G? As always, the answer is “it depends” but one common theme emerges: If your wired connection is lousy, 5G might be a win.

Or maybe 5G is just better?

On/Off on 5G

Others disagree, but point out that fiber may not be an option.

Chris Hinson on 5GSimon Bisson on 5Gapenwarr 5G

I wonder about the economics of deploying 5G vs fiber. It’s worth considering that 5G replaces the last mile and reduces the on-prem installation work to zero.

On/Off on 5g

Theme: Better coverage

Some report generally better coverage on 5G. Seems a little strange given the early stage of the build-out, but here they are.

Michael Ayres on 5gBrad Jones on 5G

Theme: Hotspotting

Here are claims that hotspotting is better on 5G.

Lewis Ellis on 5GNed Letcher on 5G

Not everyone agrees.

Dave Taht on 5G

The link in his tweet goes here.

I’m dubious too, probably because I live (and wrote this essay, including gathering and editing all the screenshots) on a 4G hotspot.

Theme: Rural coverage

As a special case, I got multiple reports of better rural coverage. Which, once again, is surprising — I thought 5G’s sweet spot was short-range applications.

Jordan Sissel on 5GFariborz Tavakkolian on 5G

Theme: Crowded places

There is personal experience of Internet-in-a-crowded-place. There was another, mentioning rugby, but I seem to have lost it.

Doggzilla on 5G

Theme: Skeptics

Some people share my general skepticism.

Lenne Fiat on 50M being enoughSören on 5GGlen Ross on 5GUmair Qayyum on 5G

Now the developers

My online following isn’t huge, but it contains quite a few senior software developers, people who build high-impact applications and services for Big Tech corporations and who work on foundational open-source projects. So, let’s hear about what they’re building that’s enabled by 5G technology.

Exactly one such developer weighed in on the subject.


I had to go look up what “ATSSS” is. Here you go: “3GPP has started to standardize the Access Traffic Steering, Switching & Splitting (ATSSS) function to enable 5G devices to use different types of access networks, including Wi-Fi. The ATSSS service leverages the Multipath TCP protocol to enable 5G devices to simultaneously utilize different access networks.”

Doesn’t sound terrible. Life-changing? Your call. Um, any other developers out there working on things that will move the needle?

Then there’s this

John Gruber links to and expands on a WSJ report on how 5G burns your mobile’s battery faster than LTE, and offers advice for how to disable it, because: There is nothing I do with my iPhone — nothing — where I find LTE even just a little bit “too slow”.

Me too.

What I think

Well, they built 4G and it worked pretty well. So they had to build 5G, didn’t they? It’s perfectly possible that, ten years from now, we’ll look in the rear-view and say “That was a good investment.” I wouldn’t bet on it, but it’s possible.

Professor Dale and Me 21 Mar 2022, 7:00 pm

To misquote William Gibson: “The past is still here with us, just unevenly distributed.” I have a story to tell that stars Sergei Rachmaninoff and me. It starts just before 1900 in Moscow. My part ends around 1972 in the mountains of Lebanon, looking down over Beirut and the blue Mediterranean. It doesn’t have to do with computers and nothing in it matters to anyone living except me, but maybe you’ll enjoy a faintly exotic musical echo from a long time ago a long way away.

Rachmaninoff in 1897

In 1900, Russian composer Sergei Rachmaninoff (1873-1943), pictured in 1897, was in rough shape. Which is surprising, given that in 1892 he’d graduated from the Moscow Conservatory with the highest possible mark and a “Great Gold Medal”. He had a brief spell in the sun, enjoying a summer-of-1893 retreat in Kharkiv, a name much in the news as I write. But then in 1893 his hero Tchaikovsky died and then his First Symphony got brutal reviews, and a serious case of composer’s block silenced his musical output. Rachmaninoff was reduced to giving lessons to pay the rent.

Nikolai Dahl

Enter Dahl

Meet Nikolai Dahl (Николай Владимирович Даль) (1860-1939), a Russian Empire physician who studied psychology in France after graduating from Moscow University. A lover of music and competent violist, he was engaged by Rachmaninoff in January 1900. Between then and April, a daily program of hypnosis and other supportive therapies had good effect, leading to the composition of the famous and much-played Piano Concerto #2, premiered in October 1901 and dedicated to Dahl.

Dahl had a son, also named Nikolai, whose life intersected mine.

But before we leap forward in time, let history record that Rachmaninoff’s story had a mostly happy ending, in that he kept on writing and playing music until his last days, seemed reasonably satisfied with life, and was never broke. Some regret that performing (he was a hell of a piano player) was more lucrative than composing and thus we are left with less of his music, which is almost all pretty great, than we might have wished.

Professor Dale

In 1962, when I was seven, my father graduated from Oregon State University with a Ph.D. in Agriculture and accepted a faculty appointment at the American University of Beirut (everyone says AUB).

At some point in my youth, music lessons were thought appropriate and I was asked which instrument. When I said “something that plays low notes” they decided that my hands were too small for double bass, so I took up cello. When I bothered to practice I made good progress.

Some years later my parents were looking for a new cello teacher, and someone suggested “Professor Dale”, who lived up in the mountains. Specifically in Brummana, a prosperous, verdant town overlooking Beirut from the slopes of Mount Lebanon. Its climate is pleasantly moderate in summer; that and the view have made it long-time escape for residents of hot climates around the Middle East.

So I became a student of Professor Dale — yes, the son of the physician who treated Rachmaninoff. I don’t know why or when his surname’s spelling changed. More on him later but first, the question of travel.

The road to Brummana

I was a sixteen-year-old kid in Beirut and needed to get a half-hour’s drive up the mountainside for my cello lesson. It was pretty easy, because of the service option. A service (a French word, rhymes with “bear lease”) is a taxi with a fixed route, either circular inside the city or back-and-forth between two points. You could hail them anywhere on their route, you’d be one of up to five passengers, and the price was the same no matter where you got off or on. And it was cheap.

At this point in time basically all the services were “Ponton” style Mercedes sedans with bench seats front and back; thus room for five paying customers.

Meredes-Benz 180

Photo: Lothar Spurzem, CC BY-SA 2.0,
via Wikimedia Commons

Every town in Lebanon had a designated corner in Beirut where its services left from. Brummana’s was in the Burj, the old downtown that used to be a warren of fantastically picturesque souks (bazaars) full of gold and spices and colors and stories. That was until it was shelled flat during the Lebanese Civil War, I forget whether it was by the asshole Phalangists or the asshole Islamists, now it’s a totally boring mall. But I digress.

Souk Ayass in 1966. Photo: Mounir Diab, CC BY-SA 4.0,
via Wikimedia Commons

Anyhow, I’d take my cello downtown on the city bus (cost: pennies), stroll over to the Brummana corner, and catch the next service, they left all the time. My Arabic wasn’t great but plenty good enough to say the equivalent of “C’mon man don’t gimme that, I know the right price.” Nobody’s feelings were hurt, I was a blond foreigner and presumably clueless until I proved I wasn’t. The right price was something like the equivalent of $2 for a half-hour ride. Those Mercedes trunks were cavernous, there was always room for the cello. No driver ever seemed to find it weird when I asked him to pop the lid for the instrument.

I told the driver where in Brummana to pull over, the Professor’s place was only a couple blocks from the highway, and after my lesson I’d stroll back up to the highway and flag a downhill service. At the time, it never struck me that any of this was exotic.

The Professor’s story

With Professor Dale (I never called him anything else) I spoke French; the natural second (perhaps first) language of a pre-Revolutionary Russian aristocrat. He told me that he’d been Second Cello in the Kyiv Opera before the revolution, then he’d had to run for his life. He ended up in Bombay, where there were no orchestra jobs, so he taught himself saxophone and worked in a nightclub playing jazz. “Mais je n'aime pas cette sensibilité musicale des N*, ni ces rythmes” he said — I’m pretty sure “N*” is That Word en français. I’m not going to defend his language, he was 80-ish in 1971 and it might not have occurred to him to deny racism.

I gather that after Bombay he lived in South America for a while; he showed me pictures of him and his wife, young and glamorous on horseback on the Pampas. He never said how he came to the mountains of Lebanon but later I found out; his father, Rachmaninoff’s guy, had settled in Beirut in 1925 and died there in 1939. He played viola in the AUB orchestra, an ensemble in which I briefly played cello; there’s another connection.

Professor Dale loved to ramble on about music; I remember few of the stories. But I remember his repeated criticism of my fingering, his explanation that in the German style, the fingers of the left hand hit the strings like little hammers, while in the French style, they caress them like lovers. And the French style was really much to be preferred. Couldn’t and can’t disagree but to this day I hit the strings a little hard.

Manuscript and Ruth

One time the Professor had a favor to ask. He had a Rachmaninoff manuscript score (I forget of what), and he understood that my father was a Professor and that at the University there would be these new-fangled “Xerox machines”; could I get him a copy? My family was terribly impressed. It never dawned on me to keep one for myself.

It turns out that as I write this there’s a living link to Rachmaninoff. Pianist Ruth Slenczynska, now 97, studied with him, and has just released a new album, featuring Rachmaninoff of course.

Ruth Slenczynska in 1957

Ruth Slenczynska in 1957. Photo: James J. Kriegsmann,
Public domain, via Wikimedia Commons

You can watch a concert she gave just last month and if you do, your mind is connecting through just one other’s to the nineteenth century.

Later on

A colleague of my Dad’s in the faculty of Agriculture had a daughter who also became a student of Professor Dale and was happy to drive us up to Brummana once a week, so I didn’t have to take the service any more. Professor Dale’s wife (always and only “Madame”) was very happy to entertain the Agriculture prof and the student not being taught on the porch swing, serving tea and gateaux while we took in the lavish view down over the pine forests to the jewel-blue sea. Then Professor Dale would join us for more tea and company.

The cello is still in my living room. It is of Czech manufacture, dating from between the wars, so nearing its hundredth year. It sounds fine when I play it, which is rarely; these days I engage in West-African and Afro-Cuban percussion once or twice a week.

I’m going to close with a couple of pictures my Dad took. This is an impromptu quartet that happened one time on the back porch of our family’s temporary residence at the Faculty of Agriculture’s experimental farm, out in the Beka’a valley, not far from the Syrian border.

Flute / Horn / Cello / violin quartet in Lebanon, about early Seventies

It’s a historical curiosity, not really much of a picture. But Dad was a good photographer and that day he also got this.

Cello, Lebanon, 1971

Thank you for listening.

Get Computer Glasses 4 Mar 2022, 8:00 pm

Let me guess: You spend a lot of time in front of a computer, to which is attached a large high-resolution outboard screen, and you’re over 40, and you have glasses on your face. A lot of you will already know this, but if you don’t, run not walk to the nearest optometrist, get your eyes checked, and order a set of “computer glasses”. Trust me on this.

If you’re wearing “reading glasses”

You should go get an eye checkup anyhow, so you can get computer glasses, but also because an eye checkup can find out all sorts of problems that might become serious if untreated, like glaucoma and macular degeneration. Also, they’re pretty cheap, most places.

My eyes are decent for someone past sixty: Galloping presbyopia (“age-related farsightedness”) which has gradually extended far enough that my prescription is for progressive lenses with a mild long-distance boost.

Two pairs of glasses resting on a computer keyboard

Smaller yet bigger

To illustrate the problem, check out the picture above: My ordinary everyday glasses on the left, computer glasses on the right. When I first looked at the picture after taking it I was shocked, because I’d been convinced that the computer glasses were much the bigger of the two, which they’re not. I was fooled because the proportion of my everyday glasses that works at screen distance is pretty small, and located way down in the lower half of the lens. On the computer glasses, the proportion is 100%.

Which means that on my big 4K outboard screen, when I want to glance at the top right corner to check out the time of day, I had to bend my neck way back to get the glasses in the right place so I could read the numbers. When I was working on a blog piece in one window and editing a photo in another I was forever swiveling around to aim the glasses’ tiny sweet spot. A big IDE window with panes for code, project files, debug output, and so on was a problem too. I kept increasing font sizes, but that’s really not a good solution.

With the computer specs, things are just dramatically better, and I’ve turned the fonts back down, and I just do everything faster, and I know what time it is at a glance.

Now, for a laptop down on my lap, the everyday glasses work just fine, which is a good thing because I probably want to switch attention between that and the ball game or costume drama on the big screen across the room.

How to do it

First, grab a tape measure, then sit down and and measure the distance between your eyes and the screen. Then, go to the optometrist and get that examination, then go to a good glasses merchant and show them your prescription and tell them the measurement and say you want computer glasses.

Um, I don’t pretend to be a fashion consultant, but do bear in mind that these glasses are going to feature prominently in all those videoconferences.

I’ll be honest, this is my biggest quality-of-life improvement in several years.

Long Links 26 Feb 2022, 8:00 pm

Welcome to another curation of long-form pieces, the kind of thing that I, as a semi-retired person, have time for. While you probably don’t have the space for all of these, one or two might reward an investment of your attention. These are obviously grim times and indeed there’s grim stuff here but I’ve tried to add some smiles for balance.

The world’s terrible but there’s great new music being created all the time! Let’s start with a palate-clearing blast of excellent hard rock from the “EQD Japan All-Stars”; it’s actually a promo piece for Earthquake Devices (thus “EQD”) who make guitar effects. It’s called the Reincarnation Rose Session, featuring Japanese women-metallistas Wata (from Boris), Tokie, suGar Yoshinaga, and Yuka Yoshimura, the latter two of METALCHICKS. I cannot lie: I have always been attracted to women with large amplifiers. What a great performance, and the drumming is out of this world. Next up, Sierra Ferrell sings Bells of Every Chapel; I just discovered Ms Ferrell a few weeks ago and have tickets to her upcoming Vancouver show. She’s sort of halfway between Dolly Parton and Leon Redbone and she’s wonderful. By the way, did you know that there’s a Guitar Girls magazine? And just to be fair to the old-guy demographic, check out Welcome Back, from Neil Young’s recent Barn project. What a pretty and reassuring sound the band makes.

We’re all weeping for Kyiv, there’ve been a few background pieces about the place when there’s not a war on; the one I read was Kyiv's ancient normality (redux) by Timothy Snyder. Boy, does it ever have a lot of history. Some of it’s even happy. And just to be clear: Vladimir Putin can eat shit and die. Also on my shit list are the opinionators ranging from Jacobin on the left to various GOP hacks on the right saying “yes, Putin bad, but we have to acknowledge that it’s mostly our fault because NATO is bad too, and also we’re too woke.” People who start land wars in Europe deserve eternal torment in the Hell I don’t believe in, and their apologists are standing on perilous ground.

Since we’re talking about unhappy zones, let’s look at Israel/Palestine. The Israelis finally managed to rid themselves of that scuzzbag Netanyahu, but the voters there overwhelmingly prefer far-right parties of various flavors, where by “various flavors” I mean the spectrum between “ethno-nationalist” and “stone racist”. Prime Minister Bennett is trying to “shrink the conflict” by improving Palestinian lives but never, never talking about the bigger Palestinian-future issue. Over at Israel Policy Forum, Michael J. Koplow asks If someone shrinks a conflict but nobody sees it, has it been shrunk? which explains in erudite detail why this isn’t working that well. In particular, there seems to have been an upsurge of settler-on-Palestinian violence; over at +972, Meron Rapoport asks What’s behind the Israeli right’s renewed war on Palestinian citizens? going way deeper into Israeli political dynamics than, well, anything that’s written outside Israel. It’s gotten to the point where Amnesty International released a report using the word “Apartheid” to discribe the situation in the West Bank. At Religion Dispatches, Joshua Shanes offers Rather than attack Amnesty for labeling Israel an apartheid state critics could address the actual problems; as you can guess from the title, he’s not happy. What with Covid and Russia, that part of the world has pretty well fallen out of the news, but the problems don’t seem to be going away.

In my own country, Canada, the big story of course was the protesting truckers and their “Freedom Convoy”, which caught our ruling class entirely by surprise; it took the authorities a big four weeks to get organized and free up the streets of Ottawa. It was pleasing that they managed it with astonishingly little violence. Now that we know the approach works, I expect it to be applied to demonstrations which are led by indigenous people and other marginalized communities as opposed to good ole boys. There’s a pretty decent write-up by Michelle Goldberg in the NYTimes, published about fifteen minutes before the cops waded in. Here’s another take by Canadian finance/real-estate blogger Garth Turner, In the crosshairs. It explores the interesting and I think genuinely new intersections of class and culture and politics that are stirring up North America. Let me quote:

But have you noticed the demographics? Yup, mostly white. Mainly guys.

On the surface this may be centred on vaccines and ‘freedoms’ but it’s really about disenfranchising and exclusion. Covid has exacerbated the wealth divide. The white-collar WFH crowd has been in the bedroom, employed, tapping on a laptop for two years watching their real estate equity grow. The rest have been handed the dirty end of the stick. On top of this economic injustice are governments whose diversity and inclusion policies have favoured certain groups (women, indigenous, racialized) while others (like white dudes in trucks) feel left out. Have you seen a TV commercial for a Canadian bank lately that portrays such a person? Me neither.

The appeal to bank commercials makes sense in Canada, whose economy is dominated by five insanely-profitable too-big-to-fail ones. These institutions advertise like hell on TV all the time, and those ads are sort of The Future Liberals Want; all the families are either multiracial or gay or both. Which is to say, they work hard on avoiding all the -isms (including able-ism, good on ’em), and being inclusive. Which is totally fine by me and in fact what it looks like around my family dinner table these days.

Of course, the people who run the banks also don’t look like the people in those ads. It doesn’t bother me that people who look like me never appear, we had our run in the sun. But the absence is kind of striking, and I can see how it might bother someone who was also getting fucked over by current economic trends.

Not that I sympathize with the truckies in the slightest; they combine incoherence with ignorance and bigotry. But when something doesn’t make sense I want to understand it better.

Another symptom of that same malaise is the propensity of people to believe big lies in general and The Big Lie about the 2020 US election in particular. Over at 538, There Is More Than One Big Lie dives deep not just on the fact of the lie, but on the fine belief structures and their dynamics. I thought it was fascinating.

Speaking of good deep-dives from 538, here are two more: Democrats Helped Build The Social Safety Net. Why Are Many Now Against Expanding It? and Why Democrats Keep Losing Culture Wars. Both useful examinations of extremely puzzling evidence.

These days, when you think of Big Lies, you also think of cryptocurrencies and NFTs and so on. Now here’s a treat: Yanis Varoufakis on Crypto & the Left, and Techno-Feudalism. Varoufakis is one of the world’s really interesting people, and very economically literate. In this interview, he explains why, even if the “crypto” technology was good (it isn’t) and even if the people pushing it had integrity (they don’t), the whole thing doesn’t make in the context of the current landscape of egregious inequality and class war.

I think the whole crypto/NFT scene is nauseating. But it’s not the worst problem we have with online culture. That would the torrent of abusiveness directed at various out-groups, in particular women. Let’s turn the mike over to Aubrey Hirsch’s That’s How It Works When You’re a Woman on the Internet. I suspect most readers here know about the bad stuff that’s happening out there, but let’s never forget it or wave it aside or stop looking for ways to bring to a fucking halt already. Humanity can be better than this.

As I type the words you are now reading, I am engaging in Internet-mediated authoring. Which is still in its infancy and we don’t understand it very well. I think the subject is fascinating and enjoyed the hell out of Andrey Mir’s Social Media Emancipated Authors, But To What End? It includes a taxonomy of “Types of Internet Authors” and I bet anyone reading this would identify with one or more types: Flickering Author, Silent Author, Interjectional Author, Commenting Author, Emitting Author, Principal Author.

But publishing, like most things, is all about money. Brian Morrissey, in The rich niche, discovers that most of the innovative-looking new media projects have something in common: They’re seeking an audience that is described as “sophisticated”, “literate”, “insider” and various other euphemisms for “has lots of money”. Interesting, and with ideas on alternatives too.

Please bear in mind that I’m a geek by profession and taste, and if you’re not, please skip to the next paragraph. Marc Brooker, one of the Really Smart people over at AWS, has been lighting things up over at his blog; consider Will circuit breakers solve my problems?, Software Deployment, Speed, and Safety, and DynamoDB's Best Feature: Predictability. All recommended! I also enjoyed Evan Martin’s Rethinking errors, warnings, and lints, which addresses the question “should you block dev builds on compiler warnings?” and offers a good answer. Finally, for those who want a really deep dive, let me recommend Russ Cox’s excellent piece from 2007: Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, ...) This remains my single favorite piece of exposition around a core piece of both theoretical Computer Science and practical Software Engineering.

Let’s end up on a note of reassurance. Have you been having trouble understanding the dialogue in the movies and serials that you watch? Think it might be because you’re getting old? Well, it might be, but it also turns out that there really truly is an epidemic of lousy, inaudible dialogue. I was overjoyed when I read Here's Why Movie Dialogue Has Gotten More Difficult To Understand (And Three Ways To Fix It); I’m pretty old but I thought my hearing was still good, and maybe I’m right. In fact, this has become such a common piece of conventional wisdom that the New Yorker has a cartoon series: Things Easier to Hear Than Movie Dialogue. Example: “Worm fart”. We can all use a smile.

Page processed in 1.533 seconds.

Powered by SimplePie 1.3.1, Build 20121030095402. Run the SimplePie Compatibility Test. SimplePie is © 2004–2022, Ryan Parman and Geoffrey Sneddon, and licensed under the BSD License.