The old timey systems modeled this problem using accounting 101.
When things change, you don’t immediately update the balance. Instead it is written to a transaction journal aka a log. The thing is this log is the source of truth. State or the balance is derived from the log.
You don’t send a continuous stream of logs. Instead it is batched and sent asynchronously. It is also applied asynchronously. It also records if the batch was successful or not.
If you have multiple systems sending their logs to a central server. No problem. The central server orders them all before applying the batches.
Every so often. The books are “closed”. Meaning the central server won’t accept any more journal entries for things that happened older than X dates.
I'm not an academic but I've worked through practical problems in this space for longer than I'd like to admit, often in ignorance of the literature. The most CS-y fundamental paper would probably be Lamport(1978) below, but the database papers are pretty fundamental on this topic in their own right.
Here are some pointers:
* Astrahan et al., "System R: Relational Approach to Database Management" (1976), oldest paper talking about logs in a database which is kinda what at its root this is recapitulating
* Jim Gray, "Notes on Data Base Operating Systems" (1978), covers logs with the goal of transaction management, undo, redo, the famous "two phase commmit" process for aligning state across a network boundary in two systems
* Leslie Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System" (1978), not a database paper, more about state synchronization in general
* Bruce Lindsey, "Notes on Distributed Databases" (1979), standards for replicating data across multiple identical database nodes
* Jim Grey et al, "The Recovery Manager of the System R Data Manager" (1981), describes "write to the log first, then the database (WAL)" pattern
* Silberschatz, Stonebraker, Ullman: "Database Systems: Achievements and Opportunities"/"A Architecture for Heterogeneous Database Replication" (1980), discusses tricky bits about replaying logs in client database systems that work differently from the producer
* Leslie Lamport, "The Byzantine Generals Problem" (1982), on the math and needed consensus error handling when distributing state
For other later topics on state synchronization, read up on Paxos (Lamport, 1989/98), RAFT (2013), CRDTs (2011), etc.
The above approaches of "write-to-the-log-then-update-state" were applied to UNIX filesystem first in AIX 3.1 in 1990 and then adopted by other UNIX vendors and then in Linux ReiserFS/ext3/XFS(SGI) in 2001ish.
Lotus Notes took a different path in the 1990s to synchronizing state within documents which was not ACID-oriented like the above, but was more like the modern append-only-log with optimistic eventual consistency. Not sure about the best paper on this.
Then Martin Fowler popularized event sourcing in 2005 with Enterprise Application Architecture, and later described CQRS in 2011 and microservices. So "we" decoupled everything with Webhooks (2007) and Kafka (2011) and reinvented this problem. Oh, and did I mention Blockchain (2008), etc. Oh, did I mention Bitkeeper dvcs (1998) and git (2005) handling of distributed state?
Along the way databases (Snowflake, BigQuery, Iceberg, etc) started exploiting logs to show "older" state via features like "Time Travel" queries. Which is actually what Stonebreaker tried to do in the earliest versions of PostgreSQL but the first implementation in the 80s didn't perform well without good compaction support and on much more constrained resources.
Ask your local LLM for a good survey paper and it might be a bit easier to digest, but the above gives you some color and keywords.
I think my is-this-LLM alert is triggered not by mere use of phrases / constructions beloved of popular LLMs, but by things like unmotivated magpie-ish use of those phrases.
I didn't get that reading this (I didn't read the whole piece but I had read the parts you quote before reading your comment).
Often the LLM-beloved constructions are good usage in the right contexts.
Re-reading, I think the article is in a grey area, and I believe that there is a decent amount of human experience underpinning it. It certainly isn't 100% LLM, but it's a long way from 100% human.
But I wrote my commment after reading the article then the spec (https://welidev.github.io/scroll/), and so the spec was "top of mind".
The spec is just awash in LLM-isms. The cadence and rhetorical style are very Claudish. The visual style is basically "Claude's artifact plugin" (it may not be exactly that but it is an incredibly distinct signature). So the experience of reading the spec is very much an "AI slop" experience.
The reason I object to this is that the way these LLMs write is really well-tuned to gloss over small but critical details. And "small but critical details" are sort of the whole field of distributed systems.
This seems to be most true for Anthropic models (I am assuming there is some cultural defect in the way they give feedback), but it seems to be pretty universal, unless you give them some really strong stylistic anchor to a different style.
(As an aside, I sometimes wonder if this is part of the reason that LLMs seem from the outside to be succeeding disproportionately at mathematics: mathematics papers and mathematical notation may be a strong enough cultural force to override Anthropic's lack of taste and unlock the true power of the model).
I'm not saying there might not have been plenty of human guidance, but either way I don't think there's quite enough substance to this (based on everything I wrote in my comment) for this to feel like "a solution" either way.
I think I've been waiting since the 90s hoping somebody will figure out how to make this a real thing (or was it the early 2000s?)
As I recall it seemed to be just one guy, David Chaum, who did so much to show how so many of these things could work, but the rest of us have somehow managed to do very little with his ideas. What are we missing?
You're missing a drive to make billions or wield power, silly.
Of course there's always been ways to do this ethically. But the ones up top don't make money from that. And they can spend billions convincing people the only way it works is with whatever makes them money.
If you can instead construct a list of existing instances to grandfather in, that doesn't suffer from this problem. Of course many linting tools do this via "ignore" code comments.
That feels less arbitrary than a magic number (because it is!) and I've seen it work.
We used this approach to great effect when we migrated a huge legacy project from Javascript to Typescript. It gives you enough flexibility in the in between stages so you're not forced to change weird code you don't know right away, while enforcing enough of a structure to eventually make it out alive in the end.
That might be fine in your context. People's problems are real, though. What they're almost always missing is separating the source code from the compiled output ("lock files"). Pick a tool to help with that, commit both files to your ("one's") project, problem solved.
People end up committing either one or the other, not both, but:
- You need the source code, else your project is hard to update ("why did they pick these versions exactly?" - the answer is the source code).
- You need the compiled pinned versions in the lock file, else if dependencies are complicated or fast-moving or a project goes unmaintained, installing it becomes a huge mindless boring timesink (hello machine learning, all three counts).
Whenever I see people complaining about python dependencies, most of the time it seems just that somebody lacked this concept, or didn't know how to do it with python, or are put off by too many choices? That plus that ML projects are moving quickly and may have heavy "system" dependencies (CUDA).
In the source code - e.g. requirements.in (in the case of pip-tools or uv's clone of that: uv pip compile + uv pip sync), one lists the names of the projects one's application depends on, with a few version constraints explained with comments (`someproject <= 5.3 # right now spamalyzer doesn't seem to work with 5.4`).
In the compiled output - i.e. the lock files (pip-tools or uv pip sync/compile use requirements.txt for this) one makes sure every version is pinned to one specific version, to form a set of versions that work together. A tool (like uv pip compile) will generate the lock files from the source code, picking versions that are declared (in PyPI metadata) should work together.
My advice: pip-tools (pip-compile + pip-sync) does this very nicely - even better, uv's clone of pip-tools (uv pip compile + uv pip sync), which runs faster. Goes nicely with:
- pyproject.toml (project config / metadata)
- plain old setuptools (works fine, doesn't change: great)
- requirements.in: the source for pip-tools (that's all pip-tools does: great! uv has a faster clone)
- pyenv to install python versions for you (that's all it does: great! again uv has a faster clone)
- virtualenv to make separate sandboxed sets of installed python libraries (that's all it does: great! again uv has a faster clone)
- maybe a few tiny bash scripts, maybe a Makefile or similar just as a way to list out some canned commands
- actually write down the commands you run in your README
PS: the point of `uv pip sync` over `uv pip install -r requirements.txt` is that the former will uninstall packages that aren't explicitly listed in requirements.txt.
uv also has a poetry-like do-everything 'managed' everything-is-glued-together framework (OK you can see my bias). Personally I don't understand the benefits of that over its nice re-implementations of existing unix-y tools, except I guess for popularizing python lockfiles - but can't we just market the idea "lock your versions"? The idea is the good part!
> I think ['The Secret War' by Brian Johnson is] an earlier version of ['Most Secret War' by R V Jones]
Most Secret War is R V Jones' memoir of his personal involvement in radar, aerial navigation, aerial surveillance etc. in British military intelligence. I read it a long time ago but I don't remember it as a dry read at all - dramatic times after all.
Brian Johnson was involved in writing and producing the BBC documentary and a book. So it's not an earlier version of the same thing. The documentary is on youtube I think - a good watch, but I'd recommend reading Jones' original book.
For the benefit of anybody thinking "with gitlab I'm safe from this": If you're saying (and perhaps you're not) that some other git hosting service
- gives you control over gc-ing their hosted remote?
- does not to your knowledge have a third-party public reflog or an events API or brute-forceable short hashes?
if so, especially the second of those seems a fragile assumption, because this is "just" the way git works (I'm not saying the consequences aren't easy to mentally gloss over). Even if gitlab lacks those things curently (but I think for example it does support short hashes), it's easy to imagine them showing up somehow retroactively.
If you're just agreeing with the grandparent post that github's naming ("private") is misleading or that the fork feature encourages this mistake: agreed.
Curious to know if any git hosting service does support gc-ing under user control.
I found the post you're replying to helpful (and it made me laugh): I've come across the abbreviation POLA many times, with its non-jokey meaning "principle of least authority". I've also come across "principle of least astonishment" (Larry Wall or some other Perl contributor maybe?) but I'd never noticed that was (presumably?) a jokey reference to principle of least authority - I guess because I came across the joke first back was I was barely a programmer and I've never seen it abbreviated.
But maybe it never was a reference to POLA proper - "principle of least privilege" is more widespread I think, outside of the object capability community. And maybe "least astonishment" came first!
(I'm not serious about 1954 in particular, I am about hoping somebody here knows the CS literature better than me)