You're right, and in this case "foundational work" even undersells how minimal this work really is compared to the results it already gets.

I recommend that people watch Brandt Bucher's "A JIT Compiler for CPython" from last year's CPython Core Developer Sprint[0]. It gives a good impression of the current implementation and its limitations, and some hints at what may or may not work out. It also indirectly gives a glimpse into the process of getting this into Python through the exchanges during the Q&A discussion.

One thing to especially highlight is that this copy-and-patch has a much, much lower implementation complexity for the maintainers, as a lot of the heavy lifting is offloaded to LLVM.

Case in point: as of the talk this was all just Brandt Bucher's work. The implementation at the time was ~700 lines of "complex" Python, ~100 lines of "complex" C, plus of course the LLVM dependency. This produces ~3000 lines of "simple" generated C, requires an additional ~300 lines of "simple" hand-written C to come together, and no further dependencies (so no LLVM necessary to run the JIT. Also "complex" and "simple" qualifiers are Bucher's terms, not mine).

Another thing to note is that these initial performance improvements are just from getting this first version of the copy-and-patch JIT to work at all, without really doing any further fine-tuning or optimization.

This may have changed a bit in the months since, but the situation is probably still comparable.

So if one person can get this up and running in a few klocs, most of which are generated, I think it's reasonable to have good hopes for its future.

[0] https://www.youtube.com/watch?v=HxSHIpEQRjs

An important context here is that the same code was reused for interpreter and JIT implementations (that's a main selling point for copy-and-patch JIT). In the other words, this 2--9% improvement mostly represents the core interpreter overhead that JIT should significant reduce. It was even possible that JIT itself might have no performance impact by itself, so this result is actually very encouraging; any future opcode specialization and refinement should directly translate to a measurable improvement.

Copy&patch seems not much worse than compiling pure Python with Cython, which roughly corresponds to "just call whatever CPython API functions the bytecode interpreter would call for this bunch of Python", so that's roughly a baseline for how much overhead you get from the interpeter bit.

There is no reason to use copy-and-patch JIT if that were the case, because the good old threaded interpreter would have been fine. There are other optimization works in parallel with this JIT effort, including finer-grained micro operations (uops) that can replace usual opcodes at higher tiers. Uops themselves can be used without JIT, but the interpreter overhead is proportional to the number of (u)ops executed and would be too large for uops. The hope is that copy-and-patch JIT combined with uops have to be much faster than threaded code.

A threaded interpreter still has one branch per bytecode instruction; a copy-and-patch JIT removes this overhead.

From the write-up, I honestly don't understand how this paves the way. I don't see an architectural path from a cut-and-paste JIT to something optimizing. That's the whole point of a cut-and-paste JIT.

> . I don't see an architectural path from a cut-and-paste JIT to something optimizing.

One approach used in V8 is to have a dumb-but-very-fast JIT (ie. this), and keep counters of how often each block of code runs (perhaps actual counters, perhaps using CPU sampling features), and then any block of code running more than a few thousand times run through a far more complex yet slower optimizing jit.

That has the benefit that the 0.2% of your code which uses 95% of the runtime is the only part that has to undergo the expensive optimization passes.

Note that V8 didn't have a dumb-but-very-fast JIT (Sparkplug) until 2021; the interpreter (Ignition) did that block counting and sent it straight to the optimizing JIT (TurboFan).

V8 pre-2021 (i.e., only Ignition+TurboFan) was significantly faster than current CPython is, and the full current four-tier bundle (Ignition+Sparkplug+Maglev+TurboFan) only scores roughly twice as good on Speedometer as pure Ignition does. (Ignition+Sparkplug is about 40% faster than Ignition alone; compare that “dumbness” with CPython's 2–9%.) The relevant lesson should be that things like very carefully designed value representation and IR is a much more important piece of the puzzle than having as many tiers of compilation as possible.

In case anyone is interested, V8 pre-ignition/TurboFan had different tiers [1]: full-codegen (dumb and fast) and crankshaft (optimizing). It's interesting to see how these things change over time.

[1]: https://v8.dev/blog/ignition-interpreter

> keep counters of how often each block of code runs ... and then any block of code running more than a few thousand times run through a far more complex yet slower optimizing jit.

That's just all JITs. Sometimes its counters for going from interpreter -> JIT rather than levels of JITs, but this idea is as old as JITs.

There's a lot of effort going on to improve CPython performance, with optimization tiers, etc. It seems the JIT is how at least part of that effort will materialize: https://github.com/python/cpython/issues/113710

> We're getting a JIT. Now it's time to optimize the traces to pass them to the JIT.

Isn't it the case that Python allows for type specifier (type hints) since 3.5, albeit the CPython interpreter ignores them? The JIT might take advantage of them, which ought to improve performance significantly for some code.

That what makes Python flexible is what makes it slow. Restricting the flexibility were possible offers opportunities to improve performance (and allows for tools and humans to spot errors more easily).

AFAIK good JITs like V8 can do runtime introspection and recompile on the fly if types change. Maybe using the type hints will be helpful but I don't think they are necessary for significant improvement.

Doesn't Python already do this? https://www.youtube.com/watch?v=shQtrn1v7sQ

Are there any benchmarks that give an idea of how much this might improve Python's speed?

Well, GraalPython is a Python JIT compiler which can exploit dynamically determined types, and it advertises 4.3x faster, so it's possible to do drastically better than a few percent. I think that's state of the art but might be wrong.

That's for this benchmark:

https://pyperformance.readthedocs.io/

Note that this is with a relatively small investment as these things go, the GraalPython team is about ~3 people I guess, looking at the GH repo. It's an independent implementation so most of the work went into being compatible with Python including native extensions (the hard part).

But this speedup depends a lot on what you're doing. Some types of code can go much faster. Others will be slower even than CPython, for example if you want to sandbox the native code extensions.

This is great info, thanks!

Pypy is a different JIT that gives anything from slower/same to 100x speedup depending on the benchmark. They give a geometric mean of 4.8x speedup across their suite of benchmarks. https://speed.pypy.org/

Isn't CL a good counter-example to that "dynamism inherently stunts performances" mantra?

To the contrary. In CL some flexibility was given up (compared to other LISP dialects) in favor of enabling optimizing compilers, e.g. the standard symbols cannot be reassigned (also preserving the sanity of human readers). CL also offers what some now call 'gradual typing', i.e. optional type declarations. And remaining flexibility, e.g. around the OO support, limits how well the compiler can optimize the code.

But type declarations in Python are not required to be correct, are they? You are allowed to write

    def twice(x: int) -> int:
        return x + x

    print(twice("nope"))

and it should print "nopenope". Right?

The Python language server in Visual Studio Code will catch this if type checking is turned on, but by default, in CPython, that code will just work.

Yep. Therefore it’s better to

   def twice(x: int) -> int:
   if not isinstance(x, int):
           raise TypeError("Expected x to be an int, got " + str(type(x)))
    return x + x

Surely this is the job for a linter or code generator (or perhaps even a hypothetical ‘checked’ mode in the interpreter itself)? Ain’t nobody got time to add manual type checks to every single function.

Of course not. That's what MyPy is for. It was only about the answer to exactly this question in this function.

This can have substantial performance implications, not to mention DX considerations.

Of course, this is not a good example of good, high-performance code, only an answer to the specific question... the questioner certainly also knows MyPy.

I actually don't know anything about MyPy, only that it exists. Does it run that example correctly, that is, does it print "nopenope"? Because I think it's the correct behaviour, type hints should not actually affect evaluation (well, beyond the fact that they must be names that are visible in the scopes thay're used in, obviously), altough I could be wrong.

Besides, my point was that one of the reasons why languages with (sound-ish) static types manage to have better performance because they can omit all of those run-time type checks (and the supporting machinery) because they'd never fail. And if you have to put those explicit checks, then the type hints are actually entirely redundant: e.g. Erlang's JIT ignores type specs, it instead looks at the type guards in the code to generate specialized code for the function bodies.

Or use mypy.

Of course dynamism limits performance (and as said, standard symbols and class is also an unhygienic macro thing) but I meant that you can have both high performance and high dynamism in a programming language, dynamism itself is no excuse to not even try.

Standard symbols being reassigned also breaks macros.

Sort of! But also not really. If you want to get into this, I wrote a post about this: https://bernsteinbear.com/blog/typed-python/

I doubt it with a copy-and-patch JIT, not the way they work now. I'm a serious mypy/python-static-types user and as is they currently wouldn't allow you to do much optimization wise.

- All integers are still big integers

- Use of the typing opt-out 'Any' is very common

- All functions/methods can still be overwritten at runtime

- Fields can still be added and removed from objects at runtime

The combination basically makes it mandatory to not use native arithmetic, allocate everything on the heap, and need multiple levels of indirection for looking up any variable/field/function. CPU perf nightmare. You need a real optimizing JIT to track when integers are in a narrow range and things aren't getting redefined at runtime.

You can't really on type annotations to help interpret the code.

It should be fairly easy to add instruction fusing, where they recognize often-used instruction pairs, combine their C code, and then let the compiler optimize the combined code. Combining LOAD_CONST with the instruction following it if that instruction pops the const from the stack seems an easy win, for example.

If it was that easy, you'd do that in the interpreter and proportionally reduce interpretation overhead.

In the interpreter, I don’t think it wouldn’t reduce overhead much, if at all. You’d still have to recognize the two byte codes, and your interpreter would spend additional time deciding, for most byte code pairs, that it doesn’t know how to combine them.

With a compiler, that part is done once and, potentially, run zillions of times.

If fusing a certain pair would significantly improve performance of most code, you'd just add that fused instruction to your bytecode and let the C compiler optimize the combined code in the interpreter. I have to assume CPython as already done that for all the low hanging fruit.

In fact, for such a fused instruction to be optimized that way on a copy-and-patch JIT it'd need to exist as a new bytecode in interpreter. A JIT that fuses instructions is no longer a copy-and-patch JIT.

A copy-and-patch JIT reduces interpretation overhead by making sure the branches in the executed machine code are the branches in the code to be interpreted, not branches in the interpreter.

This is make a huge difference in more naive interpreters, not so much in an heavily optimized threaded-code interpreter.

The 10% is great, and nothing to sneeze at for a first commit. But I'd actually like some realistic analysis of next steps for improvement, because I'm skeptical instruction fusing and other things being hand waved are it. Certainly not on a copy-and-patch JIT.

For context: I spent significant effort trying to add such instruction fusing to a simple WASM AOT compiler and got nowhere (the equivalent of constant loading was precisely one of the pairs). Only moving to a much smarter JIT (capable of looking at whole basic blocks of instructions) started making a difference.

Support for generating machine code at all seems like a necessary building block to me and probably is quite a bit of effort to work on top of a portable interpreter code base.

I wouldn't be so enthusiastic. Look at other languages that have JIT now: Ruby and PHP. After years of efforts, they are still an order of magnitude slower than V8 and even PyPy [1]. It seems to me that you need to design a JIT implementation from ground up to get good performance – V8, Dart and LuaJIT are like this; if you start with a pure interpreter, it may be difficult to speed it up later.

[1] https://github.com/attractivechaos/plb2

PyPy is designed from the ground up and is still slower than V8 AFAIK. Don’t forget that v8 has enormous amounts of investment from professionally paid developers whereas PyPy is funded by government grants. Not sure about Ruby & PHP and it’s entirely possible that the other JIT implementations are choosing simplicity of maintenance over eking out every single bit of performance.

Python also has structural challenges like native extensions (don’t exist in JavaScript) where the API forces slow code or massive hacks like avoiding the C API at all costs (if I recall correctly I read that’s being worked on) and the GIL.

One advantage Python had is the ability to use multiple cores way before JS but the JS ecosystem remained single threaded longer & decided to use message passing instead to build WebWorkers which let the JIT remain fast.

PyPy is only twice as slow as v8 and is about an order of magnitude faster than CPython. It is quite an achievement. I would be very happy if CPython could get this performance but I doubt.

I can't answer your initial question, but I do like to pile onto the package management points.

Package consumption sucks so bad, since the sensible way of using are virtual envs where you copy all dependencies. Then for freezing venvs or dumping package versions, so you can port your project to a different system, doesn't consider only packages actually used/imported in code, but it just dumps everything in the venv. The fact you need external tools for this is frustrating.

Then there is package creation. Legacy vs modern approach, cryptic __init__ files, multiple packaging backends, endless sections in pyproject.toml, manually specifying dependencies and dev-dependencies, convoluted ways of getting package metadata actually in code without having it in two places (such as CLI programs with --version).

Cross compilation really would be a nice feature to simply distribute a single file executable. I haven' tested it, but a Linux system with Wine should in theory be capable of "cross" compiling between Linux and Windows.

Still, like you, as a beginning I would prefer a sensible package management and package creation process.

"It takes weeks to get simple packages working"

Can you expand on what you mean by that? I have trouble imagining a Python packaging problem that takes weeks to resolve - I'd expect them to either be resolvable in relatively short order or for them to prove effectively impossible such that people give up.

- Trying to figure out what versions the scripts used and specifying them in a new poetry project - Realizing some OS-dependent software is needed so making a docker file/docker-compose.yml - Getting some of it working in the container with a poetry environment - Realizing that other parts of the code work with other versions, so making a different poetry environment for those parts - Trying to tie this package/container as a dependency of another project - Oh actually, this is a dependency of a dependency - How do you call a function from a package running in a container with multiple poetry environments in a package? - What was I doing again? - 2 weeks have passed trying to get this to work, perhaps I'll just do something else

Rinse and repeat.

¯\_(ツ)_/¯ That's python!

Have you taken a look at Nuitka with GitHub actions for cross compilation? https://github.com/Nuitka/Nuitka-Action

I removed the SDKs of some big (big for the wrong reasons) open source projects which generates a lot of code using python3 scripts.

In those custom SDKs, I do generate all the code at the start of the build, which takes a significant amount of time for mostly non-pertinent anymore/inappropiately done code generation.. I will really feel python3 speed improvement for those builds.

Honestly, 2-9% already seems like a very signficant improvement, especially since as they mention "remember that CPython is already written in C". Whilst it's great to look at the potential for even greater gains by building upon this work, I feel we shouldn't undersell what's been accomplished.

> "remember that CPython is already written in C"

What is this supposed to say? Most scripting language interpreters are written in low level languages (or assembly), but that alone doesn't say anything about the performance of the language itself.

I think they mean that a lot of runtime of any benchmark is going to be spent in the C bits of the standard library, and therefore not subject to the JIT. Only the glue code and the bookkeeping or whatnot that the benchmark introduces would be improved by the JIT. This reduces the impact that the JIT can make.

Isn't the point that if pure Python was faster they wouldn't need to be written in other [compiled] languages? Having dealt with Cython it's not bad, but if I could write more of my code in native Python my development experience would be a lot simpler.

Granted we're still very far from that and probably won't ever reach it, but there definitely seems to be a lot of progress.

Since Nim compiles to C, a middle step worth being aware of is Nim + nimporter which isn't anywhere near "just python" but is (maybe?) closer than "compile a C binary and call it from python".

Or maybe it's just syntactic sugar around that. But sugar can be nice.

Also recall that a 50% speed improvement in SQLite was caused by 50-100 different optimisations that each eeked out 0.5-1% speedups. On phone now don’t have the ref but it all adds up.

Many small improvements is the way to go in most situations. It's not great clickbait, but we should remember that we got from a single cell at some time to humans through many small changes. The world would be a lot better if people just embraced the grind of many small improvements...

Marginal gains. https://www.bbc.co.uk/news/magazine-34247629

That's true, and Rust compiler speed has seen similar speedups from lots of 1% improvements.

But even if you can get a 2x improvement from lots of 1% improvements (if you work really really hard), you're never going to get a 10x improvement.

Rust is never going to compile remotely as quickly as Go.

Python is never going to be remotely as fast as Rust, C++, Go, Java, C#, Dart, etc.

Does it matter?

Trains are never going to beat jets in pure speed. But in certain scenarios, trains make a lot more sense to use than jets, and in those scenarios, it is usually preferable having a 150 mph train to a 75 mph train.

Looking at the world of railways, high-speed rail has attracted a lot more paying customers than legacy railways, even though it doesn't even try to achieve flight-like speeds.

Same with programming languages, I guess.

What is the programming analogy here?

Two decades ago, you could (as e.g. Paul Graham did at the time) argue that dynamically typed languages can get your ideas to market faster so you become viable and figure out optimization later.

It's been a long time since that argument held. Almost every dynamic programming language still under active development is adding some form of gradual typing because the maintainability benefits alone are clearly recognized, though such languages still struggle to optimize well. Now there are several statically typed languages to choose from that get those maintainability benefits up-front and optimize very well.

Different languages can still be a better fit for different projects, e.g. Rust, Go, and Swift are all statically typed compiled languages better fit for different purposes, but in your analogy they're all jets designed for different tactical roles, none of them are "trains" of any speed.

Analogies about how different programming languages are like different vehicles or power tools or etc go way back and have their place, but they have to recognize that sometimes one design approach largely supersedes another for practical purposes. Maybe the analogy would be clearer comparing jets and trains which each have their place, to horse-drawn carriages which still exist but are virtually never chosen for their functional benefits.

I cut my teeth on C/C++, and I still develop the same stuff faster in Python, with which I have less overall experience by almost 18 years. Python is also much easier to learn than, say, Rust, or the current standard of C++ which is a veritable and intimidating behemoth.

In many domains, it doesn't really matter if the resulting program runs in 0.01 seconds or 0.1 seconds, because the dominant time cost will be in user input, DB connection etc. anyway. But it matters if you can crank out your basic model in a week vs. two.

> Python is also much easier to learn than, say, Rust

I don't doubt it, but learning is only the first step to using a technology for a series of projects over years or even decades, and that step doesn't last that long.

People report being able to pick up Rust in a few weeks and being very productive. I was one of them, if you already got over the hill that was C++ then it sounds like you would be too. The point is that you and your team stay that productive as the project gets larger, because you can all enforce invariants for yourselves rather than have to carry their cognitive load and make up the extra slack with more testing that would be redundant with types.

Outside of maybe a 3 month internship, when is it worthwhile to penalize years of software maintenance to save a few weeks of once-off up-front learning? And it's not like you save it completely, writing correct Python still takes some learning too, e.g. beginners easily get confused about when mutable data structures are silently being shared and thus modified when they don't expect it. People who are already very comfortable with Python forget this part of their own learning curve, just like people very comfortable with Rust forget their first borrow check header scratcher.

I never made a performance argument in this thread so I'm not sure why 0.01 or 0.1 seconds matters here. Even the software that got you into a commercial market has to be maintained once you get there. Ask Meta how they feel about the PHP they're stuck with, for example.

I tried searching for that article because I vaguely recall it, but can't find it either. But yeah, a lot of small improvements add up. Reminds me of this talk: https://www.youtube.com/watch?v=NZ5Lwzrdoe8

Here is a source for the SQLite case: https://topic.alibabacloud.com/a/sqlite-387-a-large-number-o...

That looks like blogspam to me, rather than an actual source.

2-9%