Cutting Down Rust Compile Times From 30 to 2 Minutes With One Thousand Crates (www.feldera.com)
from ruffsl@programming.dev to rust@programming.dev on 16 Apr 15:53
https://programming.dev/post/28717646

Instead of emitting one giant crate containing everything, we tweaked our SQL-to-Rust compiler to split the output into many smaller crates. Each one encapsulating just a portion of the logic, neatly depending on each other, with a single top-level main crate pulling them all in.

#rust

threaded - newest

gressen@lemm.ee on 16 Apr 16:25 next collapse

<img alt="" src="https://lemm.ee/pictrs/image/067d616b-47de-4c00-a609-48bcec8b320f.jpeg">

They changed the graphic already but holy AI on a stick! Now his arm is still behind the saw but his fingers are already gone. Lol

maikelthedev@programming.dev on 17 Apr 20:10 collapse

well at least the saw works right

BB_C@programming.dev on 16 Apr 17:00 next collapse

Cool and all. But missing some experiments:

  • cranelift
  • multi-threaded rustc
  • undoing type erasure after the split
  • lto = "off"
  • strip = false (for good measure)
  • [PRIORITY] a website that works with Tridactyl✋
thingsiplay@beehaw.org on 17 Apr 04:31 next collapse

[PRIORITY] a website that works with Tridactyl✋

A man of culture.

bodaciousFern@lemmy.dbzer0.com on 17 Apr 21:38 collapse

multi-threaded rustc

I am still quite ignorant of the workings of rust/rustc (I’ll learn it tomorrow, I swear!) but I’m surprised that multi threaded compilation isn’t available by default. make/gcc have had it for several decades

BB_C@programming.dev on 17 Apr 22:18 collapse

make uses multiple processes for parallelism, or what the blog post (below) calls “interprocess parallelism”. cargo/rustc has that and intraprocess parallelism for code generation (the backend) already. the plan is to have parallelism all the way starting from the frontend. This blog post explains it all:

blog.rust-lang.org/2023/11/09/parallel-rustc/

FizzyOrange@programming.dev on 16 Apr 21:11 next collapse

What’s the advantage of compiling to Rust here? Maybe it would be faster if they just skipped straight to LLVM.

thingsiplay@beehaw.org on 17 Apr 04:30 next collapse

Impressive improvement! But why did they choose Rust to compile on demand in the first place, if compile time was that important?

sga@lemmings.world on 19 Apr 20:16 collapse

i don’t really know how much could they optimise more, but they can predict it by fitting the number of cores with the amount of time taken, that is amdahl’s law.