I created a custom parser recently. I built it in c# and ANTLR4. You need to define grammar and then code the AST which is an object model to handle logic. Not as difficult as it sounds, and you can use ChatGPT or something to help with parts of it.

I’ve never made one, but I really like Crafting Interpreters. It’s a really great intro and helped me understand bytecode.

There are two main directions here.

1. For General Purpose Language you need to define the language lexical idioms (control flow, data/types, state, dependency, etc.): basically how you express Turing Machine. Then you need to write a compiler that takes the AST and transforms it to LLVM IR (if we want to stay realistic). I can’t recommend any particular book regarding this, I read it in a classic Ullman’s book but it might be outdated. Take any book that has compilers in the title and a dragon on cover.

2. For Domain Specific Language, you need to parse the AST into a graph of execution and then walk and execute it. You don’t really need to design it first, as the language will have a limited application. You can even use Parse Tree both as AST and as IR, applying various optimisations in place. Here you take M. Fowler’s book on Domain Specific Languages and just read.

I’ve never touched the GP languages apart from some conference talks and uni, but I am keen on creating a DSL for every company I work in. That’s a good practice when things become very nasty and you just need something simple to express very complex logic.

A nice tool to start with before delving in more proper “language” coding, I would say, is LARK, it’s a pretty easy parser that is also relatively versatile and powerful, I doubt that it would be fitting to make an actual programming language but it could help you to the next steps.