It's been six months since I open-sourced Piko, my Go application framework. I made the first public commit at 04:09 on 2 April. Eight minutes later came Initial Dump with about 19,000 files. A modest opening paragraph.
The interpreter arrived with the templates, querying tools and shared services. I'd been building them around client work, and now somebody else could look inside.
I tend to have several projects on the go and I save progress constantly. Fallout and the other Bethesda games have done permanent damage to my saving habits, one glitch can ruin a save file and apparently I now treat changing a Go file with the same suspicion.
So I commit everything as v, then squash it down. Or don't.
Now I'm moving Piko's Go interpreter into a separate repository. The commit messages say what changed (so far). I may even be able to find the fix again without opening several versions of v.
I've been calling the interpreter Pipit since May. I've been working on making it independent of Piko for about four months. Giving it a name was considerably quicker. Inside Piko, the interpreter has become a Frankenstein of fixes and performance work, and I can explain individual additions without being happy about the structure they produced.
I want clear boundaries to tell me where a change belongs, both inside the implementation and around the public API. Moving files won't do that. The new repository gives me the occasion.
Most of Pipit's implementation stays under internal while the root pipit.sh/pipit package provides the embedding API, so I can change a compiler helper without asking every application to change with it. The architecture guide describes those boundaries.
Piko's page builder handed the interpreter a filesystem:
freshInterpreter.SetSourcecodeFilesystem(vfsAdapter)
The bytecode interpreter's adapter accepted it like this:
func (*interpreterAdapter) SetSourcecodeFilesystem(any) {}
Import resolution had moved elsewhere, but this interface remained from Piko's page-builder integration, so I'm still handing the interpreter a filesystem it does nothing with.
I could move the directory and call the extraction done, until another application tries to work out what to pass to this empty method. That application has no use for leftover instructions from Piko's old setup. I have to go through the API and decide which calls still do useful work.
The assembly generator, asmgen already lives in a separate repository, and I'm planning to pull Piko's querier out into Seagull and extract other parts over time. The hard part is deconstructing Piko so other people can use the work while keeping it easy for me to maintain. I still need to be able to change the pieces and keep them working together. It may be a while before I get through such a big job.
What the engine is allowed to know
Public calls are one boundary. Imports need rules too.
In Pipit's linter configuration, engine-below-everything keeps compiler and application imports out of the engine. It also forbids the codec and adapters, along with the verifier and debugger. The isa-is-a-leaf rule keeps instruction definitions below the packages that use them.
If I reach into the compiler from an instruction handler, the import breaks the declared boundary, which is useful help when the tempting shortcut is three lines and the architectural explanation is three months old. I know which one is easier to remember.
In Piko, compilation, execution, inlining and instruction machinery lived together in interp_domain. Sharing a package made it convenient to reach for another internal detail. It also made that convenience hard to distinguish from a necessary dependency.
Pipit separates the compiler from the engine through a program representation that carries the instructions and constants and descriptions needed to execute a program. The compiler produces it. The engine consumes it without importing the compiler.
The host chooses the available world
A bare interpreter can evaluate an expression without registering any host packages:
bare := pipit.NewInterpreter()
value, err := bare.Eval(ctx, `1 + 2`)
// value: 3, err: nil
Ask it to import strings and it refuses. The diagnostic explains that the Go standard library is opt-in and names the option that registers it.
For a page, I'm thinking of string operations and maybe....
os.Exit makes me stop there. Exposing that native function lets a script end the server process as well as its own evaluation. That's quite a lot to enable when I only needed strings.ToUpper.
The optional pipit.sh/pipit/sdk/stdlib provider adds the standard library:
withStd := pipit.NewInterpreter(stdlib.WithStandardLibrary())
value, err := withStd.Eval(ctx, `import "strings"
strings.ToUpper("hello")`)
// value: "HELLO", err: nil
An application can also register selected host functions under an import path so a script can ask the host to perform specific operations without receiving the whole standard library.
Piko can register the packages its pages need, and another application can supply a smaller or different set without adopting Piko's symbol table. That's the choice I want extraction to give the host.
I could make a package unavailable to scripts with a runtime option and call it optional, but that wouldn't necessarily make the host binary smaller. Optional for the script, then. I want it to be optional for the application too.
A symbol-table entry refers to the registered native function and makes its code reachable to the linker even if today's script never calls it. By the time a runtime option is checked, those bytes are already in the executable.
Separating the provider lets an application leave those references out of its build. The core keeps the standard-library package names for the diagnostic without linking their implementations, so I can explain what's missing without first including all of it. Symbol extraction and self-host tooling have separate modules too, so importing the interpreter doesn't bring those jobs along.
Those providers expose Go functions linked into the host. Pipit can also load Go packages from a prepared bytecode bundle, which lets me prepare the package once and choose exactly which code the service loads. That's a separate route from registering native symbols or resolving a script's imports from source. A pin is a content fingerprint identifying a particular bundle. The host checks it before unpacking, then verifies the decoded program before installation runs the package initialisers inside the service. By the time the page calls a function, that code has already had a turn.
The pin tells me these are the contents I selected, and it will faithfully preserve their bugs too.
A web request and an interactive console want different things from an interpreter. The request needs fresh execution state. The console wants later submissions to remember earlier declarations.
Pipit's API exposes both choices. Clone creates fresh execution state but shares the configured symbol registry. That table holds names and types as well as the functions and values available to scripts. Configure the base first, then create a clone for each concurrent worker:
base := pipit.NewInterpreter(stdlib.WithStandardLibrary())
worker := base.Clone()
The embedding contract forbids concurrent evaluation of the same interpreter or session. Separate clones give the engine's mutable work separate owners. Registered host functions can still close over shared Go objects.
For example, let a host function called Next increment a captured Go counter starting at zero. One clone calls it and receives 1. Another clone calls it and receives 2. Each clone has fresh interpreter state, but both registry entries lead to the same native function and counter.
Wait, two? The first call used a different clone.
I've cloned the interpreter, not the application's world, and that second reply makes the boundary much easier for me to see. Shared host objects still need the application's usual concurrency controls.
For interactive continuity, a session retains declarations between submissions:
session := base.NewSession()
_, err := session.Submit(ctx, `count := 10`)
// Handle err before continuing.
_, err = session.Submit(ctx, `count += 1`)
// Handle err before continuing.
value, err := session.Submit(ctx, `count`)
// value: 11, err: nil
With each error handled, the final submission returns 11. Only the session keeps this declaration history. Separate ordinary Eval calls go without it.
Piko still needs to register its providers and connect its development loop to the new API. Another host can choose which packages and host functions to offer and which state to keep. I'd like its first setup to involve calls that do something, rather than an empty filesystem method left over from Piko's page builder.