Compiling templates to Go with go/ast

I'm trying to make GoSSR's Go-and-template component file useful to the Go toolchain. The data stays in Go with one set of types. Now the template needs to use those types in a way the compiler can check.

The page author asks for a paragraph for each city:

<p p-for="city in state.Cities" p-text="city.Name"></p>

There's a simple enough program underneath that template. Cities belongs to the page data, city is the loop variable and Name belongs to each item. I should be able to recognise those relationships in the output.

For this small example, the generated function could look like this:

func render(pageData PageData) string {
    var buf strings.Builder
    for _, city := range pageData.Cities {
        buf.WriteString("<p>")
        buf.WriteString(html.EscapeString(city.Name))
        buf.WriteString("</p>")
    }
    return buf.String()
}

This example writes text between paragraph tags. Attribute values, URLs and scripts need their own context-aware escaping. I can't extend the emitter to those contexts by changing the literal strings alone.

With page data containing London and Paris, the function returns:

<p>London</p><p>Paris</p>

The function takes the application's PageData type, whose Cities field holds the City values. Those are the real Go definitions the function compiles against. I don't need a second description of a city for this renderer. If I ask for pageData.Citys, Go has a field access it can reject. That's the complaint I want while building the page.

Why I'm constructing a tree

I could assemble this function with strings. The first version is easy to picture as a string, with the function opening followed by the collection name in a range statement, the body and the closing braces.

Once the body contains another loop or a call with several arguments, I have more pieces to arrange. I'd rather work with those pieces directly than keep track of what a growing format string is supposed to contain.

In Go's abstract syntax tree, or AST, each of those pieces has a name and fields. A range statement contains its collection, bindings and body, while a function call contains the function and its arguments. I can build them separately and then put them together.

This takes more code, but when I'm fixing a loop I can find one in the generator too and inspect and change it before printing the program as text. I'm using the tree to organise the program. An invalid identifier or a request for a missing field will sit in it quite happily. The Go tools still need to check the result.

And I can find the node fields in the go/ast documentation. I can also parse a tiny hand-written Go example and inspect its tree as a target for my generator.

For the range statement, I need _ and city as the binding names, pageData.Cities as the collection, and the statements which write the paragraph as the body:

loop := &ast.RangeStmt{
    Key:   ast.NewIdent("_"),
    Value: ast.NewIdent("city"),
    Tok:   token.DEFINE,
    X: &ast.SelectorExpr{
        X:   ast.NewIdent("pageData"),
        Sel: ast.NewIdent("Cities"),
    },
    Body: &ast.BlockStmt{List: body},
}

Here body is the list of statements writing the paragraph, which I'll build separately, and token.DEFINE selects := for the bindings. The collection is the pageData.Cities selector and the block wraps the body. This loop still belongs inside the enclosing function. I can change its collection without rebuilding the body, or add another statement to the body without working through the loop's spelling again. Those are ordinary edits to the generator, and the tree lets me make them separately. The template asks for a loop, so I construct a loop. Verbose constructors. I enjoy how literal it all is, the generated file can be ugly on the way there, but I can still pick out the loop, its collection and its body.

The expression city.Name needs only a selector:

name := &ast.SelectorExpr{
    X: ast.NewIdent("city"),
    Sel: ast.NewIdent("Name"),
}

Escaping that value adds a call around it:

escaped := &ast.CallExpr{
    Fun: &ast.SelectorExpr{
        X: ast.NewIdent("html"),
        Sel: ast.NewIdent("EscapeString"),
    },
    Args: []ast.Expr{name},
}

The call to buf.WriteString can then use escaped as its argument. Each call used as a statement gets an ast.ExprStmt around it before going into the loop's body.

Read the field, escape the text, then write the result. I can follow that nesting in the nodes, much as I can follow it in the Go I wanted at the start. Still a lot of constructors for a paragraph.

Quote the strings and check the program

Literal content looks easier. I could put quotes around the text:

literal := &ast.BasicLit{
    Kind:  token.STRING,
    Value: `"` + text + `"`,
}

That works for <p>. Give it <p title="City"> and the inner quotes end the Go string early. Now I'm writing a Go string escaper. Not quite the paragraph renderer I came here for. Then I'd have to think about backslashes and newlines and....

This is a good point for me to remember that Go has a standard library, and that I am allowed to use it. strconv.Quote already knows how to write a string as a Go literal:

literal := &ast.BasicLit{
    Kind: token.STRING,
    Value: strconv.Quote("<p title=\"City\">"),
}

Quoting makes a valid string in Go source while HTML escaping controls what that string writes into the page. Different jobs. A generated file can compile perfectly and still write unsafe HTML.

Both Cities and Citys are valid identifiers, and I can put either into an ast.Ident and get readable Go source. The actual page type is what lets the type checker distinguish them. Checking against another loose description of the data would leave me maintaining the very connection I built GoSSR to avoid. These expressions need the application's real types. A missing field can now give me an error in the generated Go, and that's a useful place to encounter it while working on the compiler. When I'm changing a page I want the error to lead me to the template expression I wrote.

Print it in Go style

Once the file has its declarations and imports, go/format can print the tree in normal Go style. The formatter takes care of presentation, but it doesn't know that the template intended Cities rather than Citys.

Find the expression in the template

Does it render the right page? Parsing and type checking catch syntax and field mistakes, while running a fixture checks what the renderer writes, including escaping and loops. A reviewed snapshot lets me inspect changes to the generated code. Each answers a different question. I can read the neat function and feel finished with it, even while those questions about its behaviour are still waiting for an answer.

Keeping the template location while constructing the nodes or recording a map alongside the generated file lets me take the Go error to the component. Otherwise, fixing a field name still means leaving the page and searching through the generator's output.

I'd deliberately misspell a field in a fixture, then read the error as the person editing the template. If I wrote city.Nmae, that's the expression to find and correct. Being sent back to that expression would let me fix the page without learning where my generator put its loops first.

For the cities list, the template becomes a range statement and real Go field accesses. Rename a field and the old access no longer compiles, which gives me the connection I was after in GoSSR.