When I first learned about jq, I thought it was a handy tool for extracting from JSON. Little did I know that I was only scratching the surface of what jq can do.
jq is a “lightweight and flexible command-line JSON processor”. When I first heard of the tool, I mainly saw it being used for extracting relevant data from large JSON objects. If I had a sequence of large JSON objects, I could pipe them through a small jq program to extract the key(s) that I wanted.
$ jq '.first.inside_first.thing' < my_big_object.json
What a relief!
No more trying to massage the json into a form that could be mined with sed or awk!
No more writing and debugging Python scripts to transform JSON!
No more copying and pasting into my browser’s dev console!
If your interest is piqued, I highly recommend going and watching Charles Chamberlain’s delightful !!Con 2017 talk about jq. It’s short (9 minutes 32 seconds), engaging, and shows glimpses at the true power of jq. If your interest is not piqued, I recommend watching the video anyway. Maybe it’ll pique your interest.
I happily added jq to my toolbox, and used it for all my JSON parsing and extracting needs.
Fast forward to earlier this year. I started writing a Pony port of the Mustache templating language. What I thought was kinda cool is that the test suite for mustache implementations is written in YAML and JSON.
In retrospect, this approach makes sense to me. If you have a specification of some pure function, and you want to make it easier for implementers to write implementations of your function, you could do worse than to use an extremely common data representation format to represent your tests. With specification tests in JSON, implementers might have an easier time testing their implementations as long as their language can parse JSON. I know I had an easier time because of this.
I found myself with a bunch of JSON objects representing tests. What I wanted were a bunch of Pony classes representing tests. Thinking about my problem, I remembered a tweet by Kate, describing how she used jq to generate C code.
In Pony, lists of tests are collected by actors (or possibly classes)
that implement the TestList interface.
A TestList has a function tests() that accepts a PonyTest object,
and calls the PonyTest.apply() behavior with the name of each test class in the list.
With these incoming messages, the PonyTest actor knows what tests to run.
Here’s an example.
TestLists can also be comprised of other TestLists.
Mustache’s tests are categorized,
and so I adopted the same categorization for
my tests.
Some of these lists of tests are quite long, and I did not want to write and maintain all these tests without any assistance. jq to the rescue!
jq programs are essentially composed of filters that operate on some input JSON. I have some preamble to add to my Pony test files, so I start with a string literal, which can be thought of as a filter that ignores its input and produces the desired string. I then use the comma operator, which concatenates the results of two filters together, and end with a filter that extracts and transforms the names of each test into an appropriate name for a private Pony class.
#!/usr/bin/jq -Mfr
# meant to be used with comments.json from
# https://github.com/mustache/spec
def testClass(n): "_Test" + (n | gsub("[() -]"; ""));
"use \"ponytest\"
use \"json\"
use \"../..\"
actor Main is TestList
new create(env: Env) => PonyTest(env, this)
new make() => None
fun tag tests(test: PonyTest) =>",
(.tests[].name | (" test(" + testClass(.) + ")")),The only remaining thing to do is to generate the code for each class. We have a function for coming up with the name of a Pony test class. We have the templates, variable bindings, and expected results from the JSON test cases. I injected JSON string literals into the test files, and used Pony’s JSON package to parse those literals into Pony data.
(.tests[] | ("
class iso " + testClass(.name) + " is UnitTest
fun name(): String => \"interpolation/" + .name + "\"
fun apply(h: TestHelper) ? =>
let template = " + (.template | @json) + "
let expected = " + (.expected | @json) + "
let data = (recover val
JsonDoc.>parse(\"\"\"" + (.data | tojson) + "\"\"\")?
end).data
let m = Mustache(template)?
h.log(m.print_tokens())
h.assert_eq[String](expected, m.render(data))"))
Lastly, to generate a Pony file for the tests in interpretation.json,
I pass interpretation.json through my jq script.
$ ./gen.jq < /path/to/interpretation.json > interpretation/_test.ponyRepeating these steps for the remaining Mustache test suites left me with Pony files comprised of Pony classes for each of the tests. After using these tests to debug my implementation of Mustache in Pony, my next task is to improve the API to make it easier for everyday use.
Hopefully you’ve learned that jq is useful for more than shell pipelines and one-liners! If you’d like to learn more, I strongly recommend reading through jq’s manpage. It’s dense, but comprehensive and precise. Thanks for reading!