banner

For a full list of BASHing data blog posts see the index page.  RSS

gron the JSON flattener

gron is a self-contained Go executable you can download from here on GitHub. In the UNIX tradition, gron does one thing well: it flattens JSON into a structure that's easily processed by shell tools, line by line.

Yes, there are other ways to flatten JSON and other ways to explore a JSON structure. jq can do those things and much more. But as gron's developer Tom Hudson writes,

jq is awesome, and a lot more powerful than gron, but with that power comes complexity. gron aims to make it easier to use the tools you already know, like grep and sed.

jq is indeed awesome, and I've turned to it for help when dealing every 6-12 months or so with a JSON file. But please note that phrase every 6-12 months. The rest of the time I don't see JSON, I forget how jq works, and I have to re-read the docs to understand the sometimes difficult jq syntax.

I used to think I was alone in feeling annoyed with the often complicated structure of JSON files, but maybe not so:

We're all data people here, so you already know the scenario: it happens perhaps once a day, perhaps 5, or even more. There's an API you're working with, and it's great. It contains all the information you're looking for, but there's just one problem: the complexity of nested JSON objects is endless, and suddenly the job you love needs to be put on hold to painstakingly retrieve the data you actually want, and it's 5 levels deep in a nested JSON hell. Nobody feels like much of a "scientist" or an "engineer" when half their day becomes dealing with key value errors.
 
...from the Hackers and Slackers blog, here

Anyway, back to gron, which does a wonderful job of taming JSON for the shell. Here's a minified JSON ("demo.json") from this Stack Overflow query:

{"created":"2020-05-12T15:10:37Z","device":{"device_info":{"device_fw":204,"device_sn":"06-02133","device_trait":2,"device_type":190},"timeseries":[{"configuration":{"sensors":[{"measurements":["BATTERY","BATTERY_MV"],"port":7,"sensor_bonus_value":"Unavailable","sensor_firmware_ver":"Unavailable","sensor_number":133,"sensor_sn":"Unavailable"},{"measurements":["REFERENCE_KPA","TEMPC_LOGGER"],"port":8,"sensor_bonus_value":"Unavailable","sensor_firmware_ver":"Unavailable","sensor_number":134,"sensor_sn":"Unavailable"}],"valid_since":"2018-08-11T21:45:00Z","values":[[1534023900,0,19,[{"description":"Battery Percent","error":false,"units":"%","value":100},{"description":"Battery Voltage","error":false,"units":" mV","value":7864}],[{"description":"Reference Pressure","error":false,"units":" kPa","value":100.62},{"description":"Logger Temperature","error":false,"units":" °C","value":28.34}]]]}}]}}

Below, gron flattens it. Notice how the nesting is neatly organised and colorised:

gron demo.json

gron1

Here I grep for the "Battery Voltage" subsection:

gron demo.json | grep -n -A3 "Battery Voltage"

gron2

and do a line-addressed substitution with sed of 7870 millivolts for 7864. I'll then "ungron" the result, to pretty-print the modified JSON instead of saving it to a new file:

gron demo.json | sed '47s/7864/7870/' | gron --ungron

gron3

I've so far used gron to track down database-to-JSON errors as well as to simply understand a JSON file's structure. There are more interesting ways to apply this great utility — check this page on the gron GitHub site.

"ungronning" can also be done with the short option "-u", and developer Tom Hudson suggests aliasing gron --ungron as "norg". As for the name "gron", Hudson says it's grep + json.

Last update: 2022-03-23
The blog posts on this website are licensed under a
Creative Commons Attribution-NonCommercial 4.0 International License