Technology

An Introduction to JQ



18 minute be taught

    Up up to now:

Adam Gordon Bell %

   
Adam Gordon Bell

Background: Fingers, Head, and Google

Every time I attain a stopping point in my work, I utilize a bash alias called gwip1 to invent a ‘work in progress’ commit. It happens without acutely conscious pondering on my fragment. The identical means my fingers know the vim key bindings, they know gwip.

Utterly different actions, I do know the method they work, however I even must take into fable them every time. They are in my head, no longer my fingers.2

Nonetheless, some issues below no conditions stick in my head, nor my fingers, and I even must google them every time. jq is one amongst those.

I are mindful about it’s a extremely effective instrument, however I always discontinue up back at Google and then copying and pasting a solution from someplace. So I resolve my self-discipline however below no conditions be taught the instrument.

It’s time to fix that. On this text, I’m going to head over the fundamentals building blocks of jq in ample depth that you just need to well be ready to brand how jq works. Truly, you peaceful would perchance well infrequently want to head to google to search out a feature name or test your syntax, however a minimal of you’ll maintain a agency grounding in the fundamentals.

What Is JQ

jq is a light-weight, expose-line JSON processor. I install it with brew (brew install jq), however it indubitably’s a single portable executable, so it’s easy to install on Linux, Residence windows, or macOS. To place it to use, you own quite a lot of filters, and it applies those filters to a JSON doc.

Primarily the most appealing filter is the identity filter which returns all its input (.):

$ echo '{"key1":{"key2":"value1"}}' | jq '.'
{
  "key1":  {
    "key2":  "value1"
  }
}

This filter is useful for honest fairly-printing a JSON doc.3 I’m going to ignore the fairly-printing and bounce proper into the utilize of jq to remodel JSON documents.

Using JQ to Take dangle of out Aspects

I’m going to utilize jq to filter the information returned by the GitHub repository API. The information I get back by default appears to be like to be like cherish this:

$ curl https://api.github.com/repos/stedolan/jq
{
  "identification":  5101141,
  "node_id":  "MDEwOlJlcG9zaXRvcnk1MTAxMTQx",
  "name":  "jq",
  "full_name":  "stedolan/jq",
  "internal most":  fraudulent,
  "proprietor":  {
    "login":  "stedolan",
    "identification":  79765
  },
  "html_url":  "https://github.com/stedolan/jq",
  "description":  "Expose-line JSON processor",
  "stargazers_count":  19967,
  "watchers_count":  19967,
  "language":  "C",
  "license":  {
    "key":  "a quantity of",
    "name":  "Utterly different",
    "spdx_id":  "NOASSERTION",
    "url":  null,
    "node_id":  "MDc6TGljZW5zZTA="
  }
}

jq lets us deal with the JSON doc as an object and settle out system within of it.

Here is how I filter the JSON doc to make a various the payment of the name key:

$ curl https://api.github.com/repos/stedolan/jq | jq ' .name' 
"jq"

Similarly, for picking the payment of the proprietor key:

$ curl https://api.github.com/repos/stedolan/jq | jq ' .proprietor' 
{
    "login":  "stedolan",
    "identification":  79765
}

That you must well be ready to drill in as far as you desire cherish this:

$ curl https://api.github.com/repos/stedolan/jq | jq ' .proprietor.login' 
"stedolan"

What I Learned: Object Identifier-Index

jq enables you to make a various system in a JSON doc cherish it’s a JavaScript object. Good open with . ( to your total doc) and drill down to the payment you desire. It finally ends up making an strive something cherish this:

jq '.key.subkey.subsubkey'

Using JQ to Take dangle of out Arrays

For folks that curl the GitHub Disorders API, you will get back an array of concerns:

$ curl https://api.github.com/repos/stedolan/jq/concerns?per_page=5    
[
  {
    "id": 966024429,
    "number": 2341,
    "title": "Question about license.",
    "body": "I would like to create a [winget](https://github.com/microsoft/winget-cli) package for jq. 🙏🏻"
  },
  {
  
    "identification":  962477084,
    "quantity":  2340,
    "title":  "visibility of wiki pages",
    "body":  "The visibility of wiki pages to head making an strive engines will doubtless be dinky; to illustrate, the search result for "jq Cookbook" appears to be like to be like cherish this:"
  },
  {
   
    "identification":  955350543,
    "quantity":  2337,
    "title":  "Unlock 1.6 does no longer maintain pre-autoreconf'ed configure script",
    "body":  "For folks that've gotten a usage ask, please build a query to us on both Stack Overflow (https://stackoverflow.com/questions/tagged/jq) or in the #jq channel (http://irc.lc/freenode/%23jq/) on Freenode (https://webchat.freenode.discover/)."
  },
  {
    "identification":  954792209,
    "quantity":  2336,
    "title":  "Repair typo",
    "body":  ""
  },
  {
    "identification":  940627936,
    "quantity":  2334,
    "title":  "Bring collectively error messages don't provide column most appealing line quantity",
    "body":  "Bring collectively errors in filter expressions don't encompass the column quantity the set up the parser approximately or precisely locates the error. Most filter expressions are one-liners (are quite a lot of lines even supported?), so the information that the error is on line 1 is no longer beneficial."
  }
]

To salvage a particular facet in the array, give jq an index:

$ curl https://api.github.com/repos/stedolan/jq/concerns?per_page=5 | jq '.[4]' 
 {
    "identification":  940627936,
    "quantity":  2334,
    "title":  "Bring collectively error messages don't provide column most appealing line quantity",
    "body":  "Bring collectively errors in filter expressions don't encompass the column quantity the set up the parser approximately or precisely locates the error. Most filter expressions are one-liners (are quite a lot of lines even supported?), so the information that the error is on line 1 is no longer beneficial."
  }

Aspect Uncover: Array Indexing in jq

Array indexing has some beneficial convenience syntax.

That you must well be ready to settle out ranges:

$ echo "[1,2,3,4,5]" | jq '.[2:4]'

That you must well be ready to settle out one sided ranges:

$ echo "[1,2,3,4,5]" | jq '.[2:]'

Also, you need to well perchance also utilize negatives to make a various from the tip:

$ echo "[1,2,3,4,5]" | jq '.[-2:]'

That you must well be ready to utilize the array index with the item index:

$ curl https://api.github.com/repos/stedolan/jq/concerns?per_page=5 | jq '.[4].title' 
"Bring collectively error messages don't provide column most appealing line quantity"

And that you just might also utilize [] to salvage the full system in the array. As an illustration, here is how I would salvage the titles of the failings returned by my API request:

$ curl https://api.github.com/repos/stedolan/jq/concerns?per_page=5 | jq '.[].title' 
"Query about license."
"visibility of wiki pages"
"Unlock 1.6 does no longer maintain pre-autoreconf'ed configure script"
"Repair typo"
"Bring collectively error messages don't provide column most appealing line quantity"

What I Learned: Array-Index

jq enables you to make a various your total array [], a particular facet [3], or ranges [2:5] and combine these with the item index if wanted.

It finally ends up making an strive something cherish this:

Aspect Uncover: Laying aside Quotes From JQ Output

The -r possibility in jq offers you raw strings in present for you that.

$ echo '["1","2","3"]' | jq -r '.[]'
1
2
3

The -j possibility (for join) can combine collectively your output.

$ echo '["1","2","3"]' | jq -j '.[]'
123

Putting Aspects in an Array the utilize of jq

Whereas you open the utilize of the array index to make a various system, you’ve got gotten a brand modern self-discipline. The information returned obtained’t be a sound JSON doc. Within the instance above, the topic titles were modern line delimited:

"Query about license."
"visibility of wiki pages"
"Unlock 1.6 does no longer maintain pre-autoreconf'ed configure script"
"Repair typo"
"Bring collectively error messages don't provide column most appealing line quantity"
...

In actuality, every time you set up a query to jq to return an unwrapped assortment of system, it prints them every on a brand modern line. That you must well be ready to note this by explicitly asking jq to ignore its input and as an various return two numbers:

$ echo '""' | jq '1,2' 
1
2

That you must well be ready to resolve this the identical signifies that you just need to turn the textual verbalize 1,2 into an array in JavaScript: By wrapping it in an array constructor [ ... ].

Similarly, to set up a generated assortment of results true into a JSON array, you wrap it in an array constructor [ ... ].

My GitHub self-discipline title filter (.[].title) then turns into [ .[].title ] cherish this:

$ curl https://api.github.com/repos/stedolan/jq/concerns?per_page=5 | 
  jq '[ .[].title ] ' 
[
  "Question about license.",
  "visibility of wiki pages",
  "Release 1.6 does not have pre-autoreconf'ed configure script",
  "Fix typo",
  "Compile error messages don't provide column only line number"
]

Now I in fact maintain a sound JSON doc.

What I Learned: Array Constructors

In case your jq ask returns higher than one facet, they’ll be returned newline delimited.

$ echo '[{"a":"b"},{"a":"c"}]' | jq -r '.[].a'
  "b"
  "c"

To turn these values true into a JSON array, what you conclude is resembling growing an array in JavaScript: You wrap the values in an array constructor ([...]).

It finally ends up making an strive something cherish this:

$ echo '[{"a":"b"},{"a":"c"}]' | jq -r '[ .[].a ]'

Using jq to Take dangle of out Extra than one Fields

The GitHub concerns API has a quantity of critical aspects I don’t care about. I are making an strive to make a various quite a lot of fields from the returned JSON doc and waddle away the comfort on the back of.

Primarily the most appealing means to conclude that is the utilize of , to specify quite a lot of filters:

$ curl https://api.github.com/repos/stedolan/jq/concerns?per_page=2 |  
  jq ' .[].title, .[].quantity'
"Query about license."
"visibility of wiki pages"
2341
2340

Nonetheless that is returning the outcomes of 1 preference after the a quantity of. To alternate the ordering, I will facet out the array selector:

$ curl https://api.github.com/repos/stedolan/jq/concerns?per_page=2 | 
  jq '.[] |  .title, .quantity'
"Query about license."
2341
"visibility of wiki pages"
2340

This refactoring uses a pipe (|), which I’ll discuss shortly, and runs my object selectors (.title and .quantity) on every array facet.

For folks that wrap the ask in the array constructor you salvage this:

$ curl https://api.github.com/repos/stedolan/jq/concerns?per_page=2 | 
  jq '[ .[] |  .title, .number ]'
[
  "Question about license.",
  2341,
  "visibility of wiki pages",
  2340
]

Nonetheless this peaceful isn’t the JSON doc I favor. To salvage these values into an ethical JSON object, I favor an object constructor { ... }.

Putting Aspects Into an Object Using jq

Let’s search for at some simple examples forward of exhibiting how my GitHub ask can utilize an object constructor.

A Limited Instance

I even maintain an array that contains my name (["Adam","Gordon","Bell"]), and I are making an strive to expose it true into a JSON object cherish this:

{
  "first_name": "Adam",
  "last_name": "Bell"
}

I will settle out the system I favor the utilize of array indexing cherish this:

$ echo '["Adam","Gordon","Bell"]' | jq -r '.[0], .[2]'
Adam
Bell

To wrap those values into the form I favor, I will replace the values with the array indexes that return them:

{
  "first_name": .[0],
  "last_name": .[2]
}

Or on a single line cherish this:

$ echo '["Adam","Gordon","Bell"]' | jq -r '{ "first_name":.[0], "last_name": .[2]}'
{
  "first_name":  "Adam",
  "last_name":  "Bell"
}

This syntax is the identical syntax for growing an object in a JSON doc. Primarily the most appealing distinction is you need to well perchance also utilize the item and array queries you’ve constructed up as the values.

Again to GitHub

Returning to my GitHub API self-discipline, to wrap the amount and the title up into an array I utilize the item constructor cherish this:

$ curl https://api.github.com/repos/stedolan/jq/concerns?per_page=2 | 
  jq '[ .[] | { title: .title, number: .number} ]'
[
  {
    "title": "Question about license.",
    "number": 2341
  },
  {
    "title": "visibility of wiki pages",
    "number": 2340
  }
]

What I Learned: Object Constructors

To position the system you’ve chosen back true into a JSON doc, you need to well perchance also wrap them in an object constructor { ... }.

For folks that were assemble up a JSON object out of quite a lot of selectors, it will discontinue up making an strive something cherish this:

jq '{ "key1": <>, "key2": <> }'

Which is the identical syntax for an object in a JSON doc, other than with jq you need to well perchance also utilize filters as values.4

Sorting and Counting With JQ

The next self-discipline I even maintain is that I are making an strive to summarize some this JSON data. Every self-discipline returned by GitHub has a assortment of labels:

$ curl https://api.github.com/repos/stedolan/jq/concerns/2289 |    
  jq ' { title: .title, quantity: .quantity, labels: .labels} '
  {
    "title":  "Bump jinja2 from 2.10 to 2.11.3 in /doctors",
    "quantity":  2289,
    "labels":  [
      "feature request",
      "dependencies"
    ]
  }

jq Constructed-in Functions

If I desire those labels in alphabetical present I will utilize the in-constructed model feature. It in fact works cherish this:

$  echo '["3","2","1"]' | jq 'model'
["1", "2", "3"]

This is resembling how I would model an array in JavaScript:

const l = ["3","2","1"];
l.model();

Utterly different constructed-ins that reflect JavaScript performance are on hand, cherish length, reverse, and tostring and so that they’ll all be venerable in a identical means:

$  echo '["3","2","1"]' | jq 'reverse'
["1", "2", "3"]
$  echo '["3","2","1"]' | jq 'length'
3

If I will combine these constructed-ins with the selectors I’ve constructed as much as this point, I’ll maintain solved my label sorting self-discipline. So I’ll repeat that subsequent.

What I Learned: jq constructed-ins

jq has many constructed-in capabilities. There are potentially too many to attach in tips however the constructed-ins have a tendency to mirror JavaScript capabilities, so give those a strive forward of heading to jq handbook , and also that you just need to salvage fortunate.5

Pipes and Filters

Sooner than I will utilize model to model the labels from my GitHub API request, I favor to negate how pipes and filters work in jq.

jq is a filter in the UNIX expose line sense. You pipe (|) a JSON doc to it, and it filters it and outputs it to identical outdated out. I would perchance well without problems utilize this characteristic to chain collectively jq invocations cherish this:

echo '{"title":"JQ Take dangle of out"}' | jq '.title' | jq 'length'
9

This is a wordy, even though simple, means to search out out the length of a string in a JSON doc. That you must well be ready to utilize this identical belief to mix varied jq constructed-in capabilities with the aspects I’ve proven up to now. Nonetheless there is a more effective means, even though. That you must well be ready to utilize pipes within of jq and conceptually they work honest cherish shell pipes:

echo '{"title":"JQ Take dangle of out"}' | jq '.title | length'
9

Listed below are some more examples:

  • .title | length will return the length of the title
  • .quantity | tostring will return the topic quantity as a string
  • .[] | .key will return the values of key key in the array (that is equivalent to this .[].key)

This means that sorting my labels array is straightforward. I will honest alternate .labels to .labels | model:

$ curl https://api.github.com/repos/stedolan/jq/concerns/2289 | 
  jq ' { title: .title, quantity: .quantity, labels: .labels | model } '    
  {
    "title":  "Bump jinja2 from 2.10 to 2.11.3 in /doctors",
    "quantity":  2289,
    "labels":  [
      "dependencies",
      "feature request"
    ]
  }

And must you desire honest a label depend that’s simple as effectively:

$ curl https://api.github.com/repos/stedolan/jq/concerns/2289 | 
  jq ' { title: .title, quantity: .quantity, labels: .labels | length } '    
  {
    "title":  "Bump jinja2 from 2.10 to 2.11.3 in /doctors",
    "quantity":  2289,
    "labels":  2
  }

What I Learned: Pipes and Filters

All the pieces in jq is a filter that you just need to well perchance also combine with pipes (|). This mimics the habits of a UNIX shell.

That you must well be ready to utilize the pipes and the jq constructed-ins to effect sophisticated transformations from simple operations.

It finally ends up making an strive something cherish this:

jq ' .key1.subkey2[] | model ' # sorting
jq ' .key2.subkey | length' # length of string or array
jq ' .key3 | flooring | tostring | length' # etc

Maps and Selects Using JQ

The flaws list I used to be making an strive at has many low-quality concerns in it.6 Let’s say I are making an strive to discover the full objects that are labeled. This could let me skip the full force-by fix-my-self-discipline concerns.

Unfortunately, it’s impossible to conclude this with the GitHub API unless you specify the full doable labels to your ask. Nonetheless, I will without problems conclude this ask on the expose line by filtering our results with jq. Nonetheless, to conclude so, I’m going to desire a pair more jq capabilities.

My ask up to now appears to be like to be like cherish this:

  jq '[ .[] | { title: .title, number: .number, labels: .labels | length } ]'

The important thing thing I will conclude is simplify it the utilize of arrangement.

  jq 'arrangement({ title: .title, quantity: .quantity, labels: .labels | length }) 

arrangement(...) let’s you unwrap an array, note a filter and then rewrap the outcomes back into an array. That you must well be ready to judge of it as a shorthand for [ .[] | ... ] and it comes up somewhat a lot in my ride, so it’s rate it committing to memory.

I will combine that with a settle out commentary that appears to be like to be like cherish this:

settle out is a constructed-in feature that takes a boolean expression and most appealing returns system that match. It’s resembling the WHERE clause in a SQL commentary or array filter in JavaScript.

Like arrangement, I decide up settle out comes up somewhat a lot, so whereas that you just need to must intention back to this text or google it the essential few times you want it, with good fortune, this will open to stick with your memory after that.

Putting this all collectively appears to be like to be like cherish this:

curl https://api.github.com/repos/stedolan/jq/concerns?per_page=100 | 
   jq 'arrangement({ title: .title, quantity: .quantity, labels: .labels | length }) | 
   arrangement(settle out(.labels > 0))'
[
  {
    "title": "Bump lxml from 4.3.1 to 4.6.3 in /docs",
    "number": 2295,
    "labels": 1
  },
  {
    "title": "Bump pyyaml from 3.13 to 5.4 in /docs",
    "number": 2291,
    "labels": 1
  },
  {
    "title": "Bump jinja2 from 2.10 to 2.11.3 in /docs",
    "number": 2289,
    "labels": 1
  },
  {
    "title": "Debugging help through showing pipeline intermediates. ",
    "number": 2206,
    "labels": 1
  }
]

This uses three object indexes, two maps, two pipes, a length feature, and a settle out predicate. Nonetheless must you’ve followed alongside, this could well well peaceful all make sense. It’s all honest composing collectively filters till you salvage the result you want.

Now lets discuss the means you need to well perchance also build this data into note.

In Review

What I Learned

Here’s what I’ve learned up to now:

jq enables you to make a various system by starting with a . and gaining access to keys and arrays cherish it’s a JavaScript Object (which is it’s far). This characteristic uses the Object and Array index jq creates of a JSON doc and search for cherish this:

jq '.key[0].subkey[2:3].subsubkey'

jq capabilities can maintain object constructors { ... } and array constructors [ ... ]. You utilize these whereas you need to well perchance also very effectively be making an strive to wrap back up something you’ve pulled out of a JSON doc the utilize of the above indexes:

jq '[ { key1: .key1, key2: .key2 }  ]'

jq contains constructed-in capabilities (length,model,settle out,arrangement) and pipes (|), and also you need to well perchance also model these collectively honest such as you need to well perchance also combine pipes and filters on the expose line:

  jq 'arrangement({ present-of-magitude: .objects | length | tostring | length }) 

Next Steps for Mastering jq

Reading about (or writing about) a instrument is no longer ample to grasp it. Movement is wanted. Here is my route of for cementing this data:

1. Total the jq Tutorial

jq-tutorial is no longer an tutorial at all, however a assortment of round 20 interactive workout routines that test you data of jq. I’ve came upon it extremely beneficial.

2. Try Using Your Memory First

Every time I favor to extract data or remodel a JSON doc, I strive and conclude it first without making an strive anything else up. If memory fails me, veritably jqterm, which has auto-completion, is useful. Recurrently, I peaceful want to head making an strive for something up, however science has proven that repeated retrieval yields retention. So over time, my retention would perchance well peaceful give a eradicate to.

3. Spend It

For folks that don’t utilize a instrument, you need to well below no conditions grasp it. So as soon as I in fact maintain a task that will be solved the utilize of jq, then it’s far what I utilize. As a minimum for the subsequent shrimp whereas, despite the incontrovertible fact that there is a more effective means to conclude it. Whether it’s exploring a REST API or making an strive at docker glimpse results, JSON is in every single set up, so opportunities abound.

4. Study Extra

Lastly, to deepen my data, I’m learning about recursive descent, declaring variables, and defining capabilities and developed aspects came upon in the handbook. Truly, these objects no longer often ever intention up, however after writing all this, I’m twisted on this instrument.

Doing all this isn’t crucial, however must you note me in a majority of those steps, I judge the utilize of jq will become 2nd nature.

Conclusion

So far I’ve most appealing covered the fundamentals of jq. The jq ask language is a fleshy programming language and also you need to well perchance also conclude hundreds thrilling issues with it. You convert from JSON to CSV. That you must well be ready to stipulate you have capabilities and even decide up primes with jq:

# Denoting the input by $n, which is belief to be a definite integer,
# eratosthenes/0 produces an array of primes decrease than or equal to $n: 
def eratosthenes: 
  (. + 1) as $n
  | (($n|sqrt) / 2) as $s
  | [null, null, range(2; $n)]
  | reduce (2, 1 + (2 * differ(1; $s))) as $i (.; erase($i))
  | arrangement(settle out(.));

Nonetheless, simple stuff – cherish selecting system, filtering by key, or payment – is often all you want.

I am hoping this helps make jq more approachable and that you just now no longer must waddle to google every time you need to well perchance also very effectively be making an strive to ask a JSON doc7.

Whereas you’re here:

Earthly is a syntax for defining your effect. It in fact works with your existing effect system. Rep repeatable and understandable builds on the modern time.

Feedback?

I’d cherish to hear your feedback, or must you’ve got gotten any tips of your have. Let me know on twitter:

jq is huge for fairly-printing JSON however it indubitably does so grand more.

I below no conditions in fact mastered it and it used to be a self-discipline at any time as soon as I tried to place it to use.

So I took a whereas to be taught the fundamentals. Here’s what or no longer it’s far crucial to know: 🧵

— Adam Gordon Bell 🤓 (@adamgordonbell) August 24, 2021

Related Articles

Back to top button
%d bloggers like this: