<article>
<h2 id="overview">Overview</h2>
<p>In this episode we will continue looking at basic filters. Then we
will start looking at the feature that makes <code>jq</code> very
powerful, the ability to transform JSON from one form to another. In
essence we can read and parse JSON and then construct an alternative
form.</p>
<h2 id="more-basic-filters">More basic filters</h2>
<h3 id="arraystring-slice-.numbernumber">Array/String Slice:
<code>.[&lt;number&gt;:&lt;number&gt;]</code></h3>
<p>This filter allows parts of JSON arrays or strings to be
extracted.</p>
<p>The first number is the index of the elements of the array or string,
starting from zero. The second number is an ending index, but it means
"up to but not including". If the first index is omitted it refers to
the start of the string or array. If the second index is blank it refers
to the end of the string or array.</p>
<p>This example shows using an array and extracting part of it:</p>
<pre><code>$ x=&quot;[$(seq -s, 1 10)]&quot;
$ echo &quot;$x&quot;
[1,2,3,4,5,6,7,8,9,10]
$ echo &quot;$x&quot; | jq -c &#39;.[3:6]&#39;
[4,5,6]</code></pre>
<p>Here we use the <code>seq</code> command to generate the numbers 1-10
separated by commas in a JSON array. Feeding this to <code>jq</code> on
its standard input with the slice request '<code>.[3:6]</code>' results
in a sub-array from element 3 (containing value 4), up to <em>but not
including</em> element 6 (containing 7). Note that using the
<code>'-c'</code> option generates compact output, as we discussed in
the last episode.</p>
<p>For a string, the idea is similar, as in:</p>
<pre><code>$ echo &#39;&quot;Hacker Public Radio&quot;&#39; | jq &#39;.[7:10]&#39;
&quot;Pub&quot;</code></pre>
<p>Notice that we provide the JSON string quotes inside single quotes
following <code>echo</code>. The filter '<code>.[7:10]</code>' starts
from element 7 (letter "P") up to <em>but not including</em> element 10
(letter "l").</p>
<p>Both of the numbers may be negative, meaning that they are offsets
from the end of the array or string.</p>
<p>So, using '<code>.[-7:-4]</code>' in the array example gives the same
result as '<code>.[3:6]</code>', as do '<code>.[3:-4]</code>' and
'<code>.[-7:6]</code>'. This example uses the <code>x</code> variable
created earlier:</p>
<pre><code>$ for f in &#39;.[-7:-4]&#39; &#39;.[3:6]&#39; &#39;.[3:-4]&#39; &#39;.[-7:6]&#39;; do
&gt; echo &quot;$x&quot; | jq -c  &quot;$f&quot;
&gt; done
[4,5,6]
[4,5,6]
[4,5,6]
[4,5,6]</code></pre>
<p>Similarly, using '<code>.[-12:-9]</code>' gives the same result as
'<code>.[7:10]</code>' when used with the string.</p>
<pre><code>$ echo &#39;&quot;Hacker Public Radio&quot;&#39; | jq &#39;.[-12:-9]&#39;
&quot;Pub&quot;</code></pre>
<p>As a point of interest, I wrote a little Bash loop to show the
positive and negative offsets of the characters in the test string -
just to help me visualise them. See the <mark>footnote</mark><a
href="#fn1" class="footnote-ref" id="fnref1"
role="doc-noteref"><sup>1</sup></a> for details.</p>
<p>Finally, here is how to get the last character of the example string
using positive and negative offsets:</p>
<pre><code>$ echo &#39;&quot;Hacker Public Radio&quot;&#39; | jq &#39;.[18:]&#39;
&quot;o&quot;
$ echo &#39;&quot;Hacker Public Radio&quot;&#39; | jq &#39;.[-1:]&#39;
&quot;o&quot;</code></pre>
<h3 id="arrayobject-value-iterator-.">Array/Object Value Iterator:
<code>.[]</code></h3>
<p>This filter generates values from iterating through an array or an
object. It is similar to the <code>.[index]</code> syntax we have
already seen, but it returns all of the array elements:</p>
<pre><code>$ arr=&#39;[&quot;Kohinoor&quot;,&quot;plastered&quot;,&quot;downloadable&quot;]&#39;
$ echo &quot;$arr&quot; | jq &#39;.[]&#39;
&quot;Kohinoor&quot;
&quot;plastered&quot;
&quot;downloadable&quot;</code></pre>
<p>The strings in the array are returned separately, not as an array.
This is because this is an iterator, and its output can be linked to
other filters.</p>
<p>It can also be used to iterate over <u>values</u> in an object:</p>
<pre><code>$ obj=&#39;{&quot;name&quot;: &quot;Hacker Public Radio&quot;, &quot;type&quot;: &quot;Podcast&quot;}&#39;
$ echo &quot;$obj&quot; | jq &#39;.[]&#39;
&quot;Hacker Public Radio&quot;
&quot;Podcast&quot;</code></pre>
<p>This iterator does not work on other data types, just arrays and
objects.</p>
<p>An alternative iterator <code>.[]?</code> exists which ignores
errors:</p>
<pre><code>$ echo &quot;true&quot; | jq &#39;.[]&#39;
jq: error (at &lt;stdin&gt;:1): Cannot iterate over boolean (true)</code></pre>
<p>Ignoring errors:</p>
<pre><code>$ echo &quot;true&quot; | jq &#39;.[]?&#39;</code></pre>
<h2 id="using-multiple-filters">Using multiple filters</h2>
<p>There are two operators that can be placed between filters to combine
their effects: the <strong>comma</strong> (<code>','</code>) and the
<strong>pipe</strong> (<code>'|'</code>).</p>
<h3 id="comma-operator">Comma operator</h3>
<p>The comma (<code>','</code>) operator allows you to chain together
multiple filters. As we already know, the <code>jq</code> program feeds
the input it receives on standard input or from a file into whatever
filter it is given. So far we have only seen a single filter being
used.</p>
<p>With the comma operator the input to <code>jq</code> is fed to all of
the filters separated by commas in left to right order. The result is a
concatenation of the output of all of these filters.</p>
<p>For example, if we take the output from the HPR stats page which was
mentioned in <a
href="https://hackerpublicradio.org/eps/hpr4104/index.html">part 1</a>
of this series of shows, and store it in a file called
<code>stats.json</code> we can view two separate parts of the JSON like
this:</p>
<pre><code>$ curl -s https://hub.hackerpublicradio.org/stats.json -O

$ jq &#39;.shows , .queue&#39; stats.json
{
  &quot;total&quot;: 4756,
  &quot;twat&quot;: 300,
  &quot;hpr&quot;: 4456,
  &quot;duration&quot;: 7640311,
  &quot;human_duration&quot;: &quot;0 Years, 2 months, 29 days, 10 hours, 18 minutes and 31 seconds&quot;
}
{
  &quot;number_future_hosts&quot;: 6,
  &quot;number_future_shows&quot;: 18,
  &quot;unprocessed_comments&quot;: 0,
  &quot;submitted_shows&quot;: 0,
  &quot;shows_in_workflow&quot;: 51,
  &quot;reserve&quot;: 20
}</code></pre>
<p>This applies the filter <code>.shows</code> (an <em>object
identifier-index</em> filter, see <a
href="https://hackerpublicradio.org/eps/hpr4114/index.html">part 2</a>)
which returns the contents of the object with that name, then it applies
filter <code>.queue</code> which returns the relevant JSON object.</p>
<h3 id="pipe-operator">Pipe operator</h3>
<p>The pipe (<code>'|'</code>) operator combines filters by feeding the
output of the first (left-most) filter of a pair into the second
(right-most) filter of a pair. This is analogous to the way the same
symbol works in the Unix shell.</p>
<p>For example, if we extract the <code>'shows'</code> object from
<code>stats.json</code>, we can then extract the value of the
<code>total'</code> key' as follows:</p>
<pre><code>$ jq &#39;.shows | .total&#39; stats.json
4756</code></pre>
<p>Interestingly, chaining two <em>object identifier-index</em> filters
gives the same output:</p>
<pre><code>$ jq &#39;.shows.total&#39; stats.json
4756</code></pre>
<p>(Note: to answer the question in the audio, the two filters shown can
also be written as <code>'.shows .total'</code> <u>with</u> intervening
spaces.)</p>
<p>We will see the pipe operator being used in many instances in
upcoming episodes.</p>
<h2 id="parentheses">Parentheses</h2>
<p>It is possible to use parentheses in filter expressions in a similar
way to using them in arithmetic, where they group parts together and can
change the normal order of operations. They can be used in other
contexts too. The example is a simple arithmetic one:</p>
<pre><code>$ jq &#39;.shows.total + 2 / 2&#39; stats.json
4757
$ jq &#39;(.shows.total + 2) / 2&#39; stats.json
2379</code></pre>
<h2 id="examples">Examples</h2>
<h3 id="finding-country-data-1">Finding country data #1</h3>
<p>Here we are using a file called <code>countries.json</code> obtained
from the <a href="https://github.com/mledoze/countries">GitHub
project</a> listed below. This file is around 39,000 lines long so it is
not being distributed with the show. However, it's quite interesting and
you are encouraged to grab a copy and experiment with it.</p>
<p>I will show ways in which the structure can be examined and reported
with <code>jq</code> in a later show, but for now I will show an example
of extracting data:</p>
<pre><code>$ jq &#39;.[42] | .name.common , .capital.[]&#39; countries.json
&quot;Switzerland&quot;
&quot;Bern&quot;</code></pre>
<ul>
<li>The file contains an array of country objects; the one with index 42
is Switzerland.</li>
<li>The name of the country is in an object called <code>"name"</code>,
with the common name in a keyed field called <code>"common"</code>, thus
the filter <code>.name.common</code>.</li>
<li>In this country object is an object called <code>"capital"</code>
holding an array containing the name (or names) of the capital city (or
cities). The filter <code>.capital.[]</code> obtains and displays the
contents of the array.</li>
<li>Note that we used a <code>comma</code> operator between the
filters.</li>
</ul>
<h3 id="finding-country-data-2">Finding country data #2</h3>
<p>Another search of the <code>countries.json</code> file, this time
looking at the languages spoken. There is an object called
<code>"languages"</code> which contains abbreviated language names as
keys and full names as the values:</p>
<pre><code>$ jq &#39;.[42] | .name.common , .capital.[] , .languages&#39; countries.json
&quot;Switzerland&quot;
&quot;Bern&quot;
{
  &quot;fra&quot;: &quot;French&quot;,
  &quot;gsw&quot;: &quot;Swiss German&quot;,
  &quot;ita&quot;: &quot;Italian&quot;,
  &quot;roh&quot;: &quot;Romansh&quot;
}</code></pre>
<ul>
<li>Using the filter <code>.languages</code> we get the whole object,
however, using the iterator <code>.[]</code> we get just the
values.</li>
</ul>
<pre><code>$ jq &#39;.[42] | .name.common , .capital.[] , .languages.[]&#39; countries.json
&quot;Switzerland&quot;
&quot;Bern&quot;
&quot;French&quot;
&quot;Swiss German&quot;
&quot;Italian&quot;
&quot;Romansh&quot;</code></pre>
<ul>
<li>This has some shortcomings, we need the <em>construction</em>
capabilities of <code>jq</code> to generate more meaningful output.</li>
</ul>
<h2 id="next-episode">Next episode</h2>
<p>In the next episode we will look at <em>construction</em> - how new
JSON output data can be generated from input data.</p>
<h2 id="links">Links</h2>
<ul>
<li>jq:
<ul>
<li><a href="https://jqlang.github.io/jq/manual/">The <code>jq</code>
manual</a></li>
</ul></li>
</ul>
<ul>
<li>Test data sources:
<ul>
<li><a href="https://hub.hackerpublicradio.org/stats.json">HPR
Statistics</a></li>
<li><a href="https://randomuser.me/documentation">Random User Generator
API</a></li>
<li><a href="https://github.com/mledoze/countries">Github project
<code>mledoze/countries</code></a></li>
</ul></li>
</ul>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Romansh_language">Romansh
language</a></li>
</ul>
<ul>
<li>Previous episodes:
<ul>
<li><a
href="https://hackerpublicradio.org/eps/hpr4104/index.html">Introduction
to jq - part 1</a></li>
<li><a
href="https://hackerpublicradio.org/eps/hpr4114/index.html">Introduction
to jq - part 2</a></li>
</ul></li>
</ul>
<aside id="footnotes" class="footnotes footnotes-end-of-document"
role="doc-endnotes">
<hr />
<ol>
<li id="fn1"><p><strong>Footnote</strong>: A Bash loop to show positive
and negative index values relating to an example string:</p>
<pre><code>$ y=&#39;Hacker Public Radio&#39;
$ for ((i=0,j=${#y}; i&lt;&quot;${#y}&quot;; i++,j--)); do printf &#39;%02d: -%02d: %s&bsol;n&#39; &quot;$i&quot; &quot;$j&quot; &quot;${y:$i:1}&quot;; done
00: -19: H
01: -18: a
02: -17: c
03: -16: k
04: -15: e
05: -14: r
06: -13:
07: -12: P
08: -11: u
09: -10: b
10: -09: l
11: -08: i
12: -07: c
13: -06:
14: -05: R
15: -04: a
16: -03: d
17: -02: i
18: -01: o</code></pre>
<p>Note that variable <code>y</code> doesn't contain a valid JSON
string; we are just counting characters here to demonstrate the offsets
that can be used when slicing a string! <a href="#fnref1"
class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</aside>
</article>
