[lug] Convert JSON to CSV - adding header line

Jed S. Baer blug at jbaer.cotse.net
Fri Jan 15 18:43:42 MST 2021


Hi Everyone. Oh, so much fun we're having now!

I'm off in the weeds here, looking at taking the ham DMR users database,
available as a JSON file, and converting it to CSV. "DMR" is "Digital
Mobile Radio". Yes, it's also available as a CSV, but the people producing
it aren't enclosing the fields in quotation marks, so when the value
includes a comma, it's problematic. GIGO! Grabbing it as JSON, then
converting to CSV makes a better dataset, and it's also a time when I can
selectively grab fields.

So.... my incantation is:
jq -r '.users[] | [.radio_id, .callsign, .fname, .surname, .city, .state,
.country, .remarks] | @csv'

And this works fine, even for 137K rows. Yes, the whole JSON file is one
big JSON array, which seems odd to me, but whatever.

However, in trying to find how to tack on a header row with column names,
it's getting murky. The std. answers on StackOverflow, and elsewhere, all
use the map command, plus a unique function, and assume that the intent is
to grab all the columns. I don't want that, and in fact, have to slightly
rename the fields anyway.

Yes, I can make a file with the column headers, and just use cat to append
that with the output of jq, but it seems to me that there ought to be a way
to create a literal array that jq will include as the first line in the
output. I'm thinking that somehow, that's what happens with all the piping
and such in the examples using the map filter - somehow, this resolves to
an array that winds up being the first line, before the converted data.
Also, I'm hoping to hand this solution to another ham who will be using it
in Windoze, so the more self-contained it is, the better.

The first line is:
RADIO_ID,CALLSIGN,FIRST_NAME,LAST_NAME,CITY,STATE,COUNTRY,REMARKS

I actually don't know that it has to be all caps - the software used to
program the radio is very poorly documented. I also don't know whether
it'll barf if the above field names are quoted. Hmmm, I also don't whether
it'll barf if any of the field values are quoted. Well, once I have the CSV
file in a better state, and can deal with that.

Thanks for any advice.
-- 
All operating systems suck, but Linux just sucks less
 - Linus Torvalds


More information about the LUG mailing list