Bash variable substitution in a JSON string

Question

I'm trying to create a JSON in BASH where one of the fields is based on the result of an earlier command

BIN=$(cat next_entry)
OUTDIR="/tmp/cpupower/${BIN}"
echo $OUTDIR
JSON="'"'{"hostname": "localhost", "outdir": "${OUTDIR}", "port": 20400, "size": 100000}'"'"
echo $JSON

The above script when executed, returns:

/tmp/cpupower/0
, port: 20400, size: 100000}': /tmp/cpupower/0

How can I properly substitute variables inside these multi-quoted strings?

Answer 1

JSON=\''{"hostname": "localhost", "outdir": "'"$OUTDIR"'", "port": 20400, "size": 100000}'\'

That is get out of the single quotes for the expansion of $OUTDIR. We did put that expansion inside double-quotes for good measure even though for a scalar variable assignment it's not strictly necessary.

When you're passing the $JSON variable to echo, quotes are necessary though to disable the split+glob operator. It's also best to avoid echo for arbitrary data:

printf '%s\n' "$JSON"

Answer 2

If you ended up here trying to use AWS commands, @Stéphane Chazelas's answer almost works. In here, the initial escaped quotes (\') are not necessary, they actualy break the command.

IP=$(curl ipecho.net/plain ; echo)

aws ec2 authorize-security-group-ingress --group-id sg-**************** \
    --ip-permissions '[{"IpProtocol": "tcp", "FromPort": 15000, "ToPort": 15000, "IpRanges": [{"CidrIp": "'"$IP/32"'", "Description": "Service A"}]}]'

^ This works just fine

Answer 3

Stéphane's answer is great, and upvoted. Here's just a tip; instead of doing

BIN=$(cat next_entry)

You can do:

BIN=$(<next_entry)

And thus save spawning an extra process. Read more here.

Answer 4

The contents of the string in $BIN likely contains DOS-style newlines, with a carriage return character before each newline character. When printing this to the terminal, the carriage return character(s) moves the cursor back to start of the line, which is why your output looks a bit strange.

Another consequence of this is that your JSON document will be invalid as carriage returns (and newlines) will need special encoding. You also can't insert any character into a JSON value as some characters, notably quoting characters, need to be encoded.

I would suggest two things:

1. Convert your data file to Unix text format using the dos2unix conversion tool.

2. Use a command line tool to create your JSON document that knows how to encode data for JSON. The jo utility is one such tool.

Using jo:

#!/bin/sh

bin=$(cat next_entry)
outdir=/tmp/cpupower/$bin

json=$( jo hostname=localhost outdir="$outdir" port=20400 size=100000 )

printf '%s\n' "$json"

Given a file next_entry with the contents

line 1
line 2

the script above would output

{"hostname":"localhost","outdir":"/tmp/cpupower/line 1\nline 2","port":20400,"size":100000}

If jo is given its -p option for pretty-printing the constructed JSON document:

{
   "hostname": "localhost",
   "outdir": "/tmp/cpupower/line 1\nline 2",
   "port": 20400,
   "size": 100000
}

Answer 5

If I have a json file that goes like eating.json = {"diner" : {"time" : ""}}, the way I do it is the following :

declare -a NOW
NOW=$(date) 
jq '.diner.time = "'"${NOW}"'"' eating.json > tmp.$$.json && mv tmp.$$.json eating.json

So, first, I created my variable, then, I put the value I wanted in it, and, finally, using jq, I made the modification in a temporary file (specificities of the jq module I believe), and then put the modifications back in eating.json.

Answer 6

Best solution I've seen in terms of readability (and I take no credit for this) is here:

https://stackoverflow.com/a/64864511/9761598

Copying the key part of that solution here:

json=$(cat <<-END
    {
        "tag_name": "${version}", 
        "target_commitish": "${branch}", 
        "name": "${title}", 
        "body": "${notes}", 
        "draft": ${is_draft}, 
        "prerelease": ${is_prerelease} 
    }
END
)

This uses a here document to allow a readable multi-line JSON block with variable substitution while preserving the double quotes. The here doc is redirected to the cat command, which is wrapped in $() to put the output into the json variable.