Unable to resolve parameter inside json

Question

I'm using a curl command inside a script (edm.ksh) to retrieve some data from an API, it requires a password in a JSON like {"password":"myPassword"}. Now, myPassword is not hardcoded, it comes from a config file which I am extracting inside edm.ksh in a variable.

edm.ksh:

#! /usr/bin/ksh
export PASSWORD=$(grep PASSWORD /home/dummy/password.txt | awk -F " " '{print $2}')
export PASSWORD_TXT=$(echo -e "'"'{"password":"'${PASSWORD}'"}'"'")
curl -k -X POST -d $PASSWORD_TXT ${LDAP_URL} > ${TEMP_DIR}/token.dat

$ cat /home/dummy/password.txt
PASSWORD myPassword

When I run the script, I'm expecting $PASSWORD_TXT to resolve as '{"password":"myPassword"}' but it's coming out to be $'\'{"password":"myPassword"}\'' as script output. It resolves perfectly fine in command line but not in script. Can anyone assist on how to achieve a proper json resolution with escape characters?

I even tried something like

curl -k -X POST -d  '{"password":"${PASSWORD_TXT}"}' ${LDAP_URL} > ${TEMP_DIR}/token.dat

but the variable does not resolve inside a JSON.

I would really appreciate if I get any lead on this issue.

Thank you in advance!

Answer 1

There are several issues in the code that you are showing:

• The password is read from the file in a manner that assumes no spaces or other whitespace are part of the password.
• The JSON document containing the password does not JSON-encode the password string, which means you restrict the characters allowed in the password.
• Several shell variables are either single-quoted (meaning they won't expand to their values) or unquoted (meaning the shell will expand their values and split it on whitespace and perform filename globbing on the expanded bits).

Assuming the password is the rest of the line after the first space on the line that starts with the string PASSWORD in the file password.txt, then you would read that using

password=$( sed -e '/^PASSWORD /!d' -e 's///' -e q <password.txt )

This discards all lines that do not start with PASSWORD , removes that prefix string from the first line that has it, outputs the result and stops processing the file. Note that the password variable does not need to be exported.

Example:

$ cat password.txt
PASSWORD this is "the password"
$ password=$( sed -e '/^PASSWORD /!d' -e 's///' -e q <password.txt )
$ printf 'Password: %s\n' "$password"
Password: this is "the password"

Note that reading the password in this way still removes trailing whitespace from the password's value (this is due to the command substitution).

Creating a JSON document that contains the read password string as the value of the key password can be done with the JSON processing tool jq:

json_payload=$( jq -n -c --arg password "$password" '$ARGS.named' )

or, using the command line JSON composing tool jo,

json_payload=$( jo password="$password" )

or, using Miller, the go-to tool for dealing with structured document formats in general,

json_payload=$( mlr -n --ojsonl put -s password="$password" 'end { emit @password }' )

or, using Python,

json_payload=$( python -c 'import sys, json; print(json.dumps({"password": sys.argv[1]}))' "$password" )

or, using Perl,

json_payload=$( perl -MJSON -e 'print encode_json({"password" => $ARGV[0]});' "$password" )

This encodes the string as a valid JSON-encoded string, no matter what characters are part of the password, and inserts the string as the value of the key password in a JSON object. This shell variable also does not need to be exported.

Example:

$ printf 'Password: %s\n' "$password"
Password: this is "the password"
$ json_payload=$( jq -n -c --arg password "$password" '$ARGS.named' )
$ printf 'JSON document: %s\n' "$json_payload"
JSON document: {"password":"this is \"the password\""}

Using this with curl to call the API endpoint:

curl --silent \
    --request POST \
    --header 'Content-Type: application/json' \
    --data "$json_payload" "$api_endpoint_url" >output

None of the quoting in the code I presented here is accidental or superfluous. It is essential to double-quote all expansions so that the shell does not split the strings in the variables or interpret characters in special ways. See e.g. When is double-quoting necessary?

---

Collecting all of the above in a single script, without error handling, getting rid of the json_payload shell variable (as it's not needed), and using the --json shortcut option with curl:

#!/bin/sh

api_endpoint_url='some URL here'

password=$( sed -e '/^PASSWORD /!d' -e 's///' -e q <password.txt )

jq -n -c --arg password "$password" '$ARGS.named' |
curl --silent --json @- "$api_endpoint_url" >output