Sending a POST request via curl has issues with double quotes

Question

So, in-order to send a POST request to gists (github), you can do something like this as seen in https://gist.github.com/caspyi...

curl --user "user" -X POST --data '{"description":"Created via API","public":"true","files":{"file1.txt":{"content":"Demo"}}' https://api.github.com/gists

But, in the above example, the file name and the content of the file are hard coded which is the part ..file1.txt":{"content":"Demo"}..

I am replacing the above part with my variable $file":{"content":"$content"} but initialize the variable, the json request must be enclosed in double quotes, which I did as

curl --user "user" -X POST --data "{\"description\":\"Created via API\",\"public\":\"true\",\"files\":{\"$file\":{\"content\":\"$content\"}}' https://api.github.com/gists

But this does not work, I get json error.

{
  "message": "Problems parsing JSON",
  "documentation_url": "https://developer.github.com/v3/gists/#create-a-gist"
}

Even, if I replaced all the escaped double-quotes with \' single quotes.

Does anyone know how to include a variable inside this json request? btw: I have used all headers such as

 -H "Content-Type: application/json; charset=UTF-8" 

and many combinations to validate the request but to no avail

UPDATE.

This is what the entire content looks like.

function gist_controller(){
    content=$(cat $1)
    DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
    read -p "enter your password - " pass 
    public="false"
    gist_content=$(cat $1)
    curl --user "samserayo"  -H "Content-Type: application/json; charset=UTF-8" -X POST -d  "{ 'description': 'Created via API', 'public': 'true', 'files':{ ' '$1 ':{ 'content': '$gist_content'}}" https://api.github.com/gists
}

UPDATE 2

The file that causes the script to break (the file I am trying to upload is)

<?php echo 'hello world' ?>

Answer 1

You don't show what value you have set for $file or $content. What do those both expand out to? It's likely one, or the other, or both, contain characters the JSON parser doesn't like. Most likely, something inside the variable expansion value needs to be further escaped or encoded before submission to the JSON parser through curl.

Have you EXPORTed the environment variables?

Try:

echo "$file $content";

to make sure they are both set as you expect them to be.

Answer 2

You didn't specify what shell you are using, but anything within double-quotes is subject to shell filename expansion. In particular, {} has meaning to most shells. That's the reason that the example you cited used single quotes to wrap the whole thing.

Shell escapes are tricky, and sometimes it seems impossible to get what you want. Honestly, in cases like this I would write a Python script to call curl, where I know the shell isn't "helping" me out.

Answer 3

Since this got bumped for review, here's another answer.

Escape the parts separately (close the first escape, then escape your variable):

curl --user "user" -X POST --data '{"description":"Created via API","public":"true","files":{"'"${file}"'":{"content":"'"${content}"'"}}}' https://api.github.com/gists

or split up for readability:

'{"description":"Created via API","public":"true","files":{"'
"${file}"
'":{"content":"'
"${content}"
'"}}}' 

You may or may not also have to change the internal field separator, so the shell doesn't interpret any whitespace in your variables itself.

MWE: cat /tmp/myfile | ./thisscript.sh "myfilename.txt"

#!/bin/sh

FILENAME="${1}"
#CONTENT="$(cat)"
CONTENT="$(sed -e 's/"/\\"/g')" # escape stuff

OFS="${IFS}"
IFS=''

PERSONAL_ACCESS_TOKEN="e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e"

curl --user "username:${PERSONAL_ACCESS_TOKEN}" -X POST --data '{"description":"Created via API","public":"true","files":{"'"${FILENAME}"'":{"content":"'"${CONTENT}"'"}}}' https://api.github.com/gists
IFS="${OFS}"

For this to work correctly, you will of course still need to escape all the characters that would break JSON, in your file contents, such as " and control characters (DOS carriage-returns), etc.
It is ...unfortunate, that they decided to use JSON datastructures for file content uploads.

---

If I may suggest a different way of creating Gists, use the curl API call only to create a single file, say README with some boilerplate content (can't be empty).
Then grep the returned json structure for git_push_url and clone the gist git repo. After that you can just git commit, git push stuff to the gist without worrying about binary data or escaping issues.

Answer 4

You have to put the entire string inside double quotes ". And you need to escape quotes inside the string (not convert them into other types of quotes!).

So instead of:

"{ 'description': 'Created via API', 'public': 'true', 'files':{ ' '$1 ':{ 'content': '$gist_content'}}"

You want:

"{ \"description\": \"Created via API\", \"public\": \"true\", \"files\" : { \"$1\" : { \"content\": \"$gist_content\"}}}"

You should probably test with echo and make sure the string is exactly correct.

Answer 5

Here is a solution but it assumes two things.

That the file is a filename provided to the script as an first argument. And the content is the file content as text.

Also please note different POST URL than Yours (see below for explanation).

#!/bin/bash

file=$1
content=$(cat $1)

curl -H "Content-Type: application/json; charset=UTF-8" -X POST -d  "{ 'description': 'Created via API', 'public': 'true', 'files':{ ' '${file}':{ 'content': '${content}'}}" https://postman-echo.com/post

content.txt file contains:

This is content of the content file.

Example run:

./curl.sh content.txt

Example output:

{
  "args": {},
  "data": "{ 'description': 'Created via API', 'public': 'true', 'files':{ ' 'content.txt ':{ 'content': 'This is content of the content file.'}}",
  "files": {},
  "form": {},
  "headers": {
    "x-forwarded-proto": "https",
    "host": "postman-echo.com",
    "content-length": "134",
    "accept": "*/*",
    "content-type": "application/json; charset=UTF-8",
    "user-agent": "curl/7.65.3",
    "x-forwarded-port": "443"
  },
  "json": null,
  "url": "https://postman-echo.com/post"
}

NOTES:

I am using here the https://postman-echo.com/post website which echoes everything that You post to it as JSON.

If the content of the content.txt file is more complicated this solution may break as it still will have to be properly escaped. The content is being put to the "content" field so all escaping rules for json applies here.

To see the response beautifully displayed (as above json) add at the end of the curl | jq . (You may need to install it first) :

https://postman-echo.com/post | jq . 

It is possible to use jq to automatically quote entire content file (if needed for more advanced usage):

$ jq -Rs '.' content.txt
"This is just a text.\n"

$ jq -Rs '.' content.cpp
"#include <iostream>\nusing namespace std;\nint main() \n{\n    cout << \"Hello, World!\";\n    return 0;\n}\n"

You will have to slightly modify the script curl.sh for it to work :

content=$(jq -Rs '.' $1)