HowTo: Pass Data Variables to cURL to Autofill Web Forms

: Getting cozy with cURL

This isn’t a top 10 magical things to do with cURL post. There are plenty of them around the Net and you will find a few links at the bottom of here to prove that. No, this post is about one thing, using a variable to send data to cURL without it getting jumbled up in the process.

cURL is a command-line utility for fetching and sending data to servers. It’s a little like wget.

You might think you can simply create a variable and use that variable in your cURL command and expect cURL to know what to do with it but that doesn’t quite work, as anyone who’s tried doing that will attest. Doing that with spaces or characters in your data that can be confused with cURL special characters and cURL options will get you a “curl: (6) Couldn’t resolve host” error.

For example, if you want to autofill a web form that has the fields

Input: NAME="Name" (TEXT)
Input: NAME="Age" (TEXT)
Input: NAME="Town" (TEXT)
Input: NAME="Form_Submit" VALUE="Send" (SUBMIT)

To post data to that webpage you would use something like this:

curl -d Name="Captain Kirk" -d Age="123" -d Town="The USS Enterprise" -d Form_Submit="Send" http://www.example.com/process-form.php

The “-d” option tells cURL that the next item is some data to be sent to the server at http://www.example.com/process-form.php.

The first item after “-d” is the name of the form field e.g Town=”The USS Enterprise”. The equality sign and the item in quotes that follows the form-field is the data that’s to be sent to the server.

cURL expects either an option or a URL to follow a space. Without the quotes surrounding the data being sent in, for instance, Name=”Captain Kirk”, cURL would assume “Kirk” to be a URL so would try sending the data Name=Captain to the non-existent web address of Kirk. cURL would reply with “curl: (6) Couldn’t resolve host Kirk”.

So, the general format for sending data with cURL is:

curl -d form-field="data data" URL

If we used a variable to pass data to cURL we might write something like this:

DATA="Name='Captain Kirk'"
curl -d $DATA URL

But that won’t work because the quotes are stripped out of the data stored in the $DATA variable so it becomes:

Name=Captain Kirk

Which cURL sees as

curl -d Name=Captain Kirk URL

With cURL we can send the same information to multiple URLs by stating each URL separated by a space. Our example cock-up attempt to use a variable tells cURL to fill out the form field “Name” with the data “Captain” at the hosts “Kirk” and “URL”.

The solution is to replace cURL special characters and option codes with their Unicode equivalents. In this case, we would replace the space in the data with “%20”.

Replacing “space” with “%20” in our example would make it

DATA="Name=Captain%20Kirk"
curl -d $DATA URL

Now cURL receives this

curl -d Name=Captain%20Kirk URL

The server that receives the data converts “%20” to a regular space character.

Combining Form Data

cURL allows us to combine multiple form fields together with an ampersand (&). For example,

curl -d Name="Captain Kirk" -d Age="123" -d Town="The USS Enterprise" -d Form_Submit="Send" URL

can be written

curl -d Name=Captain%20Kirk&Age=123&Town=The%20USS%20Enterprise&Form_Submit=Send URL

Notice we use only one “-d” option and which is placed in front of our data string.

But, what happens when, for example..

curl -d comment="..our form data contains ampersands and other special characters such "-d", double & single quotes, ats (@) or URLs?"

I’ll give you a clue: what happens when you put lots of different fruits into a blender?

Again, we must convert the special characters in our form fields into their Unicode equivalents.

In Bash, I do this with sed. The following Bash script puts the data being passed to cURL into a text file then uses sed to convert special characters to their Unicode equivalents before recalling the data from the text file (using cat) and placing it back into the variable which is passed to cURL:

#! /bin/bash

# Put the form fill data into the $DATA variable.

DATA="comment=some data & stuff"

# Specify the URL to send info to

URL="http://example.com"

# Put $DATA into a file

echo "$DATA" > temp-file.txt

# Convert special characters to their Unicode equivalents #

sed -ir 's# #%20#g' $workbox/$tempthree # space
sed -ir 's#"#%22#g' $workbox/$tempthree # "
sed -ir "s#'#%27#g" $workbox/$tempthree # '
sed -ir 's#`#%60#g' $workbox/$tempthree # `
sed -ir 's#\\#%5c#g' $workbox/$tempthree # \
sed -ir 's#&%20#26%20#g' $workbox/$tempthree # &space;
sed -ir 's#%20&#%2026#g' $workbox/$tempthree # space&
sed -ir 's#@#%40#g' $workbox/$tempthree # @
sed -ir 's#-#%2d#g' $workbox/$tempthree # -
sed -ir 's#=%20#%3d%20#g' $workbox/$tempthree # =space
sed -ir 's#%20=#%20%3d#g' $workbox/$tempthree # space=

# Read the data in the file temp-file.txt into the variable $DATA

DATA=`cat temp-file.txt`

# Pass variables to cURL

curl -d $DATA $URL

The above code converts ampersands and equality signs only when they are preceded or proceeded by a space character. This limitation is to prevent intended special character “=” and “&” statements from being unintentionally converted to Unicode.

The same can be done in PHP by using preg_match() (external site) instead of sed.

So there you have it. Now you know how to use a variable with cURL.

A Unicode chart is available here (external site) in case you need to convert additional characters.

Working Example Script

You can see a better example of passing variables to cURL in the Auto-Form-Filler Bash script I added to Scriptilitious a few days ago.

Auto-Form-Filler uses a slightly different approach: it lets the user specify a number of form fields to complete, asks the user to fill-in those form fields then stores the user input in an array. The individual array records are then put into a file and any special characters are then converted to Unicode before the data is put back into the array. The array is then used to build the data input into a single line of data that is put into a variable that can be read by cURL.

Converting user input into a variable as Auto-Form-Filler does it allows all characters in the user input that have special meaning to cURL to be converted before the cURL required special characters are used to concatenate the data fields in the cURL required format.

Examine the Auto Form Filler script by downloading Scriptilitious and looking at the auto-form-fill file in its ScriptBox directory (Get Scriptilitious here).

Auto-Form-Filler reads a list of URLs that it sends form data to. Let me know what you think to it.

Resource Links

Here are the goods I promised at the beginning of the article: links to blogs that show you nifty cURL tricks. Oh, and a link to cURL’s homepage.

cURL’s Homepage

UTF 8 Table

10 Awesome Things to do with cURL

6 Essential cURL Commands

Do you know a better way to use variables with cURL? Or do you have a few tricks you would like to share? Leave a comment. If you have a website stuffed with cURL tips and tricks I might even add it to this resource section.