BASH: Filling Web Forms with cURL and wget

Oh gosh, another cURL and web form post! I couldn’t resist writing this one. While finding information on formfind I noticed an absence of direct help for people wishing to use it so I thought I’d fill the gap in the market.

Questions this post answers:

  • How do I fill out an online form without visiting a website?
  • How do I send information to online forms from my desktop?
  • How do I use Bash to fill online forms?
  • How do I find form fields in a website’s source code?
  • How do I download a webpage with wget?
  • How do I download a webpage with cURL?
  • How do I use formfind?

Filling Web Forms with cURL and wget

To fill out web forms using formfind you need to:

  • Download formfind
  • Download the sourcecode of the page that holds the form
  • Use formfind to locate the form fields
  • Use wget or cURL to fill in the form

All of this can be easily automated as described here in the summary.

How to Get Formfind

There are two ways to get formfind:

Download it manually

  1. Download Formfind from GitHub
    1. Note: The script appears to have moved. We can still download it from here at Archive.org.
  2. Copy the code to a text file and save it as formfind
  3. Make the script executable by right-clicking it, selecting properties then clicking “is executable”.

Let Linux do the work for you. We can download Formfind in with a one line instruction:

  1. Copy and past this command into a terminal to download formfind to your desktop and make it executable:

Downloading the Source Code

Before you can use formfind you need to download the sourcecode of the web page the form is on and save the code to a text file. You can do this with wget or cURL.

Using wget:

Using cURL

If wget and cURL are not installed on your computer, you can install them with:

Using Formfind

Formfind finds forms in HTML pages. It writes its report to screen unless its output is redirected to a file or another script.

Formfind is run from the directory it is in by typing the command

If you run formfind from another directory you will need to specify the path to formfind after the “sh” command.

To pass source code from a file to formfind, type

For example, if you had

  1. formfind saved to your desktop (formfind)
  2. The source code of a form page saved to your desktop (file.txt)
  3. A terminal open on your computer (Konsole)

You would use the following command to run formfind to find form data in file.txt

Note that the tilde symbol (~) represents the active user’s home directory and the less than sign (<) tells Bash to feed formfind whatever data is in the file written after it.

Formfind’s output would look similar to this:

We will dissect that report in a few moments.

Using Formfind with wget and cURL to Fille Forms

Both wget and cURL grab the source code of any web page that doesn’t block them. cURL prints its output to the console screen and wget saves files to whichever directory it is being called from.

To use wget to save the source code from the page that holds the form, for example:

To use cURL to do the same, you would use

The greater than sign (>) is used to tell wget and cURL where to send their output.

You need to write two instructions to pass data to formfind:

  1. One to use wget or cURL download the webpage with the form
  2. One to pass the download page source to formfind

For example,

Both those commands can be put into one line by separating them with a semi-colon, like this

Using formfind is as easy as that.

Understanding Formfind’s Report

Take a look at this image:

Formfind: Form Report Screenshot

It is the result of using cURL and Formfind command to collect information about a form.

It shows the following form details:

HTML Web forms work by letting a user click buttons, select items from drop down menus or type information into a text field and so on. What’s actually happening when someone completes a form is that he or she is assigning values to variables. That’s a complicated way of saying that the user is telling a server that “answer one = button one”, “answer two = drop down item three” and “answer four = some message”. The values of those variables is confirmed when the form filler clicks “send”. That information is then usually sent to another page which processes the form data.

The example form has three text fields shown in the screenshot:

  1. Name
  2. Age
  3. Town

The top line of the formfind report reads:

This line tells us the form uses  the “post” method to send data to “page-two.php”. This means the form data is processed by page-two.php.

The lines that begin “Input: Name=” tell us the titles for the fields in the form. Those titles are actually variables. In this case, PHP variables. The quoted texts are their names. For example, the first input line,

tells us that the form has a field called “Name”. The text in brackets tells us it is a TEXT field.

To summarize that, formfind reads the HTML source of a web page to compile a report about any forms it contains. That report tells us the address of the page where the form sends data to be processed, the names of the variables that that page expects to receive, the type of information it expects those variables to hold and – for drop-down boxes and selection boxes – the possible values those variables can hold.

For example, using the screenshot,

  1. The test form is at https://journalxtra.com/tutorials/php-data-passing/page-one.html
  2. Its variables are Name, Age and Town
  3. Clicking “Send” sends those variables to https://journalxtra.com/tutorials/php-data-passing/page-two.php

The name of the “Send” button isn’t required. When we complete a form with wget or cURL, we send data directly to the page that processes the form. In effect, wget and cURL do the job of the send button.

The example URLs shown above no longer lead to real pages. I removed the pages when I migrated JournalXtra out of Hostgator and onto Namecheap.

Click here to learn how to make a web form.

Filling Forms with cURL

cURL can both download web pages and send data back to them; which comes in very handy for filling out web forms. The format we use is this:

The “-d” option is the one that tells cURL that it’s about to be fed data that needs to be sent to whatever URL is stated after the data.

To send MY name (Lee) to a form, I could write:

To send my age (dare  I reveal this!), I would write:

To send all three of my name, age and town to page-two.php in one go, I would write:

That last command sends my answers to all three fields directly to the page that processes the form data (page-two.php).

It’s important to quote the answers to prevent cURL getting mixed up about what it’s being asked to do. For instance, If I put my town as “The Internet”, without the quotes cURL would assume the answer to “Town” is “The” before it tries to send the data to the host called “Internet”.

An alternative to quoting is to use percent-encoding of characters that hold instructional significance to cURL i.e find the UTF8 value of any special character and stick a percent sign in front of it e.g %20 is the space character.

When you complete a form with cURL, cURL will write the server response for the completed form to your screen. You can redirect it to a file with a greater than sign (>). For example,

Click here to learn more about form filling with cURL.

Filling Forms with wget

This works essentially the same as it does for cURL except the form data is expressed slightly differently. This time we use the format:

The differences you need to know about are:

  1. wget uses “–post-data” in place of cURL’s “-d” option
  2. wget combines multiple variable values with an ampersand (&)

For example,

Notice that the space used in “The Internet” is replaced with %20.

An ampersand used as part of a field answer would be replaced with %26.

For example

Note that cURL also accepts the lumping together of form data into one string with an ampersand such as curl -d “Name=Lee&Age=36…”.

Summary

You can use either cURL or wget to fill web forms without using a web browser. The method is to

  1. Download the form page with curl or wget
  2. Feed the downloaded page to formfind
  3. Examine formfind’s report to get the name of the page that processes the form
  4. Examine formfind’s report to get the name of the fields the form contains
  5. Use cURLor wget to send data to the processor page

For example, with cURL

  1. Download formfind to your desktop and make it an executable script
  2. Download the form page to a file named page-one.html on your desktop
  3. Feed form.html to formfind
  4. Examine the output
  5. Send a reply to the POST to URL page (e.g form-data.php) using the field titles shown and write the response to a file (e.g response.html) on your Desktop

I hope this guide helped you learn to use cURL, wget and formfind to complete web forms. Please let me know your results.

Win the battle. Social Warfare Dynamik Website Builder Author Pro

Leave a Reply

42 Comments on "BASH: Filling Web Forms with cURL and wget"

Notify of
Sort by:   newest | oldest | most voted

In the examples where you send the form data to the server (…page-two), it should be page-two.php and not page-two.html

It’s a test to make sure you’re following the guide properly ;)

Thanks Christian, I’ll correct it.

sh ~/Desktop/formfind < ~/Desktop/page-one.html
You need to change the above line.

Thanks for that but it wasn’t that line I needed to change. There was a line with formfind.pl in it that needed changing to “formfind” without “.pl”. I also need to white-list the directory that holds the example form to permit it to accept curl and get requests.

Please tell me from where i will get formfind.pl

Visit this link https://raw.github.com/bagder/curl/master/perl/contrib/formfind

Copy the text and paste it into a new file.

Save the file as formfind.pl

This address does not work anymore. Though, you can find formfind.pl here for instance: https://github.com/MikeRalphson/curl/blob/master/perl/contrib/formfind
or at the page indicated above in the tutorial (better to know the two adresses in case one disappears ;)

Hi Dion awesome post!

Just one question – I would like to submit data to a form with one variable that changes each time I post data to the form. This variable needs to read from another file and needs to cycle through until its read all the lines in the file.

eg something like this

curl -d Name=”Dion” -d Age=”36″ -d Town=”The Internet” FavFood=~Desktopfavfood.txt https://journalxtra.com/tutorials/php-data-passing/page-two.php

How can i achieve this?
 Cheers Bjorn

Hi Bjorn. This can be done quite easily. You just need to set a variable that takes results from a loop through items listed in the file. I’ll write something for you.

Hi Bjorn. This can be done quite easily. You just need to set a variable that takes results from a loop through items listed in the file. I’ll write something for you.

Bjorn, here’s your answer….. download my box of scripts from https://journalxtra.com/webmasterresources/scriptilitious-a-linux-scriptbox-1488/

Unzip it, enter the Scriptilitious-2.3 directory and type ./script* to run it. Use the “Auto Single Form Fill” script. It will let you set as many form variables as you like and you can use a static variable or a data file of variables for any of the form’s variables. Please let me know if you find any bugs. I will update the script but I have a couple of web development jobs to do first.

when I try to run this in terminal
sh /root/Desktop/formfind < /root/Desktop/file.txt
I get this error
/root/Desktop/formfind: 17: =: not found
/root/Desktop/formfind: 19: Syntax error: "{" unexpected (expecting "then")

I am not able to fill the form by script, please help!

Which operating system/distro are you using and how did you download the formfind script?

Completely missed your mistake. I blame tiredness. Don’t put ‘root’ in your command. Try this:

sh ~/Desktop/formfind < ~/Desktop/file.txt

If you did it the long way you would write /home/[username]/Desktop/formfind < /home/[username]/Desktop/file.txt

The tilde symbol, ~, tells Linux/Unix implies the active user's user-space For example, /home/dion/Desktop/ can be written ~/Desktop/

first of all: “chmod 711 /root/Desktop/formfind”

and then run it like:

“./root/Desktop/formfind < /root/Desktop/file.txt"

I get the same error – Ubuntu 12.10 64. Just copy pasted the three lines in the summary :-(

Since formfind is written in perl, this worked for me:
perl formfind < https://site

Greetings from linuxsake, thank you for this brief overview. It helped me understand where I was making a mistake. Thanks alot! I bookmarked you ;)

how do i know the form was filled without problems? wget downloads the same page again!!

What return code?
you mean the process-returncode $? or the http-return code?
How can i check this?

Great post!
Is there a way to display the next page that will pop up after the form was filled and sent?
Best, Bb

How would you post a name and a value?

— FORM report. Uses POST to URL “/model/ip_src.php”
Input: NAME=”search_word” VALUE=”192.168..66.*” (TEXT)
Select: NAME=”search_menu”
Option VALUE=”ip_address” (SELECTED)
Option VALUE=”mac_address”
Option VALUE=”dns_dhcp”
[end of select]

I’ve tried:
curl -d NAME=”search_word” VALUE=”192.168.66.**” http://xx
curl -d NAME=”search_word” -d VALUE=”192.168.66.**” http:/xx

But do not get values on the search criteria. Just the default response.

Try
search_word=192.168.xxxx

I have another Problem.
After i filled out a the Formulars and Things in a VBulletinboard (works fine, no problems),i would send a attachement to that Posting, but i dont known who.
Formfill tells me only
“Attachement” (BUTTON) without any value
I try to send a empty String, but it does not help.
What must i do with a (Button)?
How can i send a attachement?

curl -d ”search_word”=”192.168.66.**&search_menue=value” http://xx

Value is one from the trhee whatever you need : ip_adress, mac_adress, or dns_dhcp

Thank you for the valuable tip. My broadband requires a web login and disconnects frequently. Now i can manage this with a automated bash script.

nice howto! thanks

Pol

tired of doing fromfind but not working

Hi there,

I can’t find formfind script … https://raw.github.com/bagder/curl/master/perl/contrib/formfind 404 not found … can’t be found over google too !

Please help !

Thanks

Heh three monts seeking article about wget and –post method, only this article work for me

Thank you wery much!

hello,
first thank you for this useful post,
second how can i use formfind to press buttom ?

Hello. I am getting an error when:
sh /home/user/Desktop/formfind > /home/user/text.txt
/home/user/Desktop/formfind: 17: /home/user/Desktop/formfind: =: not found
/home/user/Desktop/formfind: 19: /home/user/Desktop/formfind: Syntax error:
“{” unexpected (expecting “then”)
I already tried the two methods: i created the script by copy-paste and also downloaded the script by curl command (as in step 1) directly to my desktop folder.
Any thoughts? Please

I too got “formfind: line 17: =: command not found” etc. like others did but none of the suggestions here worked for me. After “chmod 711 formfind” this worked: “./formfind < file.txt". EDIT: Running arch linux and using php -S localhost:8000 -t public_html/

Please can you provide me with an example of using cUrl to set a radio button on an html page
Thanks,
Geoff

A sample html script section using a radio button that I need to swap over is…

delete (from this list)

Nothing

Hopefully, this will allow a more relevant solution to be posted
Geoff

Hi, This looks great. Can you please describe how to install and use FormFind from Windows, e.g. the command prompt? Thank you.

wpDiscuz

Free to your inbox

Join our mailing list to receive the latest news and updates from JournalXtra.

You have Successfully Subscribed!