Oh gosh, another cURL and web form post! I couldn’t resist writing this one. While finding information on formfind I noticed an absence of direct help for people wishing to use it so I thought I’d fill the gap in the market.
Questions this post answers:
- How do I fill out an online form without visiting a website?
- How do I send information to online forms from my desktop?
- How do I use Bash to fill online forms?
- How do I find form fields in a website’s source code?
- How do I download a webpage with wget?
- How do I download a webpage with cURL?
- How do I use formfind?
Filling Web Forms with cURL and wget
To fill out web forms using formfind you need to:
- Download formfind
- Download the sourcecode of the page that holds the form
- Use formfind to locate the form fields
- Use wget or cURL to fill in the form
All of this can be easily automated with BASH script.
How to Get Formfind
There are two ways to get formfind:
Download it manually
- Download Formfind from GitHub
- Copy the code to a text file and save it as formfind
- Make the script executable by right-clicking it, selecting properties then clicking “is executable”.
Let Linux do the work for you. We can download Formfind in with a one line instruction:
- Copy and past this command into a terminal to download formfind to your desktop and make it executable:
curl https://raw.githubusercontent.com/VR51/formfind/master/formfind.pl > ~/Desktop/formfind.pl ; chmod +x ~/Desktop/formfind.pl
Downloading the Source Code
Before you can use formfind you need to download the sourcecode of the web page the form is on and save the code to a text file. You can do this with wget or cURL.
Using wget:
wget http://example.com > file.txt
Using cURL
curl http://example.com > file.txt
If wget and cURL are not installed on your computer, you can install them with:
sudo apt-get build-dep wget curl; sudo apt-get install wget curl
Using Formfind
Formfind finds forms in HTML pages. It writes its report to screen unless its output is redirected to a file or another script.
Formfind is run from the directory it is in by typing the command
sh /formfind.pl
If you run formfind from another directory you will need to specify the path to formfind after the “sh” command.
To pass source code from a file to formfind, type
sh /path-to-formfind/formfind.pl < /path-to-file/page.html
For example, if you had
- formfind saved to your desktop (formfind)
- The source code of a form page saved to your desktop (file.txt)
- A terminal open on your computer (Konsole)
You would use the following command to run formfind to find form data in file.txt
sh ~/formfind.pl < ~/Desktop/file.txt
Note that the tilde symbol (~) represents the active user’s home directory and the less than sign (<) tells Bash to feed formfind whatever data is in the file written after it.
Formfind’s output would look similar to this:
--- FORM report. Uses POST to URL "page-two.php" Input: NAME="Name" (TEXT) Input: NAME="Age" (TEXT) Input: NAME="Town" (TEXT) Input: NAME="Form_Submit" VALUE="Send" (SUBMIT) --- end of FORM
We will dissect that report in a few moments.
Using Formfind with wget and cURL to Fille Forms
Both wget and cURL grab the source code of any web page that doesn’t block them. cURL prints its output to the console screen and wget saves files to whichever directory it is being called from.
To use wget to save the source code from the page that holds the form, for example:
wget https://example.com > ~/Desktop/page.txt
To use cURL to do the same, you would use
curl https://example.com > ~/Desktop/page.txt
The greater than sign (>) is used to tell wget and cURL where to send their output.
You need to write two instructions to pass data to formfind:
- One to use wget or cURL download the webpage with the form
- One to pass the download page source to formfind
For example,
curl https://example.com > ~/Desktop/page-one.html sh formfind.pl < ~/Desktop/page-one.html
Both those commands can be put into one line by separating them with a semi-colon, like this
curl https://example.com > ~/Desktop/page-one.html ; sh formfind.pl < ~/Desktop/page-one.html
Using formfind is as easy as that.
Understanding Formfind’s Report
Take a look at this image:
It is the result of using cURL and Formfind command to collect information about a form.
It shows the following form details:
--- FORM report. Uses POST to URL "page-two.php" Input: NAME="Name" (TEXT) Input: NAME="Age" (TEXT) Input: NAME="Town" (TEXT) Input: NAME="Form_Submit" VALUE="Send" (SUBMIT) --- end of FORM
HTML Web forms work by letting a user click buttons, select items from drop down menus or type information into a text field and so on. What’s actually happening when someone completes a form is that he or she is assigning values to variables. That’s a complicated way of saying that the user is telling a server that “answer one = button one”, “answer two = drop down item three” and “answer four = some message”. The values of those variables is confirmed when the form filler clicks “send”. That information is then usually sent to another page which processes the form data.
The example form has three text fields shown in the screenshot:
- Name
- Age
- Town
The top line of the formfind report reads:
FORM report. Uses POST to URL "page-two.php"
This line tells us the form uses the “post” method to send data to “page-two.php”. This means the form data is processed by page-two.php.
The lines that begin “Input: Name=” tell us the titles for the fields in the form. Those titles are actually variables. In this case, PHP variables. The quoted texts are their names. For example, the first input line,
Input: NAME="Name" (TEXT)
tells us that the form has a field called “Name”. The text in brackets tells us it is a TEXT field.
To summarize that, formfind reads the HTML source of a web page to compile a report about any forms it contains. That report tells us the address of the page where the form sends data to be processed, the names of the variables that that page expects to receive, the type of information it expects those variables to hold and – for drop-down boxes and selection boxes – the possible values those variables can hold.
For example, using the screenshot,
- The test form is at https://journalxtra.com/tutorials/php-data-passing/page-one.html
- Its variables are Name, Age and Town
- Clicking “Send” sends those variables to https://journalxtra.com/tutorials/php-data-passing/page-two.php
The name of the “Send” button isn’t required. When we complete a form with wget or cURL, we send data directly to the page that processes the form. In effect, wget and cURL do the job of the send button.
The example URLs shown above no longer lead to real pages. I removed the pages when I migrated JournalXtra out of Hostgator and onto Namecheap.
Click here to learn how to make a web form.
Filling Forms with cURL
cURL can both download web pages and send data back to them; which comes in very handy for filling out web forms. The format we use is this:
curl -d [form-data] [web-page-that-processes-form]
The “-d” option is the one that tells cURL that it’s about to be fed data that needs to be sent to whatever URL is stated after the data.
To send MY name (Lee) to a form, I could write:
curl -d Name="Lee" https://example.com/page-two.php
To send my age (dare I reveal this!), I would write:
curl -d Age="36" https://example.com/page-two.php
To send all three of my name, age and town to page-two.php in one go, I would write:
curl -d Name="Lee" -d Age="36" -d Town="The Internet" https://example.com/page-two.php
That last command sends my answers to all three fields directly to the page that processes the form data (page-two.php).
It’s important to quote the answers to prevent cURL getting mixed up about what it’s being asked to do. For instance, If I put my town as “The Internet”, without the quotes cURL would assume the answer to “Town” is “The” before it tries to send the data to the host called “Internet”.
An alternative to quoting is to use percent-encoding of characters that hold instructional significance to cURL i.e find the UTF8 value of any special character and stick a percent sign in front of it e.g %20 is the space character.
When you complete a form with cURL, cURL will write the server response for the completed form to your screen. You can redirect it to a file with a greater than sign (>). For example,
curl -d Name="Lee" -d Age="36" -d Town="The Internet" https://example.com/page-two.php > ~/Desktop/form-response.html
Click here to learn more about form filling with cURL.
Filling Forms with wget
This works essentially the same as it does for cURL except the form data is expressed slightly differently. This time we use the format:
wget --post-data 'Name=Value' http://example.com
The differences you need to know about are:
- wget uses “–post-data” in place of cURL’s “-d” option
- wget combines multiple variable values with an ampersand (&)
For example,
wget --post-data 'Name=Lee&Age=36&Town=The%20Internet' https://example.com/page-two.php
Notice that the space used in “The Internet” is replaced with %20.
An ampersand used as part of a field answer would be replaced with %26.
For example
--post-data "Name=Me%20%26%20&Age=36..."
Note that cURL also accepts the lumping together of form data into one string with an ampersand such as curl -d “Name=Lee&Age=36…”.
Summary
You can use either cURL or wget to fill web forms without using a web browser. The method is to
- Download the form page with curl or wget
- Feed the downloaded page to formfind
- Examine formfind’s report to get the name of the page that processes the form
- Examine formfind’s report to get the name of the fields the form contains
- Use cURLor wget to send data to the processor page
For example, with cURL
- Download formfind to your desktop and make it an executable script
curl https://raw.githubusercontent.com/VR51/formfind/master/formfind.pl > ~/Desktop/formfind.pl ; chmod +x ~/Desktop/formfind.pl
- Download the form page to a file named page-one.html on your desktop
curl https://example.com/form.html > ~/Desktop/page-one.html
- Feed form.html to formfind
sh ~/Desktop/formfind.pl < ~/Desktop/form.html
- Examine the output
--- FORM report. Uses POST to URL "form-data.php" Input: NAME="Name" (TEXT) Input: NAME="Age" (TEXT) Input: NAME="Town" (TEXT) Input: NAME="Form_Submit" VALUE="Send" (SUBMIT) --- end of FORM
- Send a reply to the POST to URL page (e.g form-data.php) using the field titles shown and write the response to a file (e.g response.html) on your Desktop
curl -d Name="Lee" -d Age="36" -d Town="The Internet" https://example.com/form-data.php > ~/Desktop/form-response.html
I hope this guide helped you learn to use cURL, wget and formfind to complete web forms. Please let me know your results.
wpDiscuz