The Sitemap is the Holy Grail of a website. It’s the sheet (or sheets) of xml that new webmasters don’t know to use and some experienced webmasters neglect to create. Consider that every website has a front, a back, a mouthpiece, a gang of security guards and a guide. Visitors see the front, the webmaster uses the backend to create the front, the RSS feed tells the world what’s happening at the website, robots.txt and other little bits help protect it, and the sitemap guides search engine spiders around the it.
Usually, if you use a content management system (CMS) you will be blessed with automatic sitemap generation either through an inbuilt process or a plugin. In which case, you only need to locate it, submit it to search engines, link to it from your index page or the footer of every page, and regularly ping it to tell search engines about updates to it. You will usually find your sitemap sitting comfortably close to your robots.txt at the root of your domain e.g. your-domain.com/sitemap.xml
If you are not blessed with automatic sitemap generation and submission then you will need to create your own sitemap. Of course, that is what this article is all about and below here are the instructions your should follow to do that.
Most often, a sitemap needs to be manually created when a website is hand crafted in (x)html or when a sitemap is to be remotely hosted on a different domain or server to the website it maps (frequently the case when a sponsor provides a co-brand or white label site but not enough space or facility to host a sitemap).
There are programs and scripts that can be used to generate sitemaps. These scripts can be split into two categories: those that work and those that don’t work. Pedants might point out that a third category exists which includes those that only work when they feel like it or after a lot of flirtatious smooth-talking, as is often the case, but I’m not going to discuss those ones.
Those sitemap generators that do work can be divided into two subcategories:
- those that run from a desktop PC
- those that run from a web server
And they may be subdivided into paid and free sitemap makers. Guess which ones we’re going to work with?
Most of the free sitemap tools that work from a desktop PC are the same ones used to check for dead links. More often than not the “sitemaps” created by those programs need to be manually edited into the xml sitemap format, for example, the URLs
<?xml version="1.0" encoding="UTF-8"?>
<!-- The site URLs go below here -->
I’ve created a Bash script which automatically converts links into the xml sitemap format. The script is available for free with Scriptilitious. You can download it at the end of this article.
You can map your website with klinkstatus (or similar) and use Scriptilitious to convert the indexed URLs into an xml sitemap. There is a a rumor that klinkstatus will soon have a specific template for xml sitemap creation which is good news for webmasters who use Linux (like me).