What is a Web Crawler and how do I maximize my time with it?

You have probably heard this term thrown around the Internet a fair bit, along with other terms such as Web Spiders or Web Bots essentially these terms all relate to the same thing.

Web Crawlers are not creepy little web bugs that do your website harm, a Crawler is an automated software agent that crawls the content of your website and then sends this information to search engines so that they can position your website based on relevance to the users query or search.

How does it work?

When web crawlers visit a web page they crawl through all visible text and tags on the site and are also able to follow every link it can see through your website.

The search engine then takes the information that is gathered by the web crawler and determines from there what the website is about. This information is “indexed” then your website is added to the millions of other websites in a search engines database, it is then listed in the search engine ranking process according to its relevance.

Below are a few tips that will help when web crawlers visit your website

Make sure you are aware of duplicate content:

Having duplicate content is a great way to waste your websites time with the web crawlers, so to ensure that you are maximizing the time that you have with the web crawlers you will need to deal with content duplication properly. In most cases a 301 redirect that points all your duplicate content to the one page is the best solution. If the content is immensely similar you can also add a canonical tag, which will tell the crawler which page is more useful.

Make robots.txt files:

Robots.txt files tell the web crawlers what you don’t want them to index such as pages with content duplication or irrelevant folders. Creating robots.txt files is the more advanced side of SEO and you must be careful with it, creating incorrect robots.txt files could render your website un-crawlable.

Get a XML sitemap:

Remembering that Google is not exactly small so it may take time for Google to find your website and crawl it. This process can take quite a while if you have a new site and very little to no links pointing towards it. This is where the sitemap will come in handy, as you are able to submit your sitemap to Google, which will help speed up this whole process.

Google Webmaster Tools:

Adding your site to Google’s Webmaster Tools will allow Google to let you where your website has crawl or site errors. Google’s Webmaster Central will keep you up to date with information and will also be able to help with keeping your website search engine friendly.

Make sure you are sticking to the basics of on page SEO:

Considering there are over 250 factors that go into where you rank on Google it is the up most importance to use your basic on page tactics correctly:

• Make sure your H1 tag is a headline and contains your main keyword
• Change your internal linking so that your anchor text is relevant and that your internal linking structure runs smoothly throughout your website
• Relevant keywords are included in your image alt tags without keyword stuffing
• Your content has enough keyword density and is readable and avoid keyword stuffing.
• Page titles include important keywords at the start

Now you have the bare basics for making sure that your website is maximizing its time with the web crawlers.

Webfirm is a Melbourne-based specialist online marketing agency – we can optimise your website in the best possible way to improve your rankings. Talk to us today about any online marketing needs – phone us on (03) 9111 5386 or email us at info@webfirm.com.