Tracking Web Spiders / Search Engine vists to your site.
Aug 7, 2001
Search engine traffic is some of the most important traffic that your
website gets for a whole host of reasons. First off, the majority of web
servers start their quest for information at a search engine. Having a
high placement in the search engines will generate a lot of traffic to your
website. This is why there are so many Search Engine Optimization companies
out there. Another reason why search engine traffic is so important to your
site is that it's free. Advertising is expensive, even at the dirt cheap
banner ad prices these days. (Banner ads aren't working anymore anyhow.)
So you've gone out and spent lots of time and/or money on getting the
pages of your website optimized for the search engines. The next question
is when will the search engines pick up your modifications? That's a waiting
game and like any waiting game, it can be nerve wracking. Patience isn't my
strong suit either. One option is to pay for a frequent scan of your website.
AltaVista has an Express Inclusion
Service that guarantees weekly scans of your website. This is probably
the way to go if you want your weekly/monthly specials to be in the search
engine databases. However if you're a small site like us, then your site
doesn't change to often and it probably isn't worth the money.
One little thing I do to make the wait easier is to track and log any
visits from web spiders or crawlers to my website. Knowing that Goodle
or Fast came through the site recently is a help. How is this done?
What I do is to create a PHP page that will send me email when it is
requested from the server. I include a bunch of server variables in that
email, such as REMOTE_ADDR and HTTP_USER_AGENT which tells me the IP address
and the browser type of whatever is requesting that page. Most spiders
are good enough to announce their identity via the HTTP_USER_AGENT variable.
The only link to this page is hidden on my site by a single pixel transparent
gif. Human surfers stand a slim chance of finding this link, but any spider
that is looping through all of the links on that page will easily locate it.
I'm using PHP to implement this, it would be just as easy to do it in ASP,
Perl, ColdFusion or whatever scripting language you have available. Automate
like crazy and stay on top of what's happening to your website.
If you have any comments or questions, please feel free to drop Rene an
email at firstname.lastname@example.org. If
you would like assistance in setting up this kind of feature on your website,
please contact us for a quote.