On occasions you may find that you need to stop a bot
(robot) from crawling your website, or maybe just part of your website. One way of doing this is the create an iRule
which blocks traffic based on the User-Agent which the bot advertises itself
as. An example of the User-Agent field
might be “DodgyRobot”:
when HTTP_REQUEST {
if { [HTTP::header "User-Agent"] contains "DodgyRobot" } {
drop
return }
}
Note that the “contains” operator looks for a substring so
as an example will actually catch:
"DodgyRobot/4.0" and
"AnotherDodgyRobot/mozilla"
Next we need the iRule to search against a list of
User-Agents that we stipulate. We do
this by defining a “class” or “datagroup” (both terms are synonymous
in F5 speak)
You do this by using the GUI to create a "string"
type datagroup named "userAgentsToBlock" that contains the list of
User-Agents to block:
DodgyRobot
NicosCustomUserAgent
MoreStuffToBlock
SpoofedUserAgent
AngryUserAgent
So our iRule will now look like this:
when HTTP_REQUEST {
if { [matchclass [HTTP::header "User-Agent"] contains $::userAgentsToBlock ] } {
drop
return }
}
Lastly apply the iRule to the VIP (virtual server) in question and you're done!
0 comments:
Post a Comment