--

I don't, because I think it's interesting to see which bots crawl the website too.

However, most bots make their presence known in their user agent, for example this is from a google bot: "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/125.0.6422.66 Safari/537.36".

You can easily get lists from the internet with common bot user agents and filter the results with them.

--

--

Christian Behler
Christian Behler

Written by Christian Behler

M. Sc. Computer Science and Physics, Indie Game/Software/Web Developer, Writer, 3D Artist, and too many other interests. https://pingpoli.medium.com/membership

No responses yet