May 30, 2024
I don't, because I think it's interesting to see which bots crawl the website too.
However, most bots make their presence known in their user agent, for example this is from a google bot: "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/125.0.6422.66 Safari/537.36".
You can easily get lists from the internet with common bot user agents and filter the results with them.