Differences

This shows you the differences between two versions of the page.

--- apache:extract_all_user_agents_from_apache_logs [2016/11/10 10:02] – peter
+++ apache:extract_all_user_agents_from_apache_logs [2023/07/17 12:10] (current) – removed peter
@@ Line 1: / Line 1: @@
-====== Apache - Extract all user agents from Apache logs ======
-<code bash>
-cat test.log | awk -F\" '{print $6}' | sort | uniq -c | sort -n
-</code>
-(where “test.log” is the access logfile you want to analyse).
-Returns
-<code>
-MetaURI API/2.0 +metauri.com
-Twitterbot/1.0
-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
-Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
-Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
-facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
-</code>
-The first number (bolded) is the amount of times this spider/crawler/user agent/ has accessed your site. Beware, these are not all crawlers, as the data is intermixed with actual human user traffic and other useful traffic.
-<WRAP info>
-In my example above, you see that the “Facebookexternalhit” user agent accessed the site 390,477 times per month. That is roughly 541x per hour. Excessive. On the kill list, you go!
-</WRAP>