Earlier today, I gave the following talk at Kahena Digital Marketing‘s KahenaCon Advanced conference in Tel Aviv for log analytics software company Logz.io. Here is my presentation with notes below!

2 – Your server log files contain the only information that is 100% in terms of how people, Google, and anyone is interacting with your website. If you want to get the most insights into your web server data, then you need to analyze your log files.

3 – Servers, applications, networks, and more all output log entries whenever they perform an action. Servers output log entries whenever a human or bot requests a page, image, script, or anything else on a web server. If a website receives 100,000 visitors every day, and each visitor goes to ten pages, then there will be one million log entries in that day’s log file.

No one has the time to go through that. So, system administrators and DevOps engineers tend to look at log files only in response to something — a bag, a hack, a compliance issue, or something like that. However, technical SEOs can get a lot of information from log files if they are proactive about it.

4 – The linked article in Moz suggests that people export a batch of log files over a period of time to Excel for analysis. However, that practice lets one see only a moment in time rather than trends. So, it’s crucial to use log analysis software (if one cannot or does not want to set up and manage such an IT environment). There are many different types for different types of companies.

5 – Bot crawl volume is something to see in log files. If you want to get found in search in Russia and Yandex stops crawling your website, then that is a problem.

6 and 7 – Technical SEOs are knowledgable on response code errors. Log files will show the full picture.

8 – Google assigns a crawl budget to every website, and if Google hits its limit before crawling your new content or products then that is a problem. You want to reduce wasted crawls by looking at the log files to be sure that advertising landing pages, internal scripts (outside of CSS and Javascript), and other items are not be crawled.

9 – Duplicate URL crawling as a result of URL parameters (from marketing or navigation) is also a problem. Parameters can be blocked in Google Webmaster Tools.

10 – Crawl priority is also important. By grouping log files into batches based on directory, subdomain, website section, or more, you can see which parts of your website gets the most and least attention from Google. Ideally, the parts that are updated the most often with new content and products should get the most attention. This can be addressed in your XML sitemap.

11 – You can also see the last crawl date of any specific page, photo, and more.

12 – You can learn the overall crawl budget by seeing how many times Googlebot requests something over a given period of time. Hopefully, this number will increase over time!

13 – To get all of the specific information on these uses, see my essay on Logz.io here!