understanding log files in linux…where to look when things go wrong

If you spend any amount of time working on Linux environments, either as an user or as an administrator then it is inevitable that you will need to check the log files at some point. It is quite possible that the system or an application is misbehaving or is not working and the best way to fix the issue would be to find out what is wrong by looking at the log files. Looking into the log files is usually the first step in troubleshooting an issue.

Linux has a log file for almost every part of the system, which allows you to see what might be wrong when something is not working correctly. There are separate log files for the kernel, the system, the desktop environment, the boot process, services etc. and also all applications. Almost all of the system wide log files are located in a single folder called /var/log/. This is the single location that you need to know and have access to in order to view log files on the system. You will need to have root or superuser level permission in order to view almost all of the log files.

important log files

Many of the files are consistent across all distros of Linux. However, there are still some variations and sometimes minor differences as to how some distros handle the naming and folder structure. Having said that, the following are some of the most common log files in the system.

  • /var/log/messages: This is probably one of the most important log file on your system. This is also probably one of the files to look into for most of the generic issues in your system (in addition to the syslog, if you have one). This file contains general system messages related to almost all parts of the core system. These include kernel, cron, mail, auth and daemon related messages.
  • /var/log/auth.log: This log file contains all the authentication and authorization related messages. This contains information about who logged in when and the authentication mechanisms that were used.
  • /var/log/boot.log: This contains information about the boot process. Although not available by that exact name in all distros, this file if it exist will contain messages from the last boot process.
  • /var/log/daemon.log: Your system run various different processes as daemons or services. The messages from such daemons are logged into this file. This will include messages from daemons such as Network Manager, netmount, dnsmasq etc.
  • /var/log/dmesg: During the boot process or kernel load, the system detects and identifies much of the hardware devices on your system. This includes several things such as the hard disk, memory, printers, webcams etc etc. All of these detection is also referred to as the kernel ring buffer information and is logged in the dmesg file. You can also view this information by using dmesg command tool.
  • /var/log/kern.log: As the name suggests, this file is used by the kernel to log its messages. This is best place to start if you are having issues with your custom built kernel.
  • /var/log/mail.log: If you use a local mail server, then the mail system logs its message to this file. In some distros, this could be a sub-folder often named as mail.
  • /var/log/sddm.log or gdm.log or kdm.log: Depending on the display manager that you use, the messages will be logged to one of these files. There are several different login manager you can use with your desktop such as sddm, kdm or gdm.
  • /var/log/syslog: This is very similar to the messages file and contains all messages except for the auth messages. This file is usually much more exhaustive than the messages files and will contain debug information as well as other log levels.
  • /var/log/Xorg.0.log: These are messages related to your Xorg server. If you having issues with your X server or graphical environment, then this is probably the file to look into.
  • /var/log/emerge.log and /var/log/emerge-fetch.log: This is very specific to the Gentoo distro. If you use Gentoo, then these files log all the information about the emerge process when install and uninstall different applications.

log files in the linux systems

In addition to the above log files, there are also several application specific log files that you can find the in /var/log/ folder. Obviously, these files exist only if you have these application installed and running. Some of the log files are located in subfolders that are specific to an application.

  • /var/log/apache2/ or /var/log/httpd/: If you run Apache web server on your system, then you could find the apache log files in one of these folders. The folder name itself will vary depending on your distro but is usually either apache2 or httpd.
  • /var/log/mysql/ or /var/log/mysqld.log: Again you can find your mysql database log messages in one these files. Many distros use the mysql/ folder to store the log files. Some older versions still use the single log file named mysqld.log
  • /var/log/cups/: If you use cups as your print system, then this is one file to check when you have printer related issues. All the printer and cups system logs are stored inside the cups/ folder.
  • /var/log/openconnect/: If you use openconnect to manage your VPN connection, then all the connection related messages are logged into this folder.
  • /var/log/portage: This is a Gentoo distribution related folder. Portage is the package manager for Gentoo, and you will see similar log files for the package manager on your system.

It is very likely that you will see other log files in your /var/log/ folder in addition to the ones mentioned here. This is dependent on what configuration you have on the system and also what different applications and services you have installed and running on that particular system.

viewing log files

Almost all log files are written out as plain text files. The exception being lastlog which is written out as data and can be viewed using the lastlog command utility. That means you can use any of your favorite file viewer tools to view these text log files. The most commonly used tools are usually command line utilities such as less, more, cat, tail, grep or zcat.

There is no reason why you cannot use a graphical editor to view these files, as long as you have the appropriate permissions to the files as well the X graphical interface. But, command line utilities usually work better as you can view them right on the console and even while logged in remotely into the server or machine.

In my opinion, the best tool to view the log files is tail. Usually, many of these log files can be very large with logs dating back to months or years which means loading or opening them with an editor is a memory intensive process. Also, it is very likely that the only logs that are of interest is the latest ones. tail is a command line utility that allows you to see the latest logs without being a memory hog.

In order to view the last 100 lines of log messages, you can use the following command:

$ tail -n 100 /var/log/messages

Sometimes you want to watch the log files as they are written to… in real time. Again, tail is a good tool to do that. The command line option -F allows you to monitor the log files. You can do that using the following command:

$ tail -F -n 100 /var/log/messages

Some distros provide a graphical interface that allows you to view system log files. The System Log File Viewer in Gnome desktop is one such tool if you don’t want to deal with command line tools. If you use KDE as your desktop then you can use KSystemLog to view your log files.