When troubleshooting issues with your Linux server one of the most useful resources at your disposal is the humble log file, recording your system’s activity and any problems it’s encountered. The following guide shows the basics of accessing the logs your server has created.

By default most Linux logs can be found in the directory /var/log/. To take a look at the logs navigate to that directory and and list the files it contains. Most logs in that directory will be readable by the superuser only, so you may need to use sudo or be root to view them.

System log directory

As you can see in the above example there are all sorts of logs – some for specific applications, others for certain events.

Some of the common log files and directories you might see in /var/log:

Which of those logs you’ll see on your system will depend on the applications you have installed and running.

Searching configuration files

If you are unsure which log file has the information you need, take a look at the configuration file for the service you are troubleshooting. The config will usually have an entry that defines where the application keeps its log.

Two easy ways to search a file are the less and grep commands.

less

You can use the command less to browse and search files. The less command allows you to scroll through a service’s configuration file and highlight all the occurrences of a search term. This can be helpful because it allows you to the see each result in situ. To use less type:

$ sudo less [path to your file]

This will display the contents of the file and allow you to scroll through it using the cursor keys. To search for text, type a “/” followed by your search term then hit enter. You can move through matches by hitting “n” for the next one and “N” for the previous match.

grep

To use grep, pass it the search term and the file to be checked and grep will return a list of lines containing the search term. For a general search like this it’s also a good idea to use the “-i” option, which causes the search to be case-insensitive.

As an example, to search the sshd config file for “log” you could run the command:

$ sudo grep -i 'log' /etc/ssh/sshd_config

The result might look like:

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO
#LoginGraceTime 2m
PermitRootLogin no
# the setting of "PermitRootLogin without-password".
#PrintLastLog yes
#UseLogin no

The above example returns several matching lines. In most config files lines starting with a “#” symbol are comments, so they won’t be the ones controlling where the log is located. In this case, the line reading “SyslogFacility AUTHPRIV” indicates that the sshd service is using syslog.

Syslog

Syslog is a standard used by many applications that defines how messages are logged. There will be a syslog-compatible service running on your server that implements the standard.

Some applications, such as the apache web server, do not use the syslog service and instead manage their own logging. If applications implement their own logging their configuration files will say where to find the log files.

When an application uses syslog it references a “facility” that tells syslog where to categorize that application’s log entries. In the grep example we found that sshd was logging messages to the authpriv syslog facility. To see where authpriv messages are stored we need to check the configuration file for the syslog daemon.

Syslog variants

There are several implementations of the syslog standard, including syslog-ng, sysklogd and rsyslogd. These applications are examples of the message logging software used by Linux, and their configuration files can be found in the /etc directory.

Many systems use sysklogd to provide syslog functionality. Its configuration file can be found at /etc/syslog.conf.

Some other distributions use rsyslogd. Depending on the version, its configuration files will be in either /etc/rsyslog.confor /etc/rsyslog.d/ (the default configuration file in that directory is /etc/rsyslog.d/50-default.conf).

Still other distributions might use syslog-ng, which stores its config file at /etc/syslog-ng.conf.

Once you find your syslog config file you can search for the facility entry to see where those entries are logged.

$ sudo grep 'authpriv' /etc/syslog.conf
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

Checking syslog.conf in that example shows us that authpriv messages, and therefore sshd messages, are stored in/var/log/secure (the “authpriv.*” line).

Searching log files

One you have found the log file you can use grep and less to review it. There is another tool that is commonly used when looking at logs called “tail”. If tail is used to view a file it will, by default, show the last 10 lines of the file. This is useful because new log entries are appended to the end of the file. Another way to use tail is with the “-f” (follow) option:

$ sudo tail -f /path/to/log

This gives you a real-time view of the end of the log file so that when new entries are added to the log you automatically see them on-screen. This means that if you replicate your issue you will instantly be able to see any entries this adds to the log file.

When looking at log files you will usually find that each entry is timestamped. This can help with narrowing down the problem to a particular entry, or correlating that error with entries in other application logs. You can also play around with different searches using grep and less to locate any useful messages.

You may find you want to review logs older than are in the current log file. Old logs files are appended with a ‘.’ followed by a number and may also be compressed, in which case they will end with ‘.gz’.

On some systems the number will show the data the log was rotated (in a “yyyymmdd” form) or will just number the additional logs with 1, 2, 3, etc.

$ ls /var/log
audit                 httpd                 secure
boot.log              lastlog               secure-20111016.gz
boot.log-20111016.gz  maillog               secure-20111023.gz
boot.log-20111023.gz  maillog-20111016.gz   secure-20111030.gz
boot.log-20111030.gz  maillog-20111023.gz   secure-20111106.gz
boot.log-20111106.gz  maillog-20111030.gz   spooler
btmp                  maillog-20111106.gz   spooler-20111016.gz
btmp-20111101.gz      messages              spooler-20111023.gz
cron                  messages-20111016.gz  spooler-20111030.gz
cron-20111016.gz      messages-20111023.gz  spooler-20111106.gz
cron-20111023.gz      messages-20111030.gz  tallylog
cron-20111030.gz      messages-20111106.gz  wtmp
cron-20111106.gz      mysqld.log            yum.log
dracut.log            prelink

If the files are compressed an easy way to view them is to use zless or zgrep. These can be used on compressed files in the same way less and grep were used on uncompressed files in our previous examples.  If you don’t have zless or zgrep on your system you can also run “gunzip filename.gz” to uncompress the archived log.

Using the information you find

Once you have identified the error message(s) that relate to your service you can search the Rackspace Knowledge Center or use your favorite search engine to find the cause.

You can often find information by running a search for the error message itself. If you use Google or another search engine you can also put the error message in quotes to help narrow the results to pages that only mention that specific error message.

Views: 5