15 Command Line Tools for Monitoring Linux Systems

Do you need to monitor your Linux server’s performance? Most Linux distributions come equipped with many built-in monitoring tools. These tools allow you to retrieve information about system activities, and can be used to find possible causes for your server’s performance issues.

The commands discussed in this article are some of the most basic commands when it comes to system analysis and debugging server issues, such as discovering disk, CPU, memory and network bottlenecks.


1 – top – The Process Activity Command

The top command provides a dynamic, real-time view of the running system (i.e. actual process activity). By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds.

Commonly Used Hot Keys

Most Linux distributions come equipped with many built-in monitoring tools.

There are several useful hot keys used with the top command:

  • t — toggles summary information off and on.
  • m — toggles memory information off and on.
  • A — sorts the display by top consumers of various system resources. This is useful for quickly identifying performance-hungry tasks.
  • f — enters an interactive configuration screen for top. It’s helpful for configuring top for a specific task.
  • o — enables you to interactively select the order of the displayed fields.
  • r — issues the renice command.
  • k — issues the kill command.
  • z — toggles between color and monochrome.

2 – vmstat – System Activity, Hardware and System Information

The vmstat command reports virtual memory statistics: processes, memory, paging, block IO, traps and cpu activity.

Syntax:

# vmstat 3

Sample output:

     procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------   
  r b swpd free buff cache si so bi bo in cs us sy id wa st    
 0 0 0 2540988 522188 5130400 0 0 2 32 4 2 4 1 96 0 0  
 1 0 0 2540988 522188 5130400 0 0 0 720 1199 665 1 0 99 0 0  
 0 0 0 2540956 522188 5130400 0 0 0 0 1151 1569 4 1 95 0 0   
 0 0 0 2540956 522188 5130500 0 0 0 6 1117 439 1 0 99 0 0    
 0 0 0 2540940 522188 5130512 0 0 0 536 1189 932 1 0 98 0 0   
 0 0 0 2538444 522188 5130588 0 0 0 0 1187 1417 4 1 96 0 0    
 0 0 0 2490060 522188 5130640 0 0 0 18 1253 1123 5 1 94 0 0

Display memory utilization slabinfo with the following command:

# vmstat -m

And you can retrieve information about active and inactive memory pages with:

# vmstat -a

3 – w – Logged In Users

The w command displays information about the currently logged in users and their processes. Its syntax is:

     # w [user]

Running this command gives you output similar to the following:

 17:58:47 up 5 days, 20:28, 2 users, load average: 0.36, 0.26, 0.24  
 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT    
 root pts/0 10.1.3.145 14:55 5.00s 0.04s 0.02s vim /etc/resolv.conf  
 root pts/1 10.1.3.145 17:43 0.00s 0.03s 0.00s w

4 – uptime – System Uptime

The uptime command not only displays the amount of time the server has been running, but the current time, how many users are currently logged on and the system load average for the past 1, 5, and 15 minutes.

# uptime

Its output will look similar to:

11:57:40 up 96 days, 17:23,  
0 users, 
load average: 1.17, 1.15, 1.14

5 – ps – The Processes

The ps command reports a snapshot of the current processes. To select all processes, use the -A or -e option.

# ps -A

Running this command on your system will provide similar results to the following:

PID TTY         
TIME CMD     
6538 ?        
00:00:00 anytermd     
6543 pts/4    
00:00:00 bash     
6855 pts/4    
00:00:00 ps

To show long format output, add “l” at the end:

# ps -Al

Turn on extra-full mode to show the command line arguments passed to the processes by adding “F”:

# ps -AlF

Add an “H” to see threads (LWP and NLWP):

# ps -AlFH

To see threads after processes, add “m”:

# ps -AlLm

Or print a process tree:

# ps -ejH     
# ps axjf    
# pstree

Display only the process IDs of lighttpd:

# ps -C lighttpd -o pid=

Or:

# pgrep lighttpd

You can also find the top ten processes using the most CPU with:

# ps -auxf | sort -nr -k 3 | head -10

6 – free – Memory Usage

The free command displays the total amount of physical and swap memory in the system, as well as the buffers used by the kernel.

# free

This simple command’s output looks like:

 total used free shared buffers cached    
 Mem: 12302896 9739664 2563232 0 523124 5154740   
 -/+ buffers/cache: 4061800 8241096     Swap: 1052248 0 1052248

7 – iostat – Average CPU Load and Disk Activity

The iostat command reports CPU and input/output statistics for devices, partitions and network filesystems (NFS).

# iostat

And the obligatory sample output:

Linux 3.4.5-hardened-v3 (tryit)      
12/14/12
_i686_  (1 CPU)       
avg-cpu:  %user   %nice  %system %iowait  %steal  %idle                
2.47    6.52    2.54    1.57    0.00   86.90          
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn

8 – sar – Collect and Report System Activity

The sar command collects, reports and saves system activity information. To see the network counter, enter:

# sar -n DEV | more

To display the network counters from the 24th:

# sar -n DEV -f /var/log/sa/sa24 | more

You can also display real time usage using sar:

# sar 4 5

Sample output:

     Linux 3.4.5-hardened-v3 (tryit)         
12/14/12        _i686_  (1 CPU)      
12:45:25        CPU     %user     %nice   %system   %iowait    %steal     %idle     
12:45:29        all      0.43     19.70      1.07      0.00      0.00     78.80     
12:45:33        all      0.43     19.35      1.51      1.51      0.00     77.20    
12:45:37        all      0.43     18.49      1.94      0.00      0.00     79.14    
12:45:41        all      0.43     19.02      2.14      0.00      0.00     78.42   
12:45:45        all      0.65     18.49      2.37      0.00      0.00     78.49   
Average:        all      0.47     19.01      1.80      0.30      0.00     78.41

9 – mpstat – Multi-processor Usage

The mpstat command displays each available processor’s activities, with processor 0 being the first. Use the following command to display the average CPU utilization per processor:

# mpstat -P ALL

And this is the output:

     Linux 3.4.5-hardened-v3 (tryit)      
12/14/12        _i686_  (1 CPU)     
12:47:46     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest    %idle     
12:47:46     all    2.47    6.52    1.48    1.57    0.00    1.06    0.00    0.00    86.90    
12:47:46       0    2.47    6.52    1.48    1.57    0.00    1.06    0.00    0.00    86.90

10 – pmap – Process Memory Usage

The pmap command reports the memory map of a process. Use this command to find the causes of memory bottlenecks.

# pmap -d PID

To display process memory information for pid # 47394, enter:

# pmap -d 47394

Sample output:

     47394: /usr/bin/php-cgi    
 Address Kbytes Mode Offset Device Mapping  
 0000000000400000 2584 r-x-- 0000000000000000 008:00002 php-cgi  
 0000000000886000 140 rw--- 0000000000286000 008:00002 php-cgi  
 00000000008a9000 52 rw--- 00000000008a9000 000:00000 [ anon ]   
 0000000000aa8000 76 rw--- 00000000002a8000 008:00002 php-cgi   
 000000000f678000 1980 rw--- 000000000f678000 000:00000 [ anon ]   
 000000314a600000 112 r-x-- 0000000000000000 008:00002 ld-2.5.so    
 000000314a81b000 4 r---- 000000000001b000 008:00002 ld-2.5.so    
 000000314a81c000 4 rw--- 000000000001c000 008:00002 ld-2.5.so   
 000000314aa00000 1328 r-x-- 0000000000000000 008:00002 libc-2.5.so 
 000000314ab4c000 2048 ----- 000000000014c000 008:00002 libc-2.5.so   
  .....     ......     ..    
 00002af8d48fd000 4 rw--- 0000000000006000 008:00002 xsl.so   
 00002af8d490c000 40 r-x-- 0000000000000000 008:00002 libnss_files-2.5.so  
 00002af8d4916000 2044 ----- 000000000000a000 008:00002 libnss_files-2.5.so   
 00002af8d4b15000 4 r---- 0000000000009000 008:00002 libnss_files-2.5.so  
 00002af8d4b16000 4 rw--- 000000000000a000 008:00002 libnss_files-2.5.so 
 00002af8d4b17000 768000 rw-s- 0000000000000000 000:00009 zero (deleted)  
 00007fffc95fe000 84 rw--- 00007ffffffea000 000:00000 [ stack ] 
 ffffffffff600000 8192 ----- 0000000000000000 000:00000 [ anon ]  
 mapped: 933712K writeable/private: 4304K shared: 768000K

The last line is very important:

 mapped: 933712K total amount of memory mapped to files writeable/private: 4304K the amount of private address space 
shared: 768000K the amount of address space this process is sharing with others

11 – netstat – Network Statistics

The netstat command displays both incoming and outgoing network connections, routing tables and a number of network interface statistics. It is available on Unix, Unix-like systems, and Windows NT-based operating systems.

 # netstat

12 – iptraf – Real-time Network Statistics

The iptraf command is an colorful, interactive IP LAN monitor. It is an ncurses-based IP LAN monitor that generates various network statistics including TCP info, UDP counts, ICMP and OSPF information, Ethernet load info, node stats, IP checksum errors and more. It can provide the following info in easy to read format:

  • Network traffic statistics by TCP connection.
  • IP traffic statistics by network interface.
  • Network traffic statistics by protocol.
  • Network traffic statistics by TCP/UDP port and packet size.
  • Network traffic statistics by Layer2 address.

13 – tcpdump – Detailed Network Traffic Analysis

The tcpdump is a simple command that dumps a network’s traffic. You need, however, a good understanding of the TCP/IP protocol in order to use this tool. For example, to display traffic info about DNS, enter:

 # tcpdump -i eth1 'udp port 53'

To display all IPv4 HTTP packets from port 80 (i.e. print only packets that contain data; not, for example, SYN abd FIN packets and ACK-only packets), enter:

 # tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

To display all HTTP sessions to 192.168.1.5:

# tcpdump -ni eth0 'dst 192.168.1.5 and tcp and port http'

14 – strace – System Calls

The strace command traces system calls and signals. This is useful for debugging the webserver and other server problems.

The following command runs strace against /bin/foo and captures its output to output.txt:

 # strace -o output.txt /bin/foo

15 – /proc – Various Kernel Statistics

The /proc file system provides detailed information about various hardware devices and other Linux kernel information. Common /proc examples are:

     
# cat /proc/cpuinfo     
# cat /proc/meminfo     
# cat /proc/zoneinfo    
# cat /proc/mounts

Conclusion

And there you have it: fifteen useful commands that let you monitor different aspects of your Linux system. Naturally, these commands are only a small subset of the many that Linux provides. But for day to day operations, they are usually enough.

Have a favorite command? Let’s keep the conversation going within the comments area.

Nettuts+

Leave a comment

Your email address will not be published.