Usually kernel processes are safe and clean. For kernel processes, either PID (process id) is 2 or PPID (parent process id) is 2. Here is how to get all non-kernel processes.

ps --ppid 2 -p 2 -p 1 \
   --deselect -o uid,pid,rss,%cpu,command

Returns:

UID   PID   RSS %CPU COMMAND
   0   411  1848  0.0 /lib/systemd/systemd-
   0   572  2904  0.0 dhclient -1 -v -pf /r
 102   902  1244  0.0 dbus-daemon --system
   0   912  1948  0.0 /lib/systemd/systemd-
   0  5869   388  0.0 upstart-socket-bridge
 200  1953   904  0.0 /usr/sbin/apache2 -k
 200  3463  3700  0.0 /usr/sbin/apache2 -k
  ...  ...
  ...  ...
   0  5098  4224  0.0 sshd: ubuntu [priv]
   0  5139  1748  0.0 /usr/bin/python /usr/
 200  5140  3484  0.0 /usr/bin/python /usr/
 200  5176  1904  0.0 sshd: ubuntu@pts/3
 200  5177  3860  0.0 -bash
 200  5193  1200  0.0 tmux attach -t denny
   0  5297  4224  0.0 sshd: ubuntu [priv]
  ...  ...
  ...  ...

NOTE:

• rss (resident set size): real RAM usage.
• -deselect: rule out matched processes.

We may have many processes running, which are expected and trusted. e.g apache2, tomcat7, mysqld, etc. To avoid distraction, build a white list especial for your project.

We’re more concerned about suspicious processes using noticeable resource.

# Sort by memory first, then cpu
ps --ppid 2 -p 2 -p 1 --deselect \
  -o uid,pid,rss,%cpu,command, \
  --sort -rss,-cpu

We hide all the complexities and white list configuration in a python script (detect_suspicious_process.py). If you issue the python command, you may see output like “Identified processes count: XXX.” Define a scheduled task to run periodical check and confirm the number.

If the number is not 0 or it changes, send alerts. It might take a while to build a suitable white list. Once it’s done, your servers are always more secured and managed!

wget -O /tmp/detect_suspicious_process.py \
https://raw.githubusercontent.com/\
DennyZhang/devops_public/tag_v2/python/\
detect_suspicious_process/\
detect_suspicious_process.py
 
# Detect suspicious process
python /tmp/detect_suspicious_process.py
 
# Detect by customized whitelist
python /tmp/detect_suspicious_process.py \
   --whitelist_file /tmp/whitelist.txt