How to find and Kill all ‘Zombie processes’ running on server? ~ AwsTechNix

Did you ever notice some processes with status “Z” on your server/system? 

 

These are Zombies. On this article I’m explaining the ways to find and kill all Zombies on the server.

A process is called a zombie process if the process has been completed, but its PID and process entry remains in the Linux process table. A process is removed from the process table when the process is completed, and its parent process reads the completed process' exit status by using the wait() system call. If a parent process fails to call wait() for whatever reason, its child process will be left in the process table, becoming a zombie. It almost always means that the parent is still around. If the parent exited, the child would be orphaned and re-parented to init, which would immediately perform the wait. In other words, they should go away once the parent process is done.

A zombie process doesn’t react to signals.

1. How can I get the Zombies from process list?

Its very simple! You can find out the Zombie processes in different ways:

# ps aux |grep "defunct"

khim  3366  0.0  0.0    0     0 ?        Z    07:34   0:00 [chrome] defunct 

khim  3435  0.0  0.0    0     0 ?        Z    07:44   0:19 [chrome] defunct 

khim  3722  0.0  0.0    0     0 ?        Z    08:21   0:00 [pidgin] defunct 

khim  4287  0.1  0.0    0     0 ?        Z    09:26   0:38 [chrome] defunct 

khim  5378  0.1  0.0    0     0 ?        Z    11:24   0:15 [chrome] defunct

# ps aux |grep Z

USER PID %CPU %MEM  VSZ   RSS TTY  STAT START   TIME   COMMAND

khim  3366     0.0     0.0        0        0       ?        Z     07:34     0:00      [chrome] 

khim   3435    0.0     0.0        0        0       ?        Z     07:44     0:19      [chrome]
khim   3722    0.0     0.0        0        0       ?        Z     08:21     0:00      [pidgin] 

khim   4287    0.1     0.0        0        0       ?        Z     09:26     0:38      [chrome]
khim   5378    0.1     0.0        0        0       ?        Z     11:24     0:15      [chrome]

2. How many Zombie processes running on your server? Juz count it out!

This is just to make a count of Zombie processes on the server. It can be done in different ways.

Please see some examples.

# ps aux | awk {'print $8'} | grep -c Z
5

# ps aux | awk '{ print $8 " " $2 }' | grep -wc Z
5

# ps aux | awk {'print $8'} | grep Z | wc -l
5

 

3. List the PID of Zombie?

# ps aux | awk '{ print $8 " " $2 }' | grep -w Z
Z 3366
Z 3435
Z 3722
Z 4287
Z 5378

In order to kill these processes, you need to find the parent process first.

# pstree -paul

See the sample output:

[root@vps ~]# pstree -paul
init,1
  |-crond,542
  |-dovecot,6576
  |   |-anvil,6577,dovecot
  |   |-config,25099
  |   `-log,6578
  |-httpd,5047
  |   |-httpd,1900,apache
  |   |-httpd,9428,apache
  |   |   |-php-cgi,1904,ctalk
  |   |   |-php-cgi,11989,ctalk
  |   |   `-php-cgi,11994,ctalk
  |   |-httpd,19203,apache
  |   |-httpd,22975,apache
  |   |-httpd,25197,apache
  |   `-httpd,30417,apache
  |-(kthreadd/3929,2)
  |   `-(khelper/3929,3)
  |-master,5227
........

This will show the pid of the of the parent of the zombie process. Now you need to kill the parent process or restart the service.

4. To find zombie processes with PPID(Parent PID) on Linux:

 

# ps axo stat,ppid,pid,comm | grep -w defunct 

Z    27698   3366 chrome <defunct>
Z    27698   3435 chrome <defunct>
Z    27758   3722 pidgin <defunct>
Z    27698   4287 chrome <defunct>
Z    27698   5378 chrome <defunct> 

The above command searches for processes with zombie (defunct) state, and displays them in (state, PPID, PID, command-name) format. The sample output shows that there is two zombie processes associated with "chrome & pidgin", and these were spawned by parent process with PIDs 27698 & 27758 respectively.

Killing zombie processes is not obvious since zombie processes are already dead. You can try two options to kill a zombie process on Linux as follows.

First, you can try sending SIGCHLD signal to the zombie's parent process using the kill command. Note that the above command gives you PPID (PID of parent process) of each zombie. In our example, PPID of the zombie are 27698 & 27758.

# sudo kill -s SIGCHLD 27698

# sudo kill -s SIGCHLD 27758

If a zombie process still does not go away, you can kill the parent process (e.g., 27698 or 27758) of the zombie.

# sudo kill -9 27698

# sudo kill -9 27758

Once its parent process gets killed, the zombie will be adopted by the init process, which is a parent of all processes in Linux. The init process periodically calls wait() to reap any zombie process.

That’s it! Try this and let me know if you’ve any questions. 😄

Load Testing Web Servers with Siege ~ AwsTechNix

Siege is an HTTP load testing and benchmarking utility that can be used to measure the performance of a web server when under duress. It evaluates the amount of data transferred, the response time of the server, transaction rate, throughput, concurrency, and times the program returned okay. Siege offers three modes of operation: Regression, internet simulation, and brute force.

This guide is for Debian or Ubuntu systems.

Download and Configure Siege

1. Prior to installing any new programs, update your system:

sudo apt-get update && sudo apt-get upgrade --show-upgraded

2. Download the latest version of Siege (3.0.9 at the time of this publication), which is always available at Siege’s website:

wget http://download.joedog.org/siege/siege-latest.tar.gz

3. Extract the program:

tar -zxvf siege-latest.tar.gz

4. Navigate to the Siege directory:

cd siege-*/

5. If the GNU Compiler Collection (gcc) is not installed, install now:

sudo apt-get install build-essential

6. Configure and complete the installation:

./configure

make

sudo make install

7. Generate a configuration file:

siege.config

8. Open the .siegerc file located in your home directory.

9. The suggested Siege configuration is for 25 concurrent users over a period of 1 minute. Set a location for your log file. Be sure to uncomment the variables shown below, and any other commented settings you want to use by removing the pound sign (#):

File Excerpt:- ~/.siegerc

...

        

#

# Variable declarations. You can set variables here

# for use in the directives below. Example:

# PROXY = proxy.joedog.org

# Reference variables inside ${} or $(), example:

# proxy-host = ${PROXY}

# You can also reference ENVIRONMENT variables without

# actually declaring them, example:

logfile = $(HOME)/siege.log

...

#

# Default number of simulated  concurrent users

# ex: concurrent = 25

#

concurrent = 25

#

# Default duration of the siege.  The right hand argument has

# a modifier which specifies the time units, H=hours, M=minutes,

# and S=seconds. If a modifier is not specified, then minutes

# are assumed.

# ex: time = 50M

#

time = 1M

You are now ready to run Siege!

Run Siege

To run Siege with the default settings, input the following command, replacing www.example.com with your domain name or IP address:

siege www.example.com

Siege outputs the results:

** SIEGE 2.70

** Preparing 25 concurrent users for battle.

The server is now under siege...

Lifting the server siege...       done.

Transactions:              2913 hits

Availability:              100.00 %

Elapsed time:            59.51 secs

Data transferred:             0.41 MB

Response time:        0.00 secs

Transaction rate:             48.95 trans/sec

Throughput:             0.01 MB/sec

Concurrency:               0.04

Successful transactions:        2913

Failed transactions:           0

Longest transaction:        0.01

Shortest transaction:        0.00

FILE: /var/log/siege.log

You can disable this annoying message by editing

the .siegerc file in your home directory; change

the directive 'show-logfile' to false.

If there are no failed connections and the availability remains at 100%, there are no problems.

Further Configuring and Commands

Creating a URL File

If you want Siege to hit a number of pages on your website at random, configure the program to read from a urls.txt file that lists the selected pages.

1. Open the urls.txt file generally created at /usr/local/etc/urls.txt. Add a list of URLs or IP addresses to that file:

File:- /usr/local/etc/urls.txt

# URLS file for siege

# --

# Format the url entries in any of the following formats:

# http://www.whoohoo.com/index.html

# http://www/index.html

# www/index.html

# http://www.whoohoo.com/cgi-bin/howto/display.cgi?1013

# Use the POST directive for pages that require it:

# http://www.whoohoo.com/cgi-bin/haha.cgi POST ha=1&ho=2

#      or POST content from a file:

# http://www.whoohoo.com/melvin.jsp POST </home/khim/haha

# http://www.whoohoo.com/melvin.jsp POST <./haha

# You may also set and reference variables inside this file,

# for more information, man urls_txt

# -------------------------------------------------------

www.example.com

www.example.org

123.45.67.89

2. To run Siege with this file use the siege command:

siege

If using a separate file, run:

siege -f your/file/path.txt

Commands

Siege features a number of command line options to use when you want to deviate from the default configuration but do not wish to edit the file.

• -c [num]: Set the number of concurrent users. Most web servers have less than a couple hundred users trying to access their website at the same time, so setting this to more than a few hundred is often not needed.
• -t [num]: Set a time limit for which Siege runs. Siege can run with the modifiers s for seconds, m for minutes, or h for hours. There should be no space between the number and the modifier (-t10s not -t10 s).
• -d [num]: Set the delay for each Siege user. Each user is then delayed for a random amount of seconds in between 1 and the set number. The default value is 3.
• -i: Used in conjunction with a URLs file, this causes each user to randomly hit one of the URLs, with no predetermined pattern. Similar to real life (the ‘i’ stands for “internet”), where you will not know where site visitors go, not all pages may be hit.
• -v: Verbose output. This outputs the results Siege gets in real time before printing the final results.
• -f [file]: Run Siege with a file containing a list of URLs that is not the default urls.txt file.
• -g [url]: Pull down the HTTP headers.
• -l: Generates a log file.
• -m "[message]": Include a message in the log file.
• -C: Outputs Siege’s current configuration profile.
• -V: Outputs Siege’s version information.
• -h: Outputs help information.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

Siege Home

Tuning Your Apache Server ~ AwsTechNix

Apache configuration has a major affect on your Linode’s performance. The easiest way to increase server performance is to turn off unneeded modules. This guide covers Apache modules, information on which modules to turn off, and other Apache performance tuning options.

The steps in this guide require root privileges. Be sure to run the steps below as root or with the sudo prefix. 

Tools

There are a variety of tools that can assist in determining if you need to alter resource settings, including the top command and the load-testing program Siege. Siege’s own Longview service can also help in server monitoring. At minimum, familiarize yourself with the RAM and CPU usage of your server. Discover usage statistics with these commands:

echo [PID]  [MEM]  [PATH] &&  ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -20

More specific resources for resource tuning Apache includes Apache mod_status and Apache2Buddy.

Apache mod_status

Apache mod_status diplays information related to incoming server connections by generating a detailed status page. View an example of this page at Apache’s own website.

1. Open your website’s configuration file. This file is located at /etc/apache2/sites-available/example.com.conf on Debian/Ubuntu systems or /etc/httpd/conf.d/vhost.conf on CentOS/Fedora systems.
2. Add the following to the <virtual_hosts> block:

   /etc/apache2/sites-available/example.com.conf (Debian/Ubuntu)

   /etc/httpd/conf.d/vhost.conf (CentOS/Fedora)

   
   

   <Location /server-status>  SetHandler server-status  Order Deny,Allow  Deny from all  Allow from localhost</Location>

   
   

3. Apache mod_status also offers an option called ExtendedStatus, which provides additional information about each request made to Apache. To enable ExtendedStatus edit your Apache configuration file:

   /etc/apache2/apache2.conf (Debian/Ubuntu)

   /etc/httpd/confd/httpd.conf (CentOS/Fedora)

   
   

   ExtendedStatus On

   Enabling ExtendedStatus consumes additional system resources.

4. Restart Apache:
• Debian/Ubuntu:

• service apache2 restart

• CentOS/Fedora:

• /bin/systemctl reload httpd.service

5. To view the file generated, download Lynx:
   • Debian/Ubuntu:

   • apt-get install lynx

   • Fedora/CentOS:

   • yum install lynx

6. Open the file:

7. lynx http://localhost/server-status

Apache2Buddy

The Apache2Buddy script, similar to MySQLTuner, reviews your Apache setup, and makes suggestions based on your Apache process memory and overall RAM. Although it is a fairly basic program, focusing on the MaxClients directive, Apache2Buddy is useful, and can be run through a single command:

curl -L http://apache2buddy.pl/ | perl

Multi Processing Modules

Before making any changes to your Apache configuration, be sure to back up the configuration file:

On Debian/Ubuntu:

cp /etc/apache2/apache2.config ~/apache2.conf.backup

On CentOS/Fedora:

cp /etc/httpd/conf/httpd.config ~/httpd.conf.backup

Apache offers two Multi-Processing Modules, three if on Apache 2.4, for managing your settings.

Each module creates child processes, differing primarily in how they handle threads.

Prefork

The prefork module creates a number of child processes at launch, each child handles only one thread. Since these processes deal solely with one thread at a time, making request speed suffer should there be too many concurrent requests. Should this occur, some requests essentially have to wait in line to be acted upon. To handle this, increase the number of child processes spawned, but be aware that this increases the amount of RAM being used. Prefork is the safest module and should be used when using mods that are not thread safe.

Worker

The worker module’s child processes spawn many threads per process, each thread ready to take on new requests. This allows for a greater number of concurrent requests to come in, and in turn is easier on the server’s RAM usage. Overall, the worker module offers higher performance, but is less secure than prefork and cannot be used with modules that are not thread safe.

Event

The event module is only available on Apache 2.4 and is based off the worker MPM. Like the worker, it creates multiple threads per child process, with a thread dedicated to KeepAlive connections that are handed down to child threads once the request has been made. This is good for multiple concurrent connections, especially those that are not all active at the same time but make the occasional request. The event MPM functions the same as worker in the event of SSL connections.

Module Values

Once you select your MPM, you will need to change the values inside the configuration. These settings are located in the /etc/apache2/apache2.conf file on Debian/Ubuntu, and the /etc/httpd/conf/httpd.conf file on CentOS/Fedora. The MPM looks like this:

/etc/apache2/apache2.conf (Debian/Ubuntu)

/etc/httpd/conf/httpd.conf (CentOS/Fedora)

<IfModule mpm_prefork_module>
 StartServers          4
 MinSpareServers       20
 MaxSpareServers      40
 MaxClients           200
 MaxRequestsPerChild  4500
</IfModule>

For other MPMs replace <IfModule mpm_prefork_module>  with <IfModule mpm_worker_module>  or <IfModule mpm_event_module> for worker and event, respectively.

The next step to reconfiguring your Apache server is altering the above settings. To do this, you need to be aware of what each value does, and how best to change it.

Again, the best way to make configuration changes is to make incremental changes and then monitor the effects.

After making alterations to the Apache configuration, restart the service using service apache restart on Debian/Ubuntu or /bin/systemctl reload httpd.service on CentOS/Fedora.

StartServers

The StartServers value indicates the number of child processes created at startup, and is dynamically controlled depending on load. There is often little reason to alter this number, unless your server is restarted frequently and contains a large number of requests upon reboot.

MinSpareServers

Sets the minimum number of idle child processes. If there are fewer processes than the MinSpareServer number, more processes are created at the rate of one per second on Apache 2.2 or lower. With Apache 2.4, this rate increases exponentially starting with 1 and ending with 32 children spawned per second. The benefit of this value is that when a request comes in it can take an idle thread; should a thread not be available, Apache would have to spawn a new child, taking up resources and extending the time it takes for the request to go through. Note, too many idle processes would also have an adverse effect on the server.

MaxSpareServers

Sets the maximum number of idle child processes. If there are more idle processes than this number, then they are terminated. Unless your website is extremely busy, this number should not be set too high, since even idle processes consume resources.

MaxClients

The maximum amount of requests that can be served simultaneously, with any number going past the limit being queued. If this is set too low, connections sent to queue eventually time-out; however, if set too high, it causes the memory to start swapping. If this value is increased past 256, the ServerLimit value must also be increased.

One way to calculate the best value for this is to divide the amount of RAM each Apache process uses by the amount of RAM available, leaving some room for other processes. Use ApacheBuddy to help determine these values, or the commands below.

To determine the RAM each Apache process uses, replace httpd with  apache2 on Debian or Ubuntu systems:

ps -ylC httpd --sort:rss

Divide the number by 2048 for megabytes.

To get information on memory usage:

free -m

To receive a fuller view of the resources Apache is using, use the top command.

MaxRequestsPerChild

This limits the number of requests a child server handles during its life. Once the limit has been hit, the child server dies. If set to 0, the child servers are set to never expire. The suggested value for this is a few thousand, to prevent memory leakage. Be aware that setting this too low can slow down the system, since creating new processes does take up resources.

ServerLimit

If you need to increase the MaxClients above 256, then increase your ServerLimit to match. To do this, add the ServerLimit line to your MPM code and alter the value:

ServerLimit          256

KeepAlive

The KeepAlive directive, when set to on allows for multiple requests to come from the same TCP connection. When a KeepAlive connection is used, it counts as only one request against the MaxRequestsPerChild directive. This value is kept outside of your MPM, but can tie in closely to your MPM choices.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

• Apache Performance Tuning
• Apache MPM Common Directives
• Apache 2.4 Documentation

WHAT & HOW TO CONFIGURE APACHE MULTI-PROCESSING MODULES (MPMs) ~ AwsTechNix

Are you worried about your Apache server performance? Okay, let’s talk about Apache Multi-Processing Modules (MPMs). There is a documentation for Apache MPM but who has got time to read the documentations. Let’s talk in simple plain English about Apache Multi-Processing Modules (MPMs). All you need is 15 mins to learn Apache MPMs(happy-face).

What

The MPMs are used to change the basic functionality of the web server. It’s possible due to Apache’s modular design. The MPM, or Multi-Processing Module, you use is responsible for just about the entire HTTP session. Starting from listening on the network, taking requests in and most importantly, how to handle those requests. With the MPM you use Apache’s behavior will change. Apache offers three MPMs to choose from; Prefork, Worker, and Event. You might be wondering which MPM module you should choose. The answer is right below.

How do I select which Apache MPM to use?

Above links explains about the three MPM modules and when to use them. If you don’t know about the Apache MPMs or which one to use, time to start reading.

How

1. Check what your Apache server has got

Most of the Apache server comes with Prefork module. To make sure whether your server has got Prefork, type the below command and see.

apache2ctl ­-l

See the output below

khimlal@devOps$ apache2ctl -­l 
Compiled in modules:
  core.c
  mod_so.c
  mod_watchdog.c
  http_core.c
  mod_log_config.c
  prefork.c
  mod_logio.c
  mod_version.c
  mod_unixd.c
  mod_unixd.c

If the Prefork Module is installed it should be shown under compiled in modules. prefork.c is shown on the list.

2. Let’s install Apache Worker MPM

Let’s configure Worker MPM. So time to install Worker.

apt-get install apache-mpm-worker

If you are willing to install Prefork or Event MPMs it should be as below

apt-get install apache-mpm-prefork

apt-get install apache-MPM-event

When the installation is completed type apache2ctl -l and see whether the prefork.c/worker.c shows up under “Compiled in modules”.

Note in Ubuntu you can only have one MPM module at a time. It means if you install Worker while the server has got Prefork, Prefork will be automatically removed. When you are switching MPMs it’s a good idea to backup your .conf files.

3. Let’s understand Apache MPM directives

Please read the comments below to understand about the each directive and what they do. Below configurations are extracted from apache2.conf

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
#
#
# worker MPM
# StartServers: initial number of server processes to start
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadLimit: ThreadsPerChild can be changed to this maximum value during a
#              graceful restart. ThreadLimit can only be changed by stopping
#              and starting Apache.
# ThreadsPerChild: constant number of worker threads in each server process
# MaxClients: maximum number of simultaneous client connections

# MaxRequestsPerChild: maximum number of requests a server process serves

If you need more information on directives checks the documentation.

4. Time to customize the Apache Worker directives as we need

ServerLimit            10

StartServers             2

MaxClients          100

MinSpareThreads  25

MaxSpareThreads 75

ThreadsPerChild   20

Before customizing the directives you need to understand how the directives work. Let me explain in plain English. The server will start 2 child processes which are determined by StartServers directive. Each process will start 20 threads which are determined by ThreadsPerChild directive so this means 2 processes can service only 40 concurrent connections/clients(i.e. 20×2=40). So what if more requests come in.

Now if more concurrent users come, then another child process will start, that can serve another 20 users. But how many child processes can be started is controlled by ServerLimit parameter, this means that in the configuration above, I can have 10 child processes in total, with each child process can handle 20 thread, in total handling 10×20=200 concurrent users.

But there is a problem, number defined in MaxClients is 100 here, this means that after 5 child processes, no extra process will start since we have defined an upper cap of MaxClients. This also means that if I set MaxClients to 500, after 10 child processes and 200 connections, no extra process will start and we cannot service more than 200 concurrent clients even if we have increased the MaxClient parameter. In this case, we need to also increase ServerLimit to 500/20 i.e. MaxClients/ThreadsPerChild=25

Okay now you know the directives and how they work, the problem is how to calculate the directives. Let’s jump into calculating directive values.

ps -ylC apache2 | awk '{x += $8;y += 1} END {print "Apache Memory Usage (MB): "x/1024; print "Average Proccess Size (MB): "x/((y-1)*1024)}'

You can use above shell script to determine an average amount of memory consumed by one Apache process. In addition to that, it’ll show the total amount of memory consumed by all Apache processes. Just unzip and execute with sh command. Accurate results will be shown when server is under heavy load.

The output

Apache Memory Usage (MB): 57.586
Average Process Size    (MB): 10.2

if in average, let’s assume that one Apache process consumes 50MB RAM and server has got RAM is 2048MB, and you want to leave 512MB for the rest of the processes, then:

MaxClients = (2048MB – 512MB)/10MB = 153.6 ~ 153

So that’s how you configure Apache Multi-Processing Modules. If you have any questions let me know in the comments below. Your feedback is highly appreciated(happy-face).