Skip to main content

Auto Recovery of Http After Crash or Hang


In very popular and high traffic sites, it is very often to face httpd hangs. In these cases only manual restart is the only solution. I would like to share a script for the solution of this problem…
First lets write a script for checking the system state, if it is alarming, we will fire our recovery script with a mail script attached with it…
Let’s say we are going to keep our script at /usr/local/recover folder and the script name is recover.sh. Issue the below command first
# cd /usr/local
# mkdir recover
# cd recover
# vi recover.sh
Next paste the below script in the file and save it…
#!/bin/sh
monitor=`cat /proc/loadavg | sed ’s/./ /’ | awk ‘{print $1}’`
if [ $ monitor -gt 10 ]

then
/usr/local/recover/recoveraction.sh
fi
In the second line we are getting the load average of the server in 5 minuites. If it is greater than 10, we are fireing the script named recoveraction.sh
Now give this recover.sh file 755 permission by the below command…
# chmod 777 recover.sh
Now we need to write the action file. i.e recoveraction.sh
Open a new file in the same folder by this command…
# vi recoveraction.sh
and paste the below command in it…
monitor=`cat /proc/loadavg | sed ’s/./ /’ | awk ‘{print $1}’`
service httpd restart
ADMIN=mailid@example.com
host=`hostname`
res=`top -bn 1`
/usr/sbin/sendmail -f $ADMIN -t << sendemail
To: $ADMIN
Subject: High load on [$host]
Load Average : $monitor
The above script will restart the httpd service and send the admin a mail with all details.
Though tuning the server and making the application much optimized will solve your problems better, but placing this script may save you from some sleepless nights…

Comments

Popular posts from this blog

Shell Script to check tomcat status and restart

The below script checks the status of a particular tomcat status and restarts it if the tomcat does not respond....TOMCAT_HOME=/usr/local/tomcat-folder/

is_Running ()
{

        wget -O - http://yourserver.com/ >& /dev/null
 if( test $? -eq 0 ) then
  return 0
 else
  return 1
 fi
}


kill_Hanged_Processes ()
{
 echo "killing hanged processes......"
 javaProcs=`ps -efl| grep -v grep | grep java`
 if(test ! -z "$javaProcs") then
  echo "nonzero"
  processId=`echo $javaProcs | awk '{ print $2} '`
  echo "$processId"
  kill -9 $processId
 fi
}
stop_Tomcat ()
{
 echo "shutting down......"
 $TOMCAT_HOME/bin/shutdown.sh
}

start_Tomcat ()
{
 echo "starting......"
 $TOMCAT_HOME/bin/startup.sh
}

restart ()
{
 stop_Tomcat
 sleep 10
 kill_Hanged_Processes
 start_Tomcat
 sleep 60
}

send_Mail ()
{
#!/bin/bash
# script to send simple email
# email subject
SUBJECT="Telegraphindia.com went down"
# Email To ?
EMAIL="mailid1@gmail.com,mailid2@gmail.com"
# Em…

Getting access to menu from portlet in liferay

The below code describes the way of accessing menu items from the portlet. Here i have called it from jsp. The best thing about this is you can use the beautiful permission management of menu items of liferay as well without bothering about that. Writing services for fetching menu names from the layout tables will need more codes for permission managements......

Hope this helps.........


<%--
/**
* Copyright (c) 2000-2010 Liferay, Inc. All rights reserved.
*
* This library is free software; you can redistribute it and/or modify it under
* the terms of the GNU Lesser General Public License as published by the Free
* Software Foundation; either version 2.1 of the License, or (at your option)
* any later version.
*
* This library is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
* details.
*/
--%>
<%@ page import="com.li…

Virus f Opyum Team

Symptoms :: 
Several processes running named f or i. 
Network choked
Apache Dead
Unknown entry  * * * * * root f Opyum Team in crontab

Investigate ::
Try to kill the f process and even root will be unable to kill it. Check file named f and i in /etc or /bin or /usr/sbin directory. Try to delete, the file f  will not delete.  Some unknown .jpg files will be there in the same folder. 

Solve :: The root is unable to delete the file f because of the immutable bit in f
To remove it use ses or chattr
Follow the below steps to remove it completely.....



remove the unknown .jpg files in system folders

#top
(kill process f) option k

#ses -i /bin/f
#rm /bin/f
#ses -i /etc/crontab
#vi /etc/crontab (and delete last line)
#reboot


Precautions ::  In my case the server was hacked bu Brute-force attack
So set a good password with a combination of upper,lower case letters and special characters.

Post your feedback or any issues you may face removing the virus.  Very less number of good threads are there explaining …