Tuesday, January 25, 2011

Reset the NVRAM

Sometimes when you upgrade or reboot a server for any maintenance you will find that the system is not able to locate devices, OS or other stuff... a good try to recover from this failure that have no mayor impact than loose the latest state of the hardware NVRAM is the following... will save you some headaches

     Access the "service procesor" when the servers is off showing "OK" @ the panel
      Press any key and write this code... 85763
   
      when the panel is back in "OK"    Press "Intro" you will see a warning ... ignore it
      Press "Intro" (again) you will see a two options menu
      First Option will reset the NVRAM in a normal way... "recomended"
      Second Option will reset completly the NVRAM (just for xtreme situations)
      Choose what you like...
      The machine will finish the process and halt by it self now just turn the server ON

Wednesday, January 19, 2011

performance issues? first stop vmstat.

vmstat is a very useful tool because allows you to have a quick overall performance view of your system and is available in all Unix type systems
here some quick a dirty explanations on how to understand the output of the vmstat command... btw I am working on AIX and we will focus on a simple output


1. r should never be smaller than b or we may have a CPU bottleneck due processes suspended due to memory load control
2. if fre is really small and if any paging is going on pi or po this is most likely a cause of a bottleneck
3. if b and wa are high we may have an I/O bootleneck due the number of blocking processes
4. if b is low or normal and free is small and us + sy = (close to 100) then we have a memory bottleneck
5. if us+sy average more than 80% we may have a CPU bootleneck if you are at 100 our system is breathing heavily
6. if us+sy is small but wa is greather than 25 we may have I/O intensive activitie or disk subsystem might not be balanced properly which turns on cpu not being able to work as hard as he can
7. if us+sy is over 80% and r is larger than [5 * (Number of processors - Number of bound processors)] then we have a CPU bound
8. if r is greater than the number of CPUs, there is at least one thread waiting for a CPU and this is likelihood of a performance impact.
9. if sy raises over 10000 per second per processor we may be polling subroutines likes select() indicates a bad code it is advisable to have a baseline measurement that gives a count for a normal sy value.

How to block SSH Brute Force Attacks the simple way!

Sometimes you will find out on your systems that some script kiddies are trying to do a brute force attack using ssh and you notice something like this in your logs under /var/log

Jan 19 04:09:47 ZLKFSSH01v sshd[13447]: Failed password for invalid user marty from 221.238.227.35 port 43898 ssh2
Jan 19 12:09:47 ZLKFSSH01v sshd[13448]: Received disconnect from 221.238.227.35: 11: Bye Bye
Jan 19 04:09:50 ZLKFSSH01v sshd[13449]: Invalid user model from 221.238.227.35
Jan 19 12:09:50 ZLKFSSH01v sshd[13450]: input_userauth_request: invalid user model
Jan 19 04:09:50 ZLKFSSH01v sshd[13449]: error: Could not get shadow information for NOUSER
Jan 19 04:09:50 ZLKFSSH01v sshd[13449]: Failed password for invalid user model from 221.238.227.35 port 43902 ssh2
Jan 19 12:09:50 ZLKFSSH01v sshd[13450]: Received disconnect from 221.238.227.35: 11: Bye Bye
Jan 19 04:09:53 ZLKFSSH01v sshd[13451]: Invalid user hayley from 221.238.227.35
Jan 19 12:09:53 ZLKFSSH01v sshd[13452]: input_userauth_request: invalid user hayley
Jan 19 04:09:53 ZLKFSSH01v sshd[13451]: error: Could not get shadow information for NOUSER


Here you can notice a few strange things...
1 The timestamp... there is a discrepancy between your system date... and the attacker meaning he is in a different time zone... Asia maybe

2 The user name is trying to use are regular and common user and system names like (oracle, user, guess, model, client, db2 etc etc)

3 And the most important the IP address... is the same IP address trying several times with different user ids...

To avoid this script kiddies we can use the following script

#!/bin/sh
################################################################################
# Program : /sysadm/BlockSSHAttack.sh
# Description : This script veryfies the /var/log/messages log looking          
# for continous attempts of ssh access that come from the same $IP address and
# are trying to use more than $ALLOWED_USERS
# Initial Create : Juan Medina jmedinar@gmail.com
###############################################################################

### Variables ###

    ALLOW_FILE="/etc/hosts.allow"
    TMP_ALLOW="/tmp/hosts.allow"
    MESSAGES="/var/log/messages"
    TEMP="/tmp/Block.tmp"

### Main ###
   
    # Getting the IPs that have being trying to access today and fail
    grep "^$(date +"%b %d")" $MESSAGES | grep "Invalid user" | awk '{print $10}' | uniq > $TEMP
    # If $TEMP is empty there are no IPs to check so we are done
    if [ -s $TEMP ]
    then
        # For each IP grabbed let's check if have been trying more than 5 usernames
        while read IP
        do
            if [[ $(grep "^$(date +"%b %d")" $MESSAGES | grep "Invalid user" | grep $IP | awk '{print $8}' | sort | uniq | wc -l) -gt 4 ]]
            then
                # Since this IP have been trying with more than 5 different id's Let's block it
                grep $IP $ALLOW_FILE > /dev/null
                if [[ $? -ne 0 ]]       # The IP is not already listed continue
                then
                    # First get the current line from the host.allow
                    Current="$(grep sshd $ALLOW_FILE | grep deny | sed 's/:/\n/g' | grep -v sshd | grep -v deny)"
                    # Let's append the new target to the line
                    if [[ $Current == "" ]]
                    then
                        # This is only for the first time
                        echo "sshd:$IP:deny" >> $ALLOW_FILE
                    else
                        # Let's substitute the line
                        cp $ALLOW_FILE $TMP_ALLOW
                        grep -v ${Current} $TMP_ALLOW > $ALLOW_FILE
                        echo "sshd:$Current, $IP:deny" >> $ALLOW_FILE
                    fi
                fi
            fi
        done < $TEMP
    fi
    rm -rf $TEMP
    rm -rf $TMP_ALLOW
### End of Script ###