Wednesday, September 28, 2011

Graph of system usage

Sometimes managers need info from the servers and they want to see it in a fancy way... like when you want to know the amount of time users spend on a server but you need a graphic... and I know you can just take the values paste them into a spreadsheet and create a graph... but I like more the automated and lovely look of the console... so Here is one solution.

Graphic_Of_System_Usage
{
    set -A resources user1 user2 user3 user4 user5 user6 user7 user8
    for Q in $(echo ${resources[*]})
    do
       rm /tmp/resource.usage
       last | grep $(date +%h) | grep $Q | awk '{print $9}' | grep '(' | sed 's/(//g' | sed 's/)//g' | sed 's/:/ /g' > /tmp/resource.usage
       USAGE=0
       CHART=0
       J=0
       while read MIN SEC
       do
           (( TMIN = MIN * 60 ))
           (( TSEC = TMIN + SEC ))
           (( USAGE = $USAGE + $TSEC ))
       done < /tmp/resource.usage
       (( CHART = $USAGE / 100 ))
       echo "$Q: \c"
       while [ $J -lt $CHART ]
       do
           echo "|\c"
           (( J = $J +1 ))
       done   
       echo "\n"
    done
}

you will see something like this.

Wednesday, September 21, 2011

Blocking IPs with a blackhole route

A null route or blackhole route is a routing table entry that goes nowhere. Matching packets are dropped (ignored) rather than forwarded, acting as a kind of very limited firewall. The act of using null routes is often called blackhole filtering

Null routes are typically configured with a special route flag, but can also be implemented by forwarding packets to an illegal IP address such as 0.0.0.0, or the loopback address.

Null routing has an advantage over classical firewalls since it is available on every potential network router, and adds virtually no performance impact. Due to the nature of high-bandwidth routers, null routing can often sustain higher throughput than conventional firewalls. For this reason, null routes are often used on high-performance core routers to mitigate large-scale denial-of-service attacks before the packets reach a bottleneck, thus avoiding collateral damage from DDoS attacks — although the target of the attack will be inaccessible to anyone. Blackhole filtering can also be abused by malicious attackers on compromised routers to filter out traffic destined to a certain address.
Nullrouting with iproute2 on Linux:
   $ ip route add blackhole 192.168.32.128/32
Nullrouting with 'route' on Solaris and BSD:
   $ route add -host 10.10.0.1 127.0.0.1 -blackhole
   $ route add -net 10.10.64.0/18 127.0.0.1 -blackhole
 
Creating a discard route on Juniper Networks' Junos:
    set routing-options static route 192.168.0.0/16 discard 
Routing to the Null0 interface on Cisco IOS:
    ip route 192.168.0.0 255.255.0.0 Null0

Crypt and Decrypt files

Hi I am back, A few days ago I had to change (as regularly we do) the root password of all my servers and consoles because a member leave the team... remember to do that all the time!

Then I had to share the new root password to all my team but I don't like sending password over the network on a plain text email and call them one by one ... well is expensive and exhaustive ... so I found a neat solution and will like to share it!.

Basically what I did was write down the password on a text file inside unix crypt using openssl
and then send an email to my team providing the location and asking them to use the OLD root password to open this file and knew the NEW root password...

so here is the example!

# echo "This is a simple string" > MyNormal.file
# cat MyNormal.file
This is a simple string
# openssl bf -salt -in MyNormal.file > MyCrypted.file
enter bf-cbc encryption password:
Verifying - enter bf-cbc encryption password:
# rm MyNormal.file
# cat MyCrypted.file
Salted__|    qqk��=��r�3 �#DOC��@Ϝ�g�
                                            ��}�@V#
# openssl bf -d -in MyCrypted.file
enter bf-cbc decryption password:   # Example with a wrong password
bad decrypt
1710:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:330:
��q#jR�":�z�� �} ƺ�P�`}#
# openssl bf -d -in MyCrypted.file
enter bf-cbc decryption password:   # Example with the correct password
This is a simple string
#

Thursday, February 3, 2011

Cleaning solved! Hardware errors

You have a problem with a hardware part and call your support partner and the piece has been repaired or replaced ... BUT you are still getting error regarding this part on the errpt reports... Here is how to clean the log for that error.

# diag


type enter to continue


Select TASK SELECTION

Select Log Repair Action


Select all the devices you want to clean with enter and the ctrl+7 or f7


You will be prompted to confirm.. just hit enter and you are done! ctrl+0 or f10
will take you out of the diag tool

after that you can clean you errpt report with the following command.

# errclear 0

and you are all set... ready for the device to fail again and give you a true report =)

Tuesday, January 25, 2011

Reset the NVRAM

Sometimes when you upgrade or reboot a server for any maintenance you will find that the system is not able to locate devices, OS or other stuff... a good try to recover from this failure that have no mayor impact than loose the latest state of the hardware NVRAM is the following... will save you some headaches

     Access the "service procesor" when the servers is off showing "OK" @ the panel
      Press any key and write this code... 85763
   
      when the panel is back in "OK"    Press "Intro" you will see a warning ... ignore it
      Press "Intro" (again) you will see a two options menu
      First Option will reset the NVRAM in a normal way... "recomended"
      Second Option will reset completly the NVRAM (just for xtreme situations)
      Choose what you like...
      The machine will finish the process and halt by it self now just turn the server ON

Wednesday, January 19, 2011

performance issues? first stop vmstat.

vmstat is a very useful tool because allows you to have a quick overall performance view of your system and is available in all Unix type systems
here some quick a dirty explanations on how to understand the output of the vmstat command... btw I am working on AIX and we will focus on a simple output


1. r should never be smaller than b or we may have a CPU bottleneck due processes suspended due to memory load control
2. if fre is really small and if any paging is going on pi or po this is most likely a cause of a bottleneck
3. if b and wa are high we may have an I/O bootleneck due the number of blocking processes
4. if b is low or normal and free is small and us + sy = (close to 100) then we have a memory bottleneck
5. if us+sy average more than 80% we may have a CPU bootleneck if you are at 100 our system is breathing heavily
6. if us+sy is small but wa is greather than 25 we may have I/O intensive activitie or disk subsystem might not be balanced properly which turns on cpu not being able to work as hard as he can
7. if us+sy is over 80% and r is larger than [5 * (Number of processors - Number of bound processors)] then we have a CPU bound
8. if r is greater than the number of CPUs, there is at least one thread waiting for a CPU and this is likelihood of a performance impact.
9. if sy raises over 10000 per second per processor we may be polling subroutines likes select() indicates a bad code it is advisable to have a baseline measurement that gives a count for a normal sy value.

How to block SSH Brute Force Attacks the simple way!

Sometimes you will find out on your systems that some script kiddies are trying to do a brute force attack using ssh and you notice something like this in your logs under /var/log

Jan 19 04:09:47 ZLKFSSH01v sshd[13447]: Failed password for invalid user marty from 221.238.227.35 port 43898 ssh2
Jan 19 12:09:47 ZLKFSSH01v sshd[13448]: Received disconnect from 221.238.227.35: 11: Bye Bye
Jan 19 04:09:50 ZLKFSSH01v sshd[13449]: Invalid user model from 221.238.227.35
Jan 19 12:09:50 ZLKFSSH01v sshd[13450]: input_userauth_request: invalid user model
Jan 19 04:09:50 ZLKFSSH01v sshd[13449]: error: Could not get shadow information for NOUSER
Jan 19 04:09:50 ZLKFSSH01v sshd[13449]: Failed password for invalid user model from 221.238.227.35 port 43902 ssh2
Jan 19 12:09:50 ZLKFSSH01v sshd[13450]: Received disconnect from 221.238.227.35: 11: Bye Bye
Jan 19 04:09:53 ZLKFSSH01v sshd[13451]: Invalid user hayley from 221.238.227.35
Jan 19 12:09:53 ZLKFSSH01v sshd[13452]: input_userauth_request: invalid user hayley
Jan 19 04:09:53 ZLKFSSH01v sshd[13451]: error: Could not get shadow information for NOUSER


Here you can notice a few strange things...
1 The timestamp... there is a discrepancy between your system date... and the attacker meaning he is in a different time zone... Asia maybe

2 The user name is trying to use are regular and common user and system names like (oracle, user, guess, model, client, db2 etc etc)

3 And the most important the IP address... is the same IP address trying several times with different user ids...

To avoid this script kiddies we can use the following script

#!/bin/sh
################################################################################
# Program : /sysadm/BlockSSHAttack.sh
# Description : This script veryfies the /var/log/messages log looking          
# for continous attempts of ssh access that come from the same $IP address and
# are trying to use more than $ALLOWED_USERS
# Initial Create : Juan Medina jmedinar@gmail.com
###############################################################################

### Variables ###

    ALLOW_FILE="/etc/hosts.allow"
    TMP_ALLOW="/tmp/hosts.allow"
    MESSAGES="/var/log/messages"
    TEMP="/tmp/Block.tmp"

### Main ###
   
    # Getting the IPs that have being trying to access today and fail
    grep "^$(date +"%b %d")" $MESSAGES | grep "Invalid user" | awk '{print $10}' | uniq > $TEMP
    # If $TEMP is empty there are no IPs to check so we are done
    if [ -s $TEMP ]
    then
        # For each IP grabbed let's check if have been trying more than 5 usernames
        while read IP
        do
            if [[ $(grep "^$(date +"%b %d")" $MESSAGES | grep "Invalid user" | grep $IP | awk '{print $8}' | sort | uniq | wc -l) -gt 4 ]]
            then
                # Since this IP have been trying with more than 5 different id's Let's block it
                grep $IP $ALLOW_FILE > /dev/null
                if [[ $? -ne 0 ]]       # The IP is not already listed continue
                then
                    # First get the current line from the host.allow
                    Current="$(grep sshd $ALLOW_FILE | grep deny | sed 's/:/\n/g' | grep -v sshd | grep -v deny)"
                    # Let's append the new target to the line
                    if [[ $Current == "" ]]
                    then
                        # This is only for the first time
                        echo "sshd:$IP:deny" >> $ALLOW_FILE
                    else
                        # Let's substitute the line
                        cp $ALLOW_FILE $TMP_ALLOW
                        grep -v ${Current} $TMP_ALLOW > $ALLOW_FILE
                        echo "sshd:$Current, $IP:deny" >> $ALLOW_FILE
                    fi
                fi
            fi
        done < $TEMP
    fi
    rm -rf $TEMP
    rm -rf $TMP_ALLOW
### End of Script ###