Thursday, November 22, 2012

Configuring an NTP server without internet access (locally)

Is the first time I had to figure out how to configure an NTP (Network Time Protocol) server without internet access... most of the time you just configure your /etc/ntp.conf file to point to a public NTP server and your internal servers to point to this one acting as your server and you are done. Nothing interesting there... 

First make sure you have ntp package installed in all your servers

rpm -qa |grep ntp-4
 
Make sure your firewall is stopped 

service iptables stop
service ip6tables stop
chkconfig  iptables off
chkconfig ip6tables off

Or add the required rules to allow port 123 between your servers

iptables -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 123 -j ACCEPT
iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 123 -j ACCEPT
service iptables save
service iptables restart

Backup your current configuration file in all the servers... (just in case)

cp /etc/ntp.conf /etc/ntp.conf.orig

Basically you have to configure your server pointing to itself so it will be in sync... something like this...

vi /etc/ntp.conf
server 127.127.1.0
fudge 127.127.1.0 stratum 10


Notice that to access its own system clock, also called the local clock , NTP uses the pseudo IP address 127.127.1.0. This IP address must not be mixed up with 127.0.0.1, which is the IP of the localhost or loopback

Here you may want to restrict the IPs that are allowed but since this is assuming you are on a local (controlled) environment with no internet access then is not absolutely necessary

Restart the ntpd server

/etc/init.d/ntpd restart
or
service ntpd restart

On the client side you configure as follows...

vi /etc/ntp.conf
server 12.139.41.136


Where the server IP is the IP of your NTP server 

Restart the ntpd server on the clients too

/etc/init.d/ntpd restart
or
service ntpd restart

To Verify your network mask you can look at your network script

cat /etc/sysconfig/network-scripts/ifcfg-eth0

Ensure NTP will start at boot in all the servers

chkconfig ntpd on

Synchronize your local time with the server (do it 3 times):

ntpdate -u [your ntp server IP]

Determining if the NTP is synchronized properly

ntpq –p

One of the problems that I found was with the Stratum Value as you can see in the configuration file we set it to 10 you can verify the current value on the ntp server with the following command 

ntpq -c rv

Now... what does that mean... 

NTP increases the stratum for each level in the hierarchy a NTP server pulling time from a "stratum 1" server would advertise itself as "stratum 2" to its clients. A stratum value of "16" is reserved for unsynchronized servers meaning that your internal NTP server thinks not to have a reliable timesource in other words is not synchronizing to a higher-level stratum server 

Most of the time take like 15 minutes to lower the value... if you are at 16 you wont be able to sync the clients... Once dropped try again

If you need to do some debugging there look at the output of ntpq peers for clues for possible reasons

Thursday, November 8, 2012

Portmir using screen

For those AIX lovers that have by destiny the need to play with Linux and cry because in Linux there is nothing like portmir... where there is a *like solution for this... screen... if you don´t know what screen is look at my previous post ... but here is how to configure it to share a session... 

As root:
   1. Set the screen binary setuid root.
       sudo chmod +s /usr/bin/screen
       sudo chmod 755 /var/run/screen
   2. Start screen
       screen -S portmir
   3. Verify the username with w
   4. Allow multiuser access in the screen session
       CTRL-A
       :multiuser on
   5. Grant permission to the remote user to access the session
       CTRL-A
       :acladd username
   6. The remote user can now connect to the session using
       screen -x root/portmir

Screen

Screen as you may already know is a tool to handle multiple sessions in one window... very useful to remove load from your personal laptop.. avoid having to connect everyday to all those sessions... and having the capability to left processes running =D ... 

well on Linux most of the time is already installed on AIX is never so install it... but there is a little issue on the code that need to be fixed to successfully install the tool on AIX... 

Download the tool from...
ftp ftp.software.ibm.com
anonymous
cd aix/freeSoftware/aixtoolbox/RPMS/ppc/wget
bin
prompt
get wget-1.9-2.aix5.1.ppc.rpm
quit
rpm -hUv wget-1.9-2.aix5.1.ppc.rpm
wget ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/screen/screen-3.9.10-2.aix4.3.ppc.rpm
rpm -hUv screen-3.9.10-2.aix4.3.ppc.rpm
wget ftp://ftp.gnu.org/gnu/screen/screen-4.0.3.tar.gz
gunzip screen-4.0.3.tar.gz
tar -xvf screen-4.0.3.tar
cd screen-4.0.3
vi misc.c


changed following part in order to get over it:

,----[ misc.c - original part ]
| #else /* USESETENV */
| # if defined(linux) || defined(__convex__) || (BSD >= 199103)
| )
| setenv(var, value, 1);
| # else
| setenv(var, value);
| # endif /* linux || convex || BSD >= 199103 */
| #endif /* USESETENV */
| }
`----


Then I used the dirty hack by adding ", 1" to the second setenv-statement directly.

,----[ misc.c - altered part ]
| #else /* USESETENV */
| # if defined(linux) || defined(__convex__) || (BSD >= 199103) ||
defined(__aix__)
| setenv(var, value, 1);
| # else
| setenv(var, value, 1);
| # endif /* linux || convex || BSD >= 199103 */
| #endif /* USESETENV */
| }
`----


That "solved" the error-message above.

./configure
make
make install



Now that the tool is installed here some useful commands



ctrl+a
Access to the screen command line...
p
previous
n
Next
0-9
Jump with ID
w
List open consoles
[
ScrollBack” (ESC to finish)
d
detached (will appear the word [detached])
r
retach (If there are many open session will appear the list, just select the ID to open Type "screen [-d] -r [pid.]tty.host" to resume one of them.”
X
Lock
:
set password
c
open new terminal
A
rename the current session
screen –x
share a screened session without detaching
k
kill session
\
Terminate session
:multiuser
enable multiuser mode





Now to have a very nice .screenrc just copy paste the following and add your servers

autodetach on # Autodetach session on hangup instead of terminating screen completely
startup_message off # Turn off the splash screen
defscrollback 30000 # Use a 30000-line scrollback buffer
scrollback 30000
termcapinfo xterm ti@:te@
vbell off # turn off visual bell
caption string "%?%F%{= Bk}%? %C%A %D %d-%m-%Y %{= kB} %t%= %?%F%{= Bk}%:%{= wk}%? %n "
hardstatus alwayslastline
#hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]'
hardstatus string '%{= kG}[ %{R}%t %{g}]%= %{g}[%{B} %d/%m %{W}%c %{g}]'


screen -t prod-a ssh prod-a
screen -t prod-b ssh prod-b
screen -t prod-c ssh prod-c
screen -t prod-d ssh prod-d


that will create a nice screen and launch all the servers and if you have SSH trusted keys automatically connect to all of them ;) and look sort of like this

 and if you hit [ ctrl+a "  ] you will be able to select your server from there 



A simple Throughtput test using SCP

In the past it was easy to execute a throughput test using ftp now on modern environment FTP or Telnet y most of the time not allowed so we have to do our best with SCP, SFTP and SSH for security reasons... well... having a good test on your environment from all servers to all servers and keeping that info handy can be userful when you are experiencing low performance on the network 

Lets say you have servers A B and C you will execute your test as follows
A to B  
A to C
B to C

that will cover pretty much all the possibilities and you can run B to A but since you already have A to B and they go thru the same wire ... what is the point?... your chooise

   Step # 1: Create a large file (1 GB)

      dd if=/dev/zero of=/tmp/big.file bs=1024M count=1
      1+0 records in
      1+0 records out
      1073741824 bytes (1.1 GB) copied, 1.88378 seconds, 570 MB/s


   Step # 2: Use scp to transfer file
      scp -v /tmp/big.file user@remote.server.com:/tmp
   Step # 3 From the end of the output capture only the required information and build your table


Source Destination MB/s  Duration
A         B                 47.7    0.2 Seconds


Next time you have a problem... well run the test there and figure how slow you are comparing against your baseline =)

ldapsearch over SSL

Recently I found with an issue with some ldap latency between a WAS server and an ldap farm having to do several hops to get there I needed to find out if the connection was working and also how long where they taking... running tcpdump or wireshark traces help but does not give you a real view of how ldap is working ... so I decided to configure ldapclient in this server and do some testing... and this might not work the same in all the environment but will be a good guide. 

First install ldapclient in my case running on RHEL I also needed the openldap package to be installed once this is completed you are able to execute the ldapsearch command... 

But that will be pretty much enough for a regular environment but in my case I had to go thru SSL using port 636 (secure) instead of 389 (insecure) so you have to do a modification to the /etc/openldap/ldap.conf file and add the following lines... 

HOST
PORT 636
TLS_CACERT  
TLS_REQCERT demand


Easy huh? now if you wonder how can you get the certificate to be used... well use this command

echo -n | openssl s_client -connect :636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ldapserver.pem

now how can I check what is being accepted for search in the ldap server... at least

ldapsearch -x -H ldaps:// -b "o=domain.com" 

and you will receive a line sort of like this 

uniquemember: uid=########,c=us,ou=ldapserver,o=domain.com

so now you can narrow your search as follows to look for us folks

ldapsearch -x -H ldaps:// -b "c=us,ou=ldapserver,o=domain.com" 

and then you go to webpshere console and look for those fields that we can access as 


and now you can look by mail, cn, and uid as follows 

ldapsearch -x -H ldaps:// -b "c=us,ou=ldapserver,o=domain.com" "mail=name@domain.com"

Now to check the response times use the following... 

while true
do  
/usr/bin/time -f "\t%e" 2>> /tmp/ldapresponse.out ldapsearch -x -H ldaps:// -b "c=us,ou=ldapserver,o=domain.com" "mail=name@domain.com" > /dev/null
done