Friday, October 23, 2015

Configure linux (Cent OS) server with tomcat 8 for high availability to balance the load of traffic

Click here if you are looking for dedicated cloud servers with lowest price.

Prerequisites :-
Cent OS installed with Java, httpd, mod_jk, tomcat 8 and a sample java ee project to test.


Advantages of tomcat 8 :
  1. Java-centric environment.
  2. Open source.
  3. Easy to work with
  4. Comparatively light weight (especially the servlet container) than other servers like JBoss & WebLogic.
  5. Able to run multiple instances of the server for load balancing the heavy traffic.
  6. Supports Servlet, JSP, EL and Websocket specifications 3.1, 2.3, 3.0 and 1.0 respectively.
  7. Tomcat 8 + spring framework can give the complete enterprise solutions that an enterprise application server can provide.

Advantages of cent os :
  1. Light weight open source OS.
  2. Similar to Red Hat.
  3. Risk of crash and error is very low.
  4. Having wide variety of security features, powerful firewall and SELinux policy mechanism
  5. Enterprise level security updates.
  6. fewer bugs and security holes.
  7. provides reliability, speed and stability for server environment.

Advantages of mod_jk :
  1. fault tolerance
  2. load balancing
  3. dynamic configuration
  4. flexibility and robustness

The tomcat can be downloaded from here.

What is httpd? :-
The httpd provides the apache http server environment. To enable ssl (i.e. https), mod_ssl and openssl also need to be installed.

How to install httpd in cent os :-
# yum install httpd
Once the httpd is installed, a www directory will be created under /var/ i.e. /var/www. And the http request can serve files from /var/www/html directory. Specifically, if a file test.html is kept in /var/www/html, then it can be accessed by http://localhost/test.html in the browser (of course the Apache http server should be started using command # service httpd start).

How to install mod_ssl and openssl to enable ssl secured http server
yum install mod_ssl openssl

How to start httpd in cent os :-
service httpd start

How to restart httpd in cent os :-
service httpd restart

How to stop httpd in cent os :-
service httpd stop



How to install mod_jk in cent os :-
First the mod_jk source should be downloaded from the apache website.

  1. create a directory structure like this /opt/java/mod_jk using the command # yum mkdir -p /opt/java/mod_jk.
  2. change the current location to /opt/java/mod_jk using command # cd /opt/java/mod_jk.
  3. download mod_jk from apache website using command # wget http://www.eu.apache.org/dist/tomcat/tomcat-connectors/jk/tomcat-connectors-1.2.41-src.tar.gz
  4. # ls should list the file tomcat-connectors-1.2.41-src.tar.gz.
  5. extract the zip file using command # tar -xvf tomcat-connectors-1.2.41-src.tar.gz
  6. # ls should also list the extracted file tomcat-connectors-1.2.41-src.
  7. Now change the current location to /opt/java/mod_jk/tomcat-connectors-1.2.41-src/native using # cd /opt/java/mod_jk/tomcat-connectors-1.2.41-src/native
  8. Find out the location of apxs file using command # locate apxs, it may give you a list of locations for the given file name apxs. If the list contains location /usr/bin/apxs, then it should be used for the next command (configure). In some old versions of cent os it may be /usr/sbin/apxs that is why it is confirmed with the locate command.
  9. As we are located in /opt/java/mod_jk/tomcat-connectors-1.2.41-src/native, we can execute the file configure file without explicitly giving the path. Execute the file from the command line with the command # ./configure --with-apxs=/usr/bin/apxs. It may give the similar log given below
[root@localhost native]# ./configure --with-apxs=/usr/bin/apxs
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for test... /usr/bin/test
checking for grep... /usr/bin/grep
checking for echo... /usr/bin/echo
checking for sed... /usr/bin/sed
checking for cp... /usr/bin/cp
checking for mkdir... /usr/bin/mkdir
need to check for Perl first, apxs depends on it...
checking for perl... /usr/bin/perl
APRINCLUDEDIR is  -I/usr/include/apr-1 -I/usr/include/apr-1
building connector for "apache-2.0"
checking for gcc... gcc -std=gnu99
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc -std=gnu99 accepts -g... yes
checking for gcc -std=gnu99 option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc -std=gnu99... none
checking for a sed that does not truncate output... (cached) /usr/bin/sed
checking for grep that handles long lines and -e... (cached) /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking how to print strings... printf
checking for ld used by gcc -std=gnu99... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc -std=gnu99 object... ok
checking for sysroot... no
checking for mt... no
checking if : is a manifest tool... no
checking how to run the C preprocessor... gcc -std=gnu99 -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc -std=gnu99 supports -fno-rtti -fno-exceptions... no
checking for gcc -std=gnu99 option to produce PIC... -fPIC -DPIC
checking if gcc -std=gnu99 PIC flag -fPIC -DPIC works... yes
checking if gcc -std=gnu99 static flag -static works... no
checking if gcc -std=gnu99 supports -c -o file.o... yes
checking if gcc -std=gnu99 supports -c -o file.o... (cached) yes
checking whether the gcc -std=gnu99 linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking for shl_load... no
checking for shl_load in -ldld... no
checking for dlopen... no
checking for dlopen in -ldl... yes
checking whether a program can dlopen itself... yes
checking whether a statically linked program can dlopen itself... yes
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
LIBTOOL="/usr/lib64/apr-1/build/libtool --silent"
checking size of char... 1
checking size of int... 4
checking size of long... 8
checking size of short... 2
checking size of long double... 16
checking size of long long... 8
checking size of longlong... 0
checking size of pid_t... 4
checking size of pthread_t... 8
checking for snprintf... yes
checking for vsnprintf... yes
checking for flock... yes
checking for setsockopt in -lsocket... no
checking sys/filio.h usability... no
checking sys/filio.h presence... no
checking for sys/filio.h... no
checking whether to use SO_RCVTIMEO with setsockopt()... yes
checking whether to use SO_SNDTIMEO with setsockopt()... yes
checking whether to use SOCK_CLOEXEC with socket()... yes
checking poll.h usability... yes
checking poll.h presence... yes
checking for poll.h... yes
checking for poll... yes
checking netinet/in.h usability... yes
checking netinet/in.h presence... yes
checking for netinet/in.h... yes
checking netdb.h usability... yes
checking netdb.h presence... yes
checking for netdb.h... yes
checking for IPv6 with socket()... yes
checking for struct sockaddr_storage... yes
checking for getaddrinfo... yes
checking for gethostbyname_r... yes
checking for target platform... unix
no apache given
no netscape given
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating apache-1.3/Makefile
config.status: creating apache-1.3/Makefile.apxs
config.status: creating apache-2.0/Makefile
config.status: creating apache-2.0/Makefile.apxs
config.status: creating common/Makefile
config.status: creating common/list.mk
config.status: creating common/jk_types.h
config.status: creating common/config.h
config.status: common/config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands

     10. Now execute the command # make, sometimes it may not complete successfully and if it is making this error

[visruth@localhost native]$ make
CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/scripts/build/unix/missing aclocal-1.14 
/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/scripts/build/unix/missing: line 81: aclocal-1.14: command not found
WARNING: 'aclocal-1.14' is missing on your system.
         You should only need it if you modified 'acinclude.m4' or
         'configure.ac' or m4 files included by 'configure.ac'.
         The 'aclocal' program is part of the GNU Automake package:
         <http://www.gnu.org/software/automake>
         It also requires GNU Autoconf, GNU m4 and Perl in order to run:
         <http://www.gnu.org/software/autoconf>
         <http://www.gnu.org/software/m4/>
         <http://www.perl.org/>

make: *** [aclocal.m4] Error 127

then just execute # aclocal it will recreate the files with the current version (the current version can be checked by # aclocal --version command) otherwise install that missing version.

If it is making this error

[root@localhost native]# make
 cd . && /bin/sh /opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/scripts/build/unix/missing automake-1.14 --foreign
/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/scripts/build/unix/missing: line 81: automake-1.14: command not found
WARNING: 'automake-1.14' is missing on your system.
         You should only need it if you modified 'Makefile.am' or
         'configure.ac' or m4 files included by 'configure.ac'.
         The 'automake' program is part of the GNU Automake package:
         <http://www.gnu.org/software/automake>
         It also requires GNU Autoconf, GNU m4 and Perl in order to run:
         <http://www.gnu.org/software/autoconf>
         <http://www.gnu.org/software/m4/>
         <http://www.perl.org/>
make: *** [Makefile.in] Error 1

just execute command # automake, it will recreate the files with the current version. A similar issue is solved in this link.
If the # make command is successfully executed, we will get some similar log like this

[root@localhost native]# make
Making all in common
make[1]: Entering directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/common'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/common'
Making all in apache-2.0
make[1]: Entering directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0'
../scripts/build/instdso.sh SH_LIBTOOL='/usr/lib64/apr-1/build/libtool --silent' mod_jk.la `pwd`
/usr/lib64/apr-1/build/libtool --silent --mode=install cp mod_jk.la /opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0/
libtool: install: warning: remember to run `libtool --finish /usr/lib64/httpd/modules'
make[1]: Leaving directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0'
make[1]: Entering directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native'
make[1]: Nothing to be done for `all-am'.
make[1]: Leaving directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native'
target="all"; \
list='common apache-2.0'; \
for i in $list; do \
    echo "Making $target in $i"; \
    if test "$i" != "."; then \
       (cd $i && make $target) || exit 1; \
    fi; \
done;
Making all in common
make[1]: Entering directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/common'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/common'
Making all in apache-2.0
make[1]: Entering directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0'
../scripts/build/instdso.sh SH_LIBTOOL='/usr/lib64/apr-1/build/libtool --silent' mod_jk.la `pwd`
/usr/lib64/apr-1/build/libtool --silent --mode=install cp mod_jk.la /opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0/
libtool: install: warning: remember to run `libtool --finish /usr/lib64/httpd/modules'
make[1]: Leaving directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0'

     11. As warned in the log remember to run `libtool --finish /usr/lib64/httpd/modules' we have to execute that command also, its log may be like this

[root@localhost native]# libtool --finish /usr/lib64/httpd/modules
libtool: finish: PATH="/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/home/visruth/.local/bin:/home/visruth/bin:/sbin" ldconfig -n /usr/lib64/httpd/modules
----------------------------------------------------------------------
Libraries have been installed in:
   /usr/lib64/httpd/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------

    12. Now the location /opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0/ will contain mod_jk.so file, you can check it by # ls /opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0/ command.
     13. The last step is to place the mod_jk.so library in to /etc/httpd/modules directory. For that we need to execute one more command # make install, this will place the mod_jk.so library in the relevant path.

It will give somewhat similar log like this

[root@localhost native]# make install
Making install in common
make[1]: Entering directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/common'
make[1]: Nothing to be done for `install'.
make[1]: Leaving directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/common'
Making install in apache-2.0
make[1]: Entering directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0'

Installing files to Apache Modules Directory...
/usr/bin/apxs -i mod_jk.la
/usr/lib64/httpd/build/instdso.sh SH_LIBTOOL='/usr/lib64/apr-1/build/libtool' mod_jk.la /usr/lib64/httpd/modules
/usr/lib64/apr-1/build/libtool --mode=install install mod_jk.la /usr/lib64/httpd/modules/
libtool: install: install .libs/mod_jk.so /usr/lib64/httpd/modules/mod_jk.so
libtool: install: install .libs/mod_jk.lai /usr/lib64/httpd/modules/mod_jk.la
libtool: install: install .libs/mod_jk.a /usr/lib64/httpd/modules/mod_jk.a
libtool: install: chmod 644 /usr/lib64/httpd/modules/mod_jk.a
libtool: install: ranlib /usr/lib64/httpd/modules/mod_jk.a
libtool: finish: PATH="/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/home/visruth/.local/bin:/home/visruth/bin:/sbin" ldconfig -n /usr/lib64/httpd/modules
----------------------------------------------------------------------
Libraries have been installed in:
   /usr/lib64/httpd/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 755 /usr/lib64/httpd/modules/mod_jk.so

Please be sure to arrange /etc/httpd/conf/httpd.conf...

make[1]: Leaving directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native/apache-2.0'
make[1]: Entering directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native'
make[2]: Entering directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native'
make[1]: Leaving directory `/opt/java/mod_jk/tomcat-connectors-1.2.41-src/native'

Now the mod_jk installation is finished and it's the time to go in to its configuration.

How to configure mod_jk with httpd:-

  1. Create a new file mod_jk.conf in the /etc/httpd/conf directory.
    # shared memory file location, this is very important
    JKShmFile /var/run/httpd/mod_jk.shm
    
    # Load mod_jk module
    # Specify the filename of the mod_jk lib
    LoadModule jk_module modules/mod_jk.so
    
    # Where to find workers.properties
    JkWorkersFile conf/workers.properties
    
    # Where to put jk logs
    JkLogFile /var/log/httpd/mod_jk.log
    
    # Set the jk log level [debug/error/info]
    JkLogLevel info
    
    # Select the log format
    JkLogStampFormat  "[%a %b %d %H:%M:%S %Y]"
    
    # JkOptions indicates to send SSK KEY SIZE
    JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
    
    # JkRequestLogFormat
    JkRequestLogFormat "%w %V %T"
    
    # Mount your applications
    JkMount /* loadbalancer
  2. Create a new file workers.properties in the /etc/httpd/conf directory (refer apache doc for deprecated properties). 
  3. worker.list=loadbalancer
    
    worker.loadbalancer.type=lb
    worker.loadbalancer.balance_workers=ajp13_worker1,ajp13_worker2
    
    # to enable sticky session
    worker.loadbalancer.sticky_session=True
    worker.loadbalancer.sticky_session_force=True
    
    worker.ajp13_worker1.type=ajp13
    worker.ajp13_worker1.host=localhost
    worker.ajp13_worker1.port=18009
    worker.ajp13_worker1.lbfactor=50
    worker.ajp13_worker1.connection_pool_size=10
    worker.ajp13_worker1.connection_pool_timeout=600
    worker.ajp13_worker1.socket_keepalive=1
    worker.ajp13_worker1.socket_timeout=300
    worker.ajp13_worker1.route=node1
    
    worker.ajp13_worker2.type=ajp13
    worker.ajp13_worker2.host=localhost
    worker.ajp13_worker2.port=8009
    worker.ajp13_worker2.lbfactor=50
    worker.ajp13_worker2.connection_pool_size=10
    worker.ajp13_worker2.connection_pool_timeout=600
    worker.ajp13_worker2.socket_keepalive=1
    worker.ajp13_worker2.socket_timeout=300
    worker.ajp13_worker2.route=node2
     3. Edit /etc/httpd/conf/httpd.conf and add a single line at the end of the file:
# Include mod_jk's specific configuration file
Include conf/mod_jk.conf

Now we can start the service with # service httpd start but we also have to tail the mod_jk.log as # tail -f /var/log/httpd/mod_jk.log, if the log contains an error like below

[Sun Oct 24 22:08:14 2015][27481:139791102044224] [error] init_jk::mod_jk.c (3574): Initializing shm:/etc/httpd/logs/jk-runtime-status.27481 errno=13. Load balancing workers will not function properly.

then we must declare JKShmFile in mod_jk.conf and the file location must have read write permission for apache. And also need to execute # setsebool -P httpd_unified=1 command if the os is selinux.

A good reference for Apache Installation and Configuration can be found here.

Now, it's the time time to configure our /tomcat8/conf/server.xml file. Session replication is not a recommended way for large cluster so we are using sticky session. This will allow tomcat_instance1 to get the same session if it is created by it. That means the creator of the session will always receive the same session request. For mod_jk to identify for the creator of the session we have to specify jvmRoute in Engine tag so that the JSESSIONID will be appended with jvmRoute value.

Suppose we have two tomcat8s, tomcat_instance1 and tomcat_instance2. The tomcat_instance1 is for node1 and the tomcat_instance2 is for node2.
We have two things to care about,
First we have to enable ajp1.3 protocol and second  we have to specify jvmRoute for sticky sessions.

So,

  1. Go to tomcat_instance1/conf/server.xml and add <Connector port="18009" protocol="AJP/1.3" redirectPort="8443" /> and also specify jvmRoute in Engine as <Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
  2. Go to tomcat_instance2/conf/server.xml and add <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> and also specify jvmRoute in Engine as <Engine name="Catalina" defaultHost="localhost" jvmRoute="node2">

That's all!

Flow of load balancing
Http request from user goes to httpd which then goes to mod_jk's  loadbalancer worker. The loadbalancer worker routes the request to the ajp13_worker1 or ajp13_worker2 workers based on the load (and also considering the creator of the session).