Installing configuring Ansible and deploying haproxy playbook

  Basic installation 

sudo yum install ansible (or) sudo pip install ansible

Configuring your hosts file with inventory 

cat /etc/ansible/hosts 
[haproxy]
10.0.184.12   
[all_servers:vars]
ansible_connection=ssh 
ansible_user=root
ansible_ssh_private_key_file=/Users/home/yoursshkey.pem

I have only one server in my inventory where i will be setting up an haproxy For easy management we will be writing common variables under [all_servers:vars]
When we run ansible play books hosts will be picked from the above inventory file
Example : hosts: haproxy  ( this will run on all servers under haproxy )
Now test servers doing a ping ( ping in ansible will connect to the server using ssh )
 ansible haproxy -m ping
10.0.184.12 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
Then Download ready made packages from ansible galaxy 
https://galaxy.ansible.com/devops/haproxy/
Also download dependencies that will configure EPEL repositories
https://galaxy.ansible.com/sfromm/epel/
Then create a common playbook for both the roles main.yml

- hosts: haproxy
  roles:
     - { role: devops.epel }
- hosts: haproxy
  roles:
     - role: devops.haproxy
       haproxy_stats:
         name: 'global_monitor'
         ip: "{{ ansible_default_ipv4.address }}"
         port: '38888'
         stats:
           enabled: True
           hide_version: true
           uri: /slb_stats_url
           realm: Welcome\ to\ slb\ monitor
           auth: admin:admin
           refresh: 2s
       haproxy_frontends:
       - name: 'fe-testsite'
         ip: '{{ ansible_default_ipv4.address }}'
         port: '80'
         maxconn: '1000'
         default_backend: 'be-testsite'
       haproxy_backends:
       - name: 'be-testsite'
         description: 'testsite'
         servers:
           - name: 'be-testsite-01'
             ip: '192.168.1.100'


sh-3.2# ansible-playbook main.yml 

PLAY [haproxy] *****************************************************************

TASK [setup] *******************************************************************
ok: [10.0.184.12]

TASK [devops.epel : Installs python dependencies] ******************************
changed: [10.0.184.12] => (item=[u'libselinux-python'])

TASK [devops.epel : Disable SElinux] *******************************************
changed: [10.0.184.12]

TASK [devops.epel : create EPEL yum repository] ********************************
changed: [10.0.184.12]

TASK [devops.epel : import EPEL GPG key] ***************************************
changed: [10.0.184.12]

PLAY [haproxy] *****************************************************************

TASK [setup] *******************************************************************
ok: [10.0.184.12]

TASK [devops.haproxy : Include OS-specific variables.] *************************
ok: [10.0.184.12]

TASK [devops.haproxy : Installs haproxy as well as socat for socket api.] ******
changed: [10.0.184.12] => (item=[u'haproxy', u'socat'])

TASK [devops.haproxy : Ensure HAProxy is started and enabled on boot.] *********
changed: [10.0.184.12]

TASK [devops.haproxy : Ensure chroot directory exists.] ************************
ok: [10.0.184.12]

TASK [devops.haproxy : Create directory for the frontend] **********************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the frontends] *********************************
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so that the environment value uses the full variable syntax ('{{haproxy_frontends}}').
This feature will be
removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [10.0.184.12] => (item={u'ip': u'10.0.184.12', u'maxconn': u'1000', u'default_backend': u'be-testsite', u'port': u'80', u'name': u'fe-testsite'})

TASK [devops.haproxy : Create directory for the backends] **********************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the backends] **********************************
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so that the environment value uses the full variable syntax ('{{haproxy_backends}}').
This feature will be
removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [10.0.184.12] => (item={u'servers': [{u'ip': u'192.168.1.100', u'name': u'be-testsite-01'}], u'description': u'testsite', u'name': u'be-testsite'})

TASK [devops.haproxy : Create directory for the listen sections] ***************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the listen sections] ***************************
skipping: [10.0.184.12] => (item=haproxy_listen)

TASK [devops.haproxy : Create directory for the userlists] *********************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the userlist sections] *************************
skipping: [10.0.184.12] => (item=haproxy_userlists)

TASK [devops.haproxy : Create  the compiled folder] ****************************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the global config] *****************************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the default config] ****************************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the stats config] ******************************
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the frontends configuration file] **************
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the backends configuration file] ***************
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the listen sections configuration file] ********
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the userlists sections configuration file] *****
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the final configuration file] ******************
changed: [10.0.184.12]

RUNNING HANDLER [devops.haproxy : restart haproxy] *****************************
changed: [10.0.184.12]

PLAY RECAP *********************************************************************
10.0.184.12                : ok=31   changed=27   unreachable=0    failed=0




This is just a test deployment that will forward request to background test server 192.168.1.100

Kibana4 apache configuration with authentication


We need to configure httpd to listen on port 80 which does a reverse proxy to localhost on 5601.
Make sure kibana is listening only on internal port before this settings are done .


        ServerAdmin devops@learnadmin.com
        ServerName kibana.learnadmin.com

        DocumentRoot /var/www/auth

       
            AuthType Basic
            AuthName "Authentication Required"
            AuthUserFile "/var/www/auth/htpasswd"
            Require valid-user
       

        ProxyPass / http://localhost:5601/
        ProxyPassReverse / http://localhost:5601/



Use below command to create a htpasswd file and enable authentication 
htpasswd -c /var/www/auth/htpasswd devops

Monitoring of kibana and logstash services and start them if process not found

Kibana and logstash might run out of memory and application might get closed or get terminated. In that case we will be running the below script to monitor services and start them if not running .
There are other parameters that need to be verified before going for this option like java heap size etc.


#!/bin/bash

LOGSTASH=logstash
KIBANA=kibana


var1=`ps -ef|grep -i $LOGSTASH|sed -e '/grep/d'`

if [ -z "$var1" ]; then

nohup /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf &

echo "Process logstash not found" | mail -s "Process logstash not found creating a new one" devops@learnadmin.com

else

echo "process logstash  found."

fi

var2=`ps -ef|grep -i $KIBANA|sed -e '/grep/d'`

if [ -z "$var2" ]; then

nohup /opt/kibana/bin/kibana status &


echo "Process kibana  not found" | mail -s "Process kibana not found creating a new one" devops@learnadmin.com

else

echo "process kibana found."


fi

Chef provision a esxi virtual machine or a virtual server

We need to install esxi knife plugin on our workstation


gem install knife-esx

Add below params in knife.rb

knife[:esx_host] = "esx"
knife[:esx_username] = "root"
knife[:esx_password] = "password of your esxi server"

Need to create a ubuntu template ( or custom OS template as per requirement )
Below example shows how to provision a ubuntu template .

ssh root@esxi

mkdir -p /vmfs/volumes/datastore1/esx-gem/templates

vmkfstools -i /vmfs/volumes/datastore1/ubuntu-12.10-x64_template/*.vmdk –diskformat thin /vmfs/volumes/datastore1/esx-gem/templates/ubuntu-12.10-x64_template.vmdk

esx template list ( should display the template we installed )


knife esx vm create --vm-name server1 --use-template ubuntu-12.10-x64_template --verbose true --distro ubuntu12.04-19-gems --vm-memory 2048 -x provision -i ~/.ssh/id_rsa

How to use chef encrypted databags


Note: Please comment the below line in knife.rb file else it will automatically decrypt values
and show when we use "knife data bag show" command
I initially thought knife was not encrypting my values and had to debug ..which took lot of time.



[root@ec2-test .chef]# cat knife.rb | grep secret
#knife[:secret_file] ='/root/encrypted_data_bag_secret'



knife data bag create  --secret-file /root/.chef/encrypted_data_bag_secret testbag password

{
  "id": "password",
  "password": "this is test password key"


}



[root@ec2-test .chef]# knife data bag show testbag password

id:       password
password:
  cipher:         aes-256-cbc
  encrypted_data: KMHzeFQwfm0wWeHFymxfJsMo425CP+wlwoZ6xN7waVlgUNOmRrr/+jOtDLIN
  s7Xl
  
  iv:             TYPYnSqYTcmU8ZWE2sIt4A==
  
  version:        1
  


Once encrypted if you try to edit the encrypted databag it shows like this 

{
  "name": "data_bag_item_testbag_password",
  "json_class": "Chef::DataBagItem",
  "chef_type": "data_bag_item",
  "data_bag": "testbag",
  "raw_data": {
    "id": "password",
    "password": {
      "encrypted_data": "KMHzeFQwfm0wWeHFymxfJsMo425CP+wlwoZ6xN7waVlgUNOmRrr/+jOtDLIN\ns7Xl\n",
      "iv": "TYPYnSqYTcmU8ZWE2sIt4A==\n",
      "version": 1,
      "cipher": "aes-256-cbc"
    }
  }
}



To show the decrypted values we can use 

[root@ec2-test .chef]# knife data bag show  testbag password --secret-file /root/.chef/encrypted_data_bag_secret

id:       password
password: this is test password key


---------------------------
Some important commands related to data bags

Generate random secret key


openssl rand -base64 512 >encrypted_data_bag_secret

Generate random password

date | md5 in mac 
date | md5sum in any linux machine 

Setting a Secure Flag to cookie in httpd – Apache

First check if mod_headers is loaded. Else enable that module


LoadModule headers_module modules/mod_headers.so

Once the module is loaded .

Header edit Set-Cookie ^(.*)$ $1;Secure


Add above line to your virtual host in httpd.conf/ssl.conf as per your config

What this does is it adds a secure flag to your cookie. This flag is set by application server when sending
a new cookie to client/user when sending http response.This also mean that server will not send cookie over http. It sends only via https .This prevents unauthorized access to cookie data

This can be done via code in java or can be done @ apache/httpd config level.

Here is more info on code level changes click here..!!