General use case : Infrastructure as code with ansible and terraform

-

 In this article, I tried to develop infrastructure on aws with help of terraform and then tried to achieve configuration management using ansible.


## Technologies used

Project is created with:

* AWS

* Ansible: 2.9.25

* Terraform: 1.0.5

* Hashicorp vault


## Operating-System

* Redhat 8

* Ubuntu 20.04 LTS


## Pre-requisite

* An AWS Account

* An AWS CLI needs to be installed

* Your AWS credentials. 

  * You can create a new Access Key on this page.[link](https://console.aws.amazon.com/iam/home?#/security_credentials)


## Set-up

* Git needs to be installed in host machine

   * [git](https://linuxconfig.org/install-git-in-linux-redhat-8)

* Install Python3, Terraform, Ansible locally.

   * [Python3 installation](https://linuxconcept.com/how-to-install-python-3-on-rhel-8-red-hat-enterprise-linux/)

   * Terraform : 

      * [Download Terraform](https://www.terraform.io/downloads.html)

      * [Install Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli?in=terraform/aws-get-started)

   * Ansible : Install ansible on host machine

      * [Ansible installation](https://www.decodingdevops.com/how-to-install-ansible-on-rhel-8/)

      * [ubuntu ansible installation](https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-ansible-on-ubuntu-18-04)

* Set up AWS credentials in ~/.aws/credentials.

   * The easiest way to do so is by setting up the AWS CLI. 

      * [Install CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)

      * [Install CLI on linux](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html)

* once aws-cli is installed in host machine, set up aws credentials using below command

   $ aws configure

   * This will ask for your AWS access key and AWS Secret as below

      - AWS Access Key ID [None] : ******

      - AWS Secret Access Key [None] : ******

      - Default region name : [leave it none , press enter]

      - Default output format : [leave it none, press enter]

* Generate ssh keys for your host machine and save it under path '/home/[user]/.ssh'

   * [Link to generate ssh keys](https://www.cyberciti.biz/faq/how-to-set-up-ssh-keys-on-rhel-8/)

* Set up hashicorp vault in your host machine which is used to store ansible-vault password.

   * [Hashicorp vault installation](https://learn.hashicorp.com/tutorials/vault/getting-started-install?in=vault/getting-started)

   * [Vault - secret creation](https://learn.hashicorp.com/tutorials/vault/getting-started-first-secret?in=vault/getting-started)


## Usage

* Files used in this project with their functionality listed

  * ansible/Files/httpd.conf.j2 :- Template file for apache(httpd) server config

  * ansible/Files/info.php.j2 :- Template file for sample info.php

  * ansible/vars/default.yml :- contains variables, which references are used in main ansible yml file

  * ansible/ansible.cfg :- config file for ansible

  * ansible/allDistros.yml :- yml file to install lamp stack on redhat/ubuntu image

  * terraform/aws.tf :- This file contains cloud provider details for terraform

  * terraform/ec2.tf :- This file contains details for ec2 instance creation

  * ansible/install_mediawiki.sh :- bash script to install mediawiki in created ec2 instance, this script is invoked in ansible playbook command

  * ansible/install_module.sh :- bash script to reset/enable php modules in ec2 instance

  * terraform/output.tf :- This file will have public ip of ec2 instance once creation is success.

  * terraform/vars.tf :- This file contains variables for terraform

  * terraform/vpc.tf :- This file contains network details for ec2 instance

  * createInfra.sh :- This script contains terraform commands to create EC2 infra in aws

  * destroyInfra.sh :- This script contains terraform commands to destroy EC2 infra in aws

  

To run this project, execute below commands.

  * clone this project

      $ git clone https://github.com/sheldon-cooper26/InfraAsCodeWithTerraformAndAnsible.git  

  * go to directory 'InfraAsCodeWithTerraformAndAnsible'

      $ cd InfraAsCodeWithTerraformAndAnsible

  * execute createInfra script

      $ chmod +x createInfra.sh

      $ ./createInfra.sh

 

 ### Notes:

   * Here i am considering path for ssh keys would be /home/user/.ssh path, you can generate and keep ssh keys in any path and make sure you use same path while running terraform apply command

   * I am considering us-east-1 as default region, same has been set in createInfra.sh and destroyInfra.sh scripts. Please feel free to change it according to your feasibility.


## Output

When createInfra.sh executes successfully, you will get similar below message on console.

""""""""""

address = "http://ec2-54-175-133-129.compute-1.amazonaws.com/mediawiki/"

ssh = "ssh ec2-user@34.226.196.34"

Infrastructure created successfully using terraform apply command!!

"""""""""""

try to access above mediawiki url : http://[public-ip]/mediawiki/

* Eg : http://ec2-54-175-133-129.compute-1.amazonaws.com/mediawiki/

* you should see setup mediawiki page as below

    ![alt text](https://github.com/sheldon-cooper26/InfraAsCodeWithTerraformAndAnsible/blob/main/image.png?raw=true)

  

## Project-status

    In-progress

    

## Challenges

* with this automation I was only able to install mediawiki in ec2-instance. But, complete setup requires manual intervention, like we need to set language, add database details for wiki manually by following web instructions. Need to find a way to automate this, so that complete installation+configuration suite can be automated.


## Future-improvement

This project can be improved in below areas.

* Ansible variables handling 

    * I am working to upgrade this project by using ansible vault to store db credentials in encrypted format [completed]

        * encrypted vars/default.yml file using anisble-vault. please find below command to execute ansible playbook using vault password file

            $ ansible-playbook -i inventory allDistros.yml

    * Update 

      * Now Ansible vault is integrated with hashicorp vault to make it more secure. Earlier password was kept in .vault file and this file is committed in git repo (which is very bad practice). 

      * Now Ansible vault password is read from .py script

      * ansible playbook command : $ ansible-playbook -i inventory allDistros.yml --vault-password-file ~/InfraAsCodeWithTerraformAndAnsible/readPasswd.py

      * Complete details can be found in this link. [Learn-With-Tapan](https://medium.com/@hegdetapan2609/secure-your-ansible-vault-password-using-hashicorp-vault-and-python-script-afe2f7fb282a)

      * we can run hashicorp vault in remote machine and can get secrets values from remote server. For this we need to enable VAULT_ADDR and VAULT_TOKEN value as below

      $ export VAULT_ADDR='http://172.31.87.28:8200' [ http://< private ip of remote server>:8200 ]

      $ export VAULT_TOKEN="**********" [ root-token value ]

     * we need to start vault server in remote machine as below

      vault server -dev -dev-listen-address="172.31.87.28:8200" [ -dev-listen-address = 'private-ip:8200' ]

* Terraform file declaration :

    * Modules can be created and these module templates can be used in main.tf which is used ec2-instance creation.

    * Load balancing of instance can be achieved

    * Blue-green and canary deployment can be achieved

* For now this project is tested only for AWS cloud, it can be extended to other cloud providers like GCP, Azure, Digital Ocean etc

* For now this project is tested with RHEL-8 ami image, it can fine tuned to use other linux flavours like ubuntu, amazon linux, suse etc.



# Update         

        * Now this functionality is extended to both RedHat image and Ubuntu image. Refere allDistros.yml for references.    

        * Next, I am planning to create automation in terraform files to pick proper image depending on condition.

        * mediawiki site is running on http protocol(not secure) by default, we can make necessary changes at httpd web server level to make it secure(https)


## Notes:

* If you are using fresh machine and ansible is newly installed in that machine, then while running ansible command you might face below error

     * "ERROR! couldn't resolve module/action 'community.mysql.mysql_user'. This often indicates a misspelling, missing collection, or incorrect module path."

     * To resolve this error, you need to install community.mysql from ansible-galaxy collection

    $ ansible-galaxy collection install community.mysql 

    $ ansible-galaxy collection install ansible.posix

## References
* [how to use ansible with terraform on Digital ocean](https://www.digitalocean.com/community/tutorials/how-to-use-ansible-with-terraform-for-configuration-management)
* [hasicorp tutorials](https://learn.hashicorp.com/tutorials/terraform/infrastructure-as-code?in=terraform/aws-get-started)
* [Ansible documentation](https://docs.ansible.com/ansible/2.3/list_of_all_modules.html)
* [install mediawiki on linux](https://www.linuxtechi.com/install-mediawiki-on-linux/)
* [mediakwiki manual - RedHat](https://www.mediawiki.org/wiki/Manual:Running_MediaWiki_on_Red_Hat_Linux)
* [mediawiki manual - ubuntu](https://www.mediawiki.org/wiki/Manual:Running_MediaWiki_on_Debian_or_Ubuntu)
  

Comments

Post a Comment

Popular posts from this blog

OSB-12C : XA Transactional Support feature

-
Image
OSB 12c supports both XA as well as Non-XA transactions. An XA transaction is a "global transaction" that may span multiple resources. It involves a coordinating transaction manager, with one or more databases (or other resources, like JMS) all involved in a single global transaction. XA transactions come from the X/Open group specification on distributed, global transactions. JTA includes the X/Open XA spec, in modified form. XA gets involved when you want to work with multiple resources - 2 or more databases, a database and a JMS connection, all of those plus maybe a JCA resource - all in a single transaction. In this scenario, you'll have an app server like WebSphere or Weblogic or JBoss acting as the Transaction Manager, and your various resources (Oracle, Sybase, IBM MQ JMS, SAP etc) acting as transaction resources.  Your code can then update/delete/publish/whatever across the many resources. When you say "commit", the results are committe...
Read more

Static code analysis of Terraform .tf files using ‘Checkov’ — Secure your infrastructure

-
Image
  Hands on shelf! We can all agree that Terraform is great tool for infrastructure creation and provisioning. When we write terraform .tf files for infra creation we might miss something and we add something which can lead to security issues. Security should be prime factor for any devops related use-case. When I was searching internet for is there any tool which will scan terraform files and give a picture what may go wrong (from security perspective), I came across this very useful tool named ‘Checkov’ Link :  https://www.checkov.io/1.Welcome/Quick%20Start.html Checkov is a   static code analysis tool for scanning infrastructure as code (IaC) files for misconfigurations that may lead to security or compliance problems. Checkov includes more than 750 predefined policies to check for common misconfiguration issues. In this post, I am going to discuss how to do static code analysis of terraform tf files using checkov. step 1 : Install checkov in your machine. $ pip install...
Read more

OSB12c : Schema Validation - Dynamic Validation

-
Image
We can use validate node in OSB to validate schema. There was an option to validate the payload in OSB 11g but it was static, however from 12c there is a new feature included in the OSB validation activity that is validating the data dynamically. This will allow the user to validate the input data or other data dynamically based on certain condition. Thus the process can be stopped at the beginning itself the data validation goes wrong. You have the option to save the validation result or raise an error. Use case: 1. Create a WSDL based service 2. Add validate (dynamic) to validate incoming request structure. 3. If any violation in request structure compared with schema, Error will be raised with proper error code and description. Create new folder and name it as ‘DynamicValidateSchema_UseCase’. Create sub folders structure as shown in below image. For this use case we have created two XSDs and placed it inside Resources -> XSD folder EmployeeDetailsReq...
Read more