Continous Integration with Ansible + Terraform + Jenkins on AWS cloud

-

 In this blog I am going to demonstrate how to achieve CI/CD (Continous Integration and Continous Deployment) using Infrastructure as Code (Terraform), configuration management(Ansible) on AWS cloud.


Technologies used :

1. Cloud : Amazon Web service

2. Infrastructure as Code : Terraform (version - 1.0.10)

3. Configuration Management : Ansible (Version - 2.9)

4. Jenkins (Master/Slave architecture)


Basically, we will try to install mediawiki 1.36 site on ec2 instance using ansible and terraform and then we will integrate this duo with jenkins to achieve continous integration.

Note : Steps  for installation and configuring mediawiki is listed in below github repo.

https://github.com/sheldon-cooper26/InfraAsCodeWithTerraformAndAnsible

Once your ansible-terraform module is ready, we can commit this module in github repo and then we can integrate same with jenkins.

Install Jenkins on AWS:

--------------

We will spawn up two instances in AWS, one of them serves as 'master' and second as 'slave'

Master Node :-

1. Lauch AWS ec2 instance in any region (t2.micro, ubuntu ami, with default values)

    In this example : I am considering Ubuntu-18.04 ami

2. Once your instance is ready and up-and-running, do ssh login into that machine using .pem key

ssh -i <path to pem key> ubuntu@<public ip>

3. Then execute below commands to install jenkins

    * First update ubuntu package

              $ sudo apt update

    * Next we need to install java in this new machine

              $ sudo apt-get install -y openjdk-8-jdk

    * Next, add the repository key to the system

              $ wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -

    * When the key is added, the system will return OK. Next , append debian package repo address list to server's source.list

             $ sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'

    * Again run update so that apt will use new repository

            $ sudo apt update

    * Lastly, install jenkins

            $ sudo apt install jenkins -y

4. Now we need to start jenkins server.

    * Lets start jenkins server using 'systemctl' command

            $ sudo systemctl start jenkins

    * Next, check status of server, it should return 'status=active', if configuration is fine and there are no errors.

            $ sudo systemctl status jenkins

5. Open firewall

    * By default, jenkins runs on 8080 port, lets open it using 'ufw'

            $ sudo ufw allow 8080

    * Check status of ufw to confirm new rules

           $ sudo ufw status

we can see that traffic is allowed to 8080 from anywhere

Output

Status: active


To                         Action      From

--                         ------      ----

OpenSSH                    ALLOW       Anywhere

8080                       ALLOW       Anywhere

OpenSSH (v6)               ALLOW       Anywhere (v6)

8080 (v6)                  ALLOW       Anywhere (v6)


    * If firewall is in-active, we can activate it using below commands.

            $ sudo ufw allow OpenSSH

            $ sudo ufw enable

6. Setting up jenkins

    * To setup jenkins, visit jenkins on its default port. 

        For eg : http://your_server_ip_or_domain:8080

    you should see 'Unlock jenkins' screen, which displays location for initial password.

    * login to ssh terminal and copy this initial password from above location and paste it in 'Administrator password' block

    * Next screen presents option of installing suggested plugins

    * Click on 'Install suggested plugins' option, which will immediately starts installation process.

    * Once plugin installation completes, fill admin user details.

    * After filling all details, click on 'save and continue'. Next, you will see Instance configuration page

    * Click on 'save and finish'. You will get message saying 'Jenkins is ready'

    * Click on 'Start using Jenkins' to get into main jenkins dashboard.




* Once jenkins is successfully installed, we shall configure it as 'Master'

To do that, follow below steps

1. Go to 'Manage Jenkins' option is left panel.


2. Then select 'Configure Global Security' option




3. Select option Random for the settings of Security port for inbound agents. And save the settings.




4. Next, Go to 'Manage Nodes and Cloud' section under 'Manage Jenkins'




5. Click on 'New Node' in left panel and give proper name for node (Eg : Robocop-Slave-01)




6. Select 'Permanent Agent' option and click on 'OK'




7. In Node configuration under 'Remote Root Directory' give valid path.

Eg : /home/ubuntu/jenkins




8. Select the launch method as "Launch agent by connecting it to the master"

9. Select the usage method as "Use this node as much as possible"

10. Click on Save and that’s it. Jenkins master configuration is done. Now you can see the list of nodes like the following. There will red cross mark for the slave node as the node has not been configured yet.




11. Click on the slave node. There you will get the link to download two important files which will be required for configuring the slave. 1. agent.jar, 2. Slave-agent.jnlp
Also note down the first command mentioned under Run from agent command line section

"""""" --------------------------------- """""""
Run from agent command line:

java -jar agent.jar -jnlpUrl http://54.82.33.46:8080/computer/robin-slave-01/jenkins-agent.jnlp -secret 8811aee965305a935afec50ac29b0c69389c67d541a62fe022bdc3b551db7693 -workDir "/home/ubuntu/jenkins"
Run from agent command line, with the secret stored in a file:

echo 8811aee965305a935afec50ac29b0c69389c67d541a62fe022bdc3b551db7693 > secret-file
java -jar agent.jar -jnlpUrl http://54.82.33.46:8080/computer/robin-slave-01/jenkins-agent.jnlp -secret @secret-file -workDir "/home/ubuntu/jenkins"

""""" ----------------------------- """""




Configure Jenkins Slave Node : 

1. Launch the second EC2 instance. Here I am considering ubuntu-18.04 image

2. Update the apt-get once the instance is launched.
$ sudo apt-get update -y
$ sudo apt-get upgrade -y

3. Install open-jdk 8. 
Java will be required to run the jar file which will configure the system as Jenkins slave.
$ sudo apt-get install -y openjdk-8-jdk -y

4. Copy downloaded agent.jar and jenkins-agent.jnlp file from master into slave machine under /home/ubuntu path (or any relevant path). 

We can use 'scp' command to copy from local to aws instance.

format : scp -i <path to pem file> -r <files to be copied> user@public-ip:/<path where files to be copied>

Eg : scp -i /Users/tapan/.ssh/aws-jenkins.pem -r agent.jar jenkins-agent.jnlp ubuntu@ec2-102-26-107-146.compute-1.amazonaws.com:/home/ubuntu/

5. Navigate to directory in slave machine where files are copied and run below command to start slave

format : java -jar -jnlpUrl http://<master node public ip>:8080/computer/<node name>/jenkins-agent.jnlp -secret <secret content> -workDir <path to dir>

$ java -jar agent.jar -jnlpUrl http://54.82.33.46:8080/computer/robin-slave-01/jenkins-agent.jnlp -secret 8811aee965305a935afec50ac29b0c69389c67d541a62fe022bdc3b551db7693 -workDir "/home/ubuntu/jenkins"

we can also run above with secret content stored in file. (secure way)

$ echo 8811aee965305a935afec50ac29b0c69389c67d541a62fe022bdc3b551db7693 > secret-file
$ java -jar agent.jar -jnlpUrl http://54.82.33.46:8080/computer/robin-slave-01/jenkins-agent.jnlp -secret @secret-file -workDir "/home/ubuntu/jenkins"

This will configure the instance as Jenkins slave machine and will show the message Connected.

Note : You might face below error, when you try to run above java -jar command to configure jenkins slave.

""""" ------------ """""
java.io.IOException: http://54.82.33.46:8080/ provided port:39869 is not reachable
at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:311)
at hudson.remoting.Engine.innerRun(Engine.java:724)
at hudson.remoting.Engine.run(Engine.java:540)

"""" ------------ """




To resolve this error, enable 'Web socket' option in configure section of node. 


Run again java -jar command and you should be able to connect to master from node.


Now in dashboard, you will not see any 'red-mark' for node and status as 'connected'





Voila!!... Now your jenkins master-slave set up is ready in AWS environment. Hope you will find post useful. Thanks!!





























Comments

Post a Comment

Popular posts from this blog

OSB-12C : XA Transactional Support feature

-
Image
OSB 12c supports both XA as well as Non-XA transactions. An XA transaction is a "global transaction" that may span multiple resources. It involves a coordinating transaction manager, with one or more databases (or other resources, like JMS) all involved in a single global transaction. XA transactions come from the X/Open group specification on distributed, global transactions. JTA includes the X/Open XA spec, in modified form. XA gets involved when you want to work with multiple resources - 2 or more databases, a database and a JMS connection, all of those plus maybe a JCA resource - all in a single transaction. In this scenario, you'll have an app server like WebSphere or Weblogic or JBoss acting as the Transaction Manager, and your various resources (Oracle, Sybase, IBM MQ JMS, SAP etc) acting as transaction resources.  Your code can then update/delete/publish/whatever across the many resources. When you say "commit", the results are committe...
Read more

Static code analysis of Terraform .tf files using ‘Checkov’ — Secure your infrastructure

-
Image
  Hands on shelf! We can all agree that Terraform is great tool for infrastructure creation and provisioning. When we write terraform .tf files for infra creation we might miss something and we add something which can lead to security issues. Security should be prime factor for any devops related use-case. When I was searching internet for is there any tool which will scan terraform files and give a picture what may go wrong (from security perspective), I came across this very useful tool named ‘Checkov’ Link :  https://www.checkov.io/1.Welcome/Quick%20Start.html Checkov is a   static code analysis tool for scanning infrastructure as code (IaC) files for misconfigurations that may lead to security or compliance problems. Checkov includes more than 750 predefined policies to check for common misconfiguration issues. In this post, I am going to discuss how to do static code analysis of terraform tf files using checkov. step 1 : Install checkov in your machine. $ pip install...
Read more

OSB12c : Schema Validation - Dynamic Validation

-
Image
We can use validate node in OSB to validate schema. There was an option to validate the payload in OSB 11g but it was static, however from 12c there is a new feature included in the OSB validation activity that is validating the data dynamically. This will allow the user to validate the input data or other data dynamically based on certain condition. Thus the process can be stopped at the beginning itself the data validation goes wrong. You have the option to save the validation result or raise an error. Use case: 1. Create a WSDL based service 2. Add validate (dynamic) to validate incoming request structure. 3. If any violation in request structure compared with schema, Error will be raised with proper error code and description. Create new folder and name it as ‘DynamicValidateSchema_UseCase’. Create sub folders structure as shown in below image. For this use case we have created two XSDs and placed it inside Resources -> XSD folder EmployeeDetailsReq...
Read more