Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications. Automate in a language that approaches plain English, using SSH, with no agents to install on remote systems.

With Ansible and its amazing module ecosystem, you describe what needs to be accomplished (i.e. declarative), rather than describing how to accomplish each step (i.e. imperative).

Ansible is Python 2 based and uses SSH to communicate with remote hosts; the only prerequisites.

Development Setup⌗

LXC (Lexy) Primer⌗

In David Cohen’s Ansible 2 course, to keep the lab setup clean, suggests the use of LXC containers; a light and convenient way of spinning up isolated operating environments. Using LXC scales nicely, even if you’re not running Linux, simply create a single Linux VM (e.g. using Hyper-V, VirtualBox, VMWare). This single VM willbe capable of running several LXC containers.

LXC was first released in 2008, and is an OS level virtualisation method for running multiple Linux systems, using a single Linux kernel, and can underpin higher level layers such as Docker. To get going:

  1. Install with yum install epel-release then yum install lxc lxc-extra lxc-templates. lxc-extra includes helpful scripts such as lxc-ls. Verify with lxc-checkconfig.
  2. Create containers like this lxc-create -t centos -n web1. The container templates -t available can be found in /usr/share/lxc/templates. This will download the base file system for the chosen template, dumping it in /var/cache e.g. /var/cache/lxc/centos/x86_64/7/. A temporary root password to actually log into the container is placed here /var/lib/lxc/web1/tmp_root_pass.
  3. Spark it up with lxc-start -n ansibley -d. The -d runs it in daemon mode, preventing it from taking over stdout and stdin.
  4. Get a state of play with lxc-ls -f
  5. When you’re ready to jump into a container, lxc-attach -n web1
  6. Ensure that Ansibles only dependency python 2.7 is on all participating machines. This can be done using a playbook, but keep things low tech for now. lxc-attach each container, and do a yum install python27.

Installation⌗

Follow the docs

Bleeding edge:

git clone https://github.com/ansible/ansible
cd ansible
git submodule update --init --recursive
dnf install python-jinja2 python-paramiko python-yaml sshpass
source ./hacking/env-setup

Managed Node Requirements⌗

Only requirements are Python and OpenSSH.

Python 2 (or 3)⌗

If python --version doesn’t spit out a sane version e.g. 2.7.5, then install it:

yum install python

OpenSSH⌗

CentOS sshd is already installed. To configure it:

  1. firewall-cmd --permanent --add-service=ssh
  2. chkconfig sshd on
  3. systemctl start sshd.service
  4. netstat -tupln | grep :22

Static IP⌗

Edit /etc/sysconfig/network-scripts/ifcfg-eth0 (or that relevent to the network device of concern):

DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.124.122
NETMASK=255.255.255.0
HWADDR=52:54:00:55:1b:21
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no

SSH Security⌗

Create ansible accounts on all participating nodes, and give it noprompt sudo.

adduser ansible
passwd ansible
visudo

Below the root user specification add:

ansible ALL=(ALL) NOPASSWD: ALL

When running a playbook over SSH (e.g. from Jenkins or cron), to prevent credential prompts, ensure that the ansible user on the control node has a key pair (/home/ansible/.ssh/id_rsa), and that it is exchanged with all other partipating nodes:

$ su - ansible
$ ssh-keygen
$ ssh-copy-id ansible@web1.bencode.net
$ ssh-copy-id ansible@web2.bencode.net
$ ssh-copy-id ansible@db1.bencode.net
$ ssh-copy-id ansible@db2.bencode.net
$ ssh-copy-id localhost #ssh needs this

Alternatively, on the controller, copy (clipboard) the public key:

$ cat ~/.ssh/id_rsa.pub

And on the participating machines, copy and paste it into the ~/.ssh/authorized_keys

ansible.cfg⌗

The main configuration. The probe path to find one is:

  1. Check the Ansible_Config env var.
  2. Nope? ansible.cfg in current dir.
  3. Nope? ansible.cfg in home dir.
  4. Nope? /etc/ansible/ansible.cfg

Priming a target host⌗

Typically new hosts need to be “primed” to support being an Ansible target; installing a Python 2 runtime, and bedding in the SSH public key of user that Ansible will be running under. Suprisingly this initial setup can itself be achieved using a playbook (tip: disabling gather_facts prevents Ansible reaching out to Python, which is non-existant at this point).

prepare_ansible_target.yml⌗ 

---
# Run with ansible-playbook <filename> -k
#   (make sure to add the IPs of machines you want to manage to /etc/ansible/hosts first)

- hosts: all
  gather_facts: False
  remote_user: oper01
  become: yes
  become_user: root
  become_method: sudo

  tasks:
    - name: Update Packages
      raw: (apt-get update && apt-get -y upgrade)

    - name: Install Python 2
      raw: test -e /usr/bin/python || (apt-get update && apt-get install -y python)

    ```
    - name: Fancy way of doing authorized_keys
      authorized_key: user=ansible
                      exclusive=no
                      key="{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
     ```

    - name: COMMON | Set environment
      blockinfile:
        dest: /etc/environment
        block: |
          LC_ALL=en_US.UTF-8
          LANG=en_US.UTF-8          
      register: newenv

    - block:
      - name: COMMON | Generate locales
        raw: locale-gen en_US.UTF-8

      - name: COMMON | Reconfigure locales
        raw: update-locale LANG=en_US.UTF-8
      # only run this task block when we've just changed /etc/environment
      when: newenv.changed

Hosts (Inventory) File⌗

Ansible provides it own base inventory file at /etc/ansible/hosts, which is nicely documented. Create a new inventory file in your working dir, similar to the following:

[allservers]
192.168.124.120
192.168.124.121
192.168.124.122

[web]
192.168.124.120
192.168.124.121

[db]
192.168.124.122

Ansible can be pointed at specific inventory files with the -i switch like this:

ansible-playbook -i inventory-file

Adhoc Commands⌗

A neat way of running one off tasks. Syntax is:

ansible <group/machine> -m <module> -a <args> [-k for pass prompt]

Examples⌗

Run an adhoc one liner:

$ ansible allservers -a "uname -a" -i inventory
192.168.124.120 | SUCCESS | rc=0 >>
Linux web1.local 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

192.168.124.122 | SUCCESS | rc=0 >>
Linux server1.example.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

192.168.124.121 | SUCCESS | rc=0 >>
Linux web2.local 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Very nice. This scales to any task, such as registering the EPEL (Extra Packages for Enterprise Linux) package repo for the fleet of servers.

$ ansible web -a "wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm" -i inventory

Ansible has an insane library of modules for accomplishing most tasks. Lets take the ping module for a run on a specific host (this is not an actual ICMP ping, but does verify Ansible is able to run on the target):

$ ansible 192.168.124.120 -m ping -i inventory
192.168.124.120 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Install nginx across all hosts in the web group, using the package module. The -b switch here is the become option, which will by default elevate to root using su.

$ ansible web -m package -a "name=nginx state=installed" -i inventory -b
192.168.124.121 | SUCCESS => {
    "changed": true,
    "msg": "warning: /var/cache/yum/x86_64/7/base/packages/libunwind-1.2-2.el7.x86_64.rpm: Header V3 RSA/SHA256
...omitted...

Now run a one liner yum on every server to check nginx is indeed installed- only the two web hosts should return SUCCESS:

$ ansible allservers -a "yum list installed nginx" -i inventory
[WARNING]: Consider using yum module rather than running yum

192.168.124.121 | SUCCESS | rc=0 >>
Loaded plugins: fastestmirror, langpacks
Installed Packages
nginx.x86_64                        1:1.12.2-2.el7                         @epel

192.168.124.120 | SUCCESS | rc=0 >>
Loaded plugins: fastestmirror, langpacks
Installed Packages
nginx.x86_64                        1:1.12.2-2.el7                         @epel

192.168.124.122 | FAILED | rc=1 >>
Loaded plugins: fastestmirror, langpacksError: No matching Packages to listnon-zero return code

Showing off the -B (timeout) and -P (poll time) arguments:

ansible allservers -B 2400 -P 5 -a "apt-get update && apt-get upgrade -y" -u root

Playbooks⌗

Core concepts:

  • Tasks: an individual piece of desired state, such as ensuring the latest version of a package is installed. If useful tasks can be grouped into blocks.
  • Templates: a base file that can have state injected into it, annotated with handlebar style tags thanks to the Jinja template engine e.g. {{ site_name }}
  • Handlers: a hook that is triggered as late in the playbook run as possible (e.g. if multiple handlers to restart nginx are wired up, Ansible will minimise the number of restarts by deferring the trigger until the very last task that requires a restart is complete).
  • Roles: clumps of reusable tasks, templates and handlers; allowing higher level of abstraction (e.g. the web role, caching role, database role and so on).

Playbooks are run with the ansible-playbook command, like so:

ansible-playbook configure_fleet.yml -i inventory

Playbook Structure⌗

As your Ansible project grows, a common tree structure to help keep things organised is as follows:

playbook.yml              # top level playbook
group_vars/
    all                   # the main file for defining variables
roles/
    role1/                # each role (e.g. web, database, common, cache)
        files/            # role-specific files which will be copied to the remote machine
        handlers/         # role-specific handlers
            main.yml      # handler file
        meta/             # files that establish role dependencies
        tasks/            # role-specific tasks
            main.yml      # task file
        templates/        # role-specific templates
        vars              # role-specific variables, although recommended to use group_vars/all instead

As you would expect, this takes advantage of some of Ansibles default path probing, such as the top level group_vars containing global variable definitions, templates contain jinja2 templates resolvable when using the template module, and so on.

Dave has put together two (one in pure Python and one in Ansible) handy scripts for generating this tree. The vanilla Python version needs a destination directory, and one or more role names, for example:

./create_playbook.py ./boilerplate-playbook web database common

A real world playbook (playbook.yml) might look like this:

---
- name: Database Setup
  hosts: dbservers
  remote_user: ansible
  roles:
    - common
    - database

- name: Web Server Setup
  hosts: webservers
  remote_user: ansible
  roles:
    - common
    - web

Sample handlers in /roles/role1/handlers/main.yml:

---
- name: reload nginx
  service: name=nginx state=reloaded

- name: reload systemd
  command: systemctl daemon-reload

Tasks⌗

A single chore to be acheived. Here are a few tasks:

tasks:
  - name: NGINX | Remove default vhost
    file: path=/etc/nginx/sites-enabled/default state=absent

  - name: NGINX | Start
    service: name=nginx state=started
    register: nginx_started

  - name: STAT | Check dir exists
    stat: path={{ foo_directory }}
    register: foo_directory

The third task highlights the ability to store the result of a task in a variable, in this case the stat task result, which could be used for a conditional when directive like so:

when: foo_directory.stat.exists == False

Blocks⌗

Tasks can be grouped into blocks, for example:

- block:
    - name: PostgreSQL | Setup pg_hba.conf, allow connections from web servers
      template: src=pg_hba.conf dest={{ PGA_HB_CONF }}
      notify:
        - restart postgres
    - name: PostgreSQL | Listen on all IPs
      lineinfile:
        dest: "{{ PG_CONF }}"
        line: listen_addresses='*'
      notify:
        - restart postgres
  rescue:
    - name: Only run when a task blows up
      debug: msg="boom!"
  always:
    - name: Always always run
      debug: msg="Regardless of what happened, I'm running"
  when: hostvars[groups['webservers][0]]['inventory_hostname'] != inventory_hostname

Blocks can be a useful way to deal with control flow, exceptions and cleanup. They always start with the block keyword, and support the rescue and always hooks.

Templates (Jinja2)⌗

For templating Ansible leverages the Python Jinja2 templating engine.

Syntax⌗

  • {% ... %} for statements
  • {{ ... }} for expressions to print to the template output
  • {# ... #} for comments not included in the template output
  • # ... ## for line statements

In action snippet:

local all postgres peer

    {% for host in groups['webservers'] %}
    host all all {{ hostvars[host]['inventory_hostname']}}/32 md5
    {% endfor %}

Loops⌗

Great support for control structures exists, supporting the vanilla Python style for...in loop:

{# A list of webservers #}
{% for server in groups['webservers'] %}
web{{ loop.index }}.bm4cs.io
{% endfor %}

{# A navigation menu #}
<nav class="menu">
  <ul>
    {% for item in navigation %}
      <li><a href="{{ item.href }}">{{ item.label }}</a></li>
    {% endfor %}
  </ul>
</nav>

Filters⌗

Jinja2 filters are (awesome!!) built-in functions available to all templates. String functions, list and general collection functions, arithmetic, and many more.

groupby

<ul>
{% for group in persons|groupby('gender') %}
    <li>{{ group.grouper }}<ul>
    {% for person in group.list %}
        <li>{{ person.first_name }} {{ person.last_name }}</li>
    {% endfor %}</ul></li>
{% endfor %}
</ul>

join

{{ [1, 2, 3]|join('|') }}
    -> 1|2|3

{{ [1, 2, 3]|join }}
    -> 123

{ webservers|join(', ') }}

Escaping⌗

When you just want to dump out something completely, put them inbetween some raw and endraw tags.

Variables and Facts⌗

Variables are scoped as Global (visible across playbooks), Per-Play and Per-Host. They are defined using the register keyword, and referenced with {{ varname }} double handlebars style syntax.

Some common ways to define variables:

  • --extra-vars on the command line
  • in the playbook as vars:
  • in the global_vars/all file
  • import a YAML file full of variables with include_vars: foo_vars.yml
  • host or host group scoped vars can be defined in the inventory
  • role specific vars in /role/rolename/vars/

The existance of variables can be tested with a when:

- name: Check foo_var is defined
  debug: msg="Yes, foo_var is defined"
  when: foo_var is defined

Ansible will probe hosts defined in the inventory, and sniff out “facts” about them. Why is this useful? The package module for example, allows you to work with packages in an agnostic manner, insulating you from the nuances of yum, apt, package, pacman or whatever whacky package manager of the day is. Thanks to facts, the package module can dynamically target a variety of package managers and operating systems.

Facts can be leveraged throughout plays, and are represented as a dictionary called hostvars (e.g. hostvars[host]['fact_name']). Fact gathering can be disabled gather_facts: False at the playbook level; this will break modules (such as package) that rely on facts.

Useful tip: The setup module, which is implicitly invoked by playbooks to do fact gathering, can be manually called to dump facts to stdout.

ansible allservers -m setup -i inventory

Some fact testing examples:

- name: All the things
  debug: msg={{ hostvars[inventory_hostname] }}

    - name: However this machine is listed in the inventory
      debug: msg={{ inventory_hostname }}

    - name: Just the hostname
      debug: msg={{ ansible_hostname }}

    - name: Just the primary IPv4 address
      debug: msg={{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}

Built-in Variables⌗

  • hostvars: access facts and variables from other hosts (for current host key the dictionary on inventory_hostname)
  • group_names: list of groups current host is a member of
  • groups: all groups in the inventory
  • inventory_hostname: current hostname that ansible is executing tasks on
  • environment:

Task Results and Control Flow⌗

The outcome of a task can be set as a variable using register, for example:

tasks:
  - name: start nginx
    service: name=nginx state=started
    register: nginx_started

The resulting data structure can be used for making downstream decisions:

- stat: path={{ app_dir }}
  register: app_dir

- name: Another task
  module: arg1=123 arg2=456
  when: app_dir.stat.exists == False

Globally Scoped⌗

For globally scoped variables, use the group_vars/all YAML file, for example:

---
aws_region: ap-southeast-2
aws_pubkey_name: "vim-ansible"

db_master_name: "ThinkDb"
db_master_user: "su"
db_master_pass: "chaching"
db_conn_string: "Initial Catalog={{ db_master_name }}; User={{ db_master_user }}; Password={{ db_master_password }}"

Playbook Scoped⌗

As vars: above the task blocks.

--
- hosts: web
  vars:
    site_name: "benjansible"
    site_title: "Hope."
    site_url: "www.bencode.net"

  tasks:
    - name: Install nginx.
      package: name=nginx state=latest

Inventory File Variables⌗ 

[webservers]
web1.evilcorp.com server_name="yin.evilcorp.com"
web2.evilcorp.com server_name="yang.evilcorp.com"

[webservers:vars]
web_home_path=/var/www/home/

Command Line Vars⌗

Using --extra-vars during invocation. This will clobber/override any variables that may get defined with the same name. It can accept space delimitered strings (example below), or quoted JSON.

ansible-playbook install_nginx.yml --extra-vars "version=1.01 website_url=www.bencode.io"

Control Flow⌗

when⌗

Task runs if the “when” clause is truthy. Supported tests are consistent with Jinja2.

- name: If host uses systemd, create unit file
  template: src=foo.service dest=/etc/systemd/system/foo.service
  when: systemd_installed.stat.exists

block:
  - name: PostgreSQL | Setup pg_hba.conf, allow connections from web servers
    template: src=pg_hba.conf dest={{ PGA_HB_CONF }}
    notify:
      - restart postgres
  - name: PostgreSQL | Listen on all IPs
    lineinfile:
      dest: "{{ PG_CONF }}"
      line: listen_addresses='*'
    notify:
      - restart postgres
when: hostvars[groups['webservers][0]]['inventory_hostname'] != inventory_hostname

register⌗

Stores the output of a task into a variable. Useful for downstream control flow.

- stat: path=/var/www/public/foo
  register: foo_web_dir

- shell: whoami
  register: user_name

Consuming variables is easy with double handlebar syntax:

{{ user_name.stdout }}
{{ user_name.stderr }}
{{ foo_web_dir.stat.exists == False }}

Iteration⌗

When iterators (e.g. with_items, with_dict) are used with a when, the when is evaluated for each item.

with_items⌗

Iterates over a list:

- name: Install essentials
  package: name={{ item }} state=present
  with_items:
    - wget
    - vim
    - curl
    - tmux

Supports list variables too.

with_items: "{{ my_list }}"

with_dict⌗

Iterates over a YAML hash:

---
fruits:
  banana:
    awesome: 7
    cals: 89
  kiwi:
    awesome: 6
    cals: 61
  apple:
    awesome: 10
    cals: 52

Using with_dict like this:

debug: var="{{ item.key }} is {{ item.value.awesome }} out of 10, and has about {{ item.value.cals }} per serve."
with_dict: "{{ fruits }}"

with_nested⌗

Nested loops.

- name: Give users access to multiple databases
  mysql_user: name={{ item[0] }} priv={{ item[1] }}.\*:ALL append_privs=yes password=foo
  with_nested:
    - ["alice", "bob", "dave"]
    - ["clientdb", "employeedb", "providerdb", "testdb"]

changed_when⌗

Gives you finer grained control over how a task reports that it has indeed changed. Some modules like raw or shell always report a change, because Ansible doesn’t actually know if they resulted in a side effect or not.

- command: "apt-get upgrade -y"
  register: apt_upgrade
  changed_when: "'0 upgraded, 0 newly installed' not in apt_upgrade.stdout"

failed_when⌗

Same idea as changed_when, when its not clear if a task actually succeeded or failed:

- command: "ls /foo/bar/baz"
  register: listing_output
  failed_when: "'baz' not in listing_output.stderr"
  ignore_errors: yes

When you actually expect a task to produce stderr, you can ignore them with ignore_errors.

Execution Strategies⌗

The approach Ansible will take to run plays across hosts.

Serial (default)⌗

One task at a time, across all servers. A serial argument allows you to define a batch size either as an absolute number or a percentage.

name: Do things 2 servers at a time
hosts: middleware-servers
serial: 2

Or a percentage:

name: Do stuff on 30% of servers at a time
hosts: web-servers
serial: 30%

And finally multiple batch sizes are supported:

name: Do tasks on a single server, then 5 servers, then 10% of the servers
hosts: web-servers
serial:
  - 1
  - 5
  - 10%

Free (go nuts)⌗

Provided enough resources, will go full throttle on hosts as fast as possible.

- hosts: all
  strategy: free
  tasks:

Failure Percentage⌗

If a certain failure threshold is met (e.g. 5%), abort everything.

- hosts: nodejs-servers
  max_fail_percentage: 5
  serial:
    - 10%

Modules⌗

Ansible provides a HUGE ecosystem of modules. A module is specified after a given tasks` name, for example:

---
- hosts: webserver
  vars:
    site_url: bm4cs.io
  tasks:
    - name: Install nginx
      package: name=nginx state=latest

    - name: Create website dir
      file: path="/var/www/{{ site_name }}" state=directory mode=0755

    - name: Create nginx config
      template:
        src: "templates/website.conf"
        dest: "/etc/nginx/conf.d/{{ site_name }}.conf"
      notify:
        - restart nginx

Can see the package, file and template modules in action, and the arguments they are fed.

Bread & Butter Modules⌗

Package Management⌗

Could use pacman, apt, yum, pkg, rpm directly, dont. Use package instead.

Files and Directories⌗

  • template: run a jinja template, and copy the result to a location
  • file: CRUD files and directories. Important properties, are state (file, link, directory, hard, touch, absent) state=absent will delete, recurse,
  • lineinfile and blockinfile: insert/update/remove a line (or block) of text content.
  • copy: copying local files to target/s
  • fetch: copying remote files from target/s to local
  • stat: existance check

System⌗

  • service: service management agnostic of init system (e.g. SysV, systemd, upstart). The state property (started, stopped, restarted, reloaded).
  • user
  • group
  • cron
  • hostname
  • authorized_keys: asserts the existance of, if not adds.
  • iptables: managing rules
  • modprobe: kernel mods
  • kernel_blacklist
  • gluster_volume
  • lvm
  • zfs

Miscellaneous⌗

  • raw: invokes a down and dirty SSH command
  • synchronize: basically rsync
  • get_url: can pull from lots of protos (http, ftp, and more)
  • unarchive: unpack things on target/s
  • ec2: AWS compute management
  • rds: AWS DB management
  • lxc_container: manage all things LXC container related

Cool things⌗

Debugging⌗

The debug module to the rescue.

- name: Debugging dude
  debug: msg="Boom boom!"

Debug and register compliment each other, when you want to verify what data structure a task is outputting.

- shell: /usr/bin/uptime
  register: tehuptime

- name: Debug uptime
  debug: var=tehuptime

Verbosity⌗

A really handy way to dig into the underlying commands that Ansible actually throws at the inventory hosts, is by enabling very very verbose logging. When running Ansible can pass the -v or -vv or the very very verbose -vvv options.

The debug module supports filtering on the verbosity level:

- name: Debug verbosity 1
  debug: var=myuptime verbosity=1

- name: Debug verbosity 2
  debug: var=myuptime verbosity=2

- name: Debug verbosity 3
  debug: var=myuptime verbosity=3

Document!⌗

Documenting what the playbook expects in group_vars/all is a good practice, for example:

# Exactly one database server
# One or more web server/s
# systemd on the web server/s

Allowing the playbook to make some assumptions can greatly simplify things. For example, knowing that we are dealing with a single DB server, enables us to use the IP of the first DB server in inventory, for later binding into our web servers database connection strings, example:

db_server: "{{ hostvars[groups['dbservers'][0]]['ansible_default_ipv4']['address'] }}"

Same Host Multiple Roles⌗

The same host can act as multiple roles, example:

[web]
10.1.3.1

[database]
10.1.3.1

Variable Things⌗

It’s fine to consume variables just after declaring them, example:

PG_VERSION: "9.3"
PG_HBA_CONF: /etc/postgresql/{{ PG_VERSION }}/main/pg_hba.conf

Template a block of multiline text⌗ 

- name: /etc/enviroment proxy server settings
  blockinfile:
    dest: /etc/environment
    block: "{{ lookup('template', 'etc-profile') }}"
    marker: "# {mark} ANSIBLE MANAGED BLOCK"

Private SSH key passphrases⌗

Run ssh-agent.

Resources⌗

Dave Cohen’s Hands On Ansible GitHub Repo