Serverless Ansible

This is an experiment, and as you’ll read, I have no idea about what I’m doing. The workflow is easy: I do then I write, so expect rewrites, typos, and cliffhangers :D

Ansible on AWS Lambda: Is it possible? It is useful?? Let’s find out!

BoJack’s references apart, I have no idea, but if it’s possible, I already have a few scenarios where I want it.

First step: Python function calling Ansible API

Let’s write a simple python script using Ansible as a library and let see how far we can go.

So first things first… A hello world (thanks to GitHub for the project name).

#!/usr/bin/env python3

from ansible.parsing.dataloader import DataLoader
from ansible.inventory.manager import InventoryManager

if __name__ == '__main__':
    loader = DataLoader()
    inventory = InventoryManager(loader=loader, sources='localhost')
➜  silver-train (master) ✗ ./main.py
 [WARNING]: Unable to parse localhost, as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

OK… At least it does something. It seems sources want paths with inventories (which makes sense). I thought it will find the inventory as I have an ansible.cfg in at my home.

If I try with the path to my inventory instead of localhost, it does not print warnings, so cool. Let see if I can load the ansible.cfg from a standard path instead of hard-coding the path to my inventory…

There is a ConfigManager inside ansible.config.manager so let’s see what it does:

#!/usr/bin/env python3

from ansible.config.manager import ConfigManager

if __name__ == '__main__':
    config_manager = ConfigManager()
    print(config_manager.data.get_settings())

Oh cool, it prints a huge array with the options it loaded from an ansible.cfg. Nice. But we only need the inventory, and it seems the info is stored in DEFAULT_HOST_LIST so…

#!/usr/bin/env python3

from ansible.config.manager import ConfigManager

if __name__ == '__main__':
    config_manager = ConfigManager()
    print(config_manager.data.get_setting(name='DEFAULT_HOST_LIST'))
➜  silver-train (master) ✗ ./main.py
Setting(name='DEFAULT_HOST_LIST', value=['/Users/alex/Developer/inventory'], origin='/Users/alex/.ansible.cfg', type='pathlist')

Awesome. Let’s use this with the first attempt…

#!/usr/bin/env python3

from ansible.config.manager import ConfigManager
from ansible.parsing.dataloader import DataLoader
from ansible.inventory.manager import InventoryManager

if __name__ == '__main__':
    config_manager = ConfigManager()
    inventory_path = config_manager.data.get_setting(name='DEFAULT_HOST_LIST').value
    loader = DataLoader()
    inventory = InventoryManager(loader=loader, sources=inventory_path)
    print(inventory.get_hosts())
➜  silver-train (master) ✗ ./main.py
[sora, router]

Sora (空, Sky) is my laptop and router is… my router. Yeah, I managed it with Ansible, but that is another story.

This looks nice.

In Ansible, every host is part of at least two groups: all and the ones you defined. If you did not define a group for a host, that host is also part of the ungrouped group. So… Where are my groups?

#!/usr/bin/env python3

from ansible.config.manager import ConfigManager
from ansible.parsing.dataloader import DataLoader
from ansible.inventory.manager import InventoryManager
from ansible.vars.manager import VariableManager

if __name__ == '__main__':
    config_manager = ConfigManager()
    inventory_path = config_manager.data.get_setting(name='DEFAULT_HOST_LIST').value
    loader = DataLoader()
    inventory = InventoryManager(loader=loader, sources=inventory_path)
    variable_manager = VariableManager(loader=loader, inventory=inventory)
    print(variable_manager.get_vars()['groups'])
➜  silver-train (master) ✗ ./main.py   
{'all': ['sora', 'router'], 'ungrouped': ['sora', 'router']}

Much better! Before continuing, let me rewrite everything so it starts to look like an AWS Lambda.

#!/usr/bin/env python3

from ansible.config.manager import ConfigManager
from ansible.parsing.dataloader import DataLoader
from ansible.inventory.manager import InventoryManager
from ansible.vars.manager import VariableManager


def lambda_handler(event=None, context=None):
    config_manager = ConfigManager()
    inventory_path = config_manager.data.get_setting(name='DEFAULT_HOST_LIST').value
    loader = DataLoader()
    inventory = InventoryManager(loader=loader, sources=inventory_path)
    variable_manager = VariableManager(loader=loader, inventory=inventory)
    print(variable_manager.get_vars()['groups'])


if __name__ == '__main__':
    lambda_handler()

It does the same and it looks like a Lambda. Now, who wanna gather some facts?

We need to connect to the machine in a way that is indicated at the inventory (which is already loaded) so it looks like PlaybookExecutor is here to handle this.

A few hours later…

After digging a lot inside the Ansible code and almost lose any hope, I write this horrible thing:

#!/usr/bin/env python3

from collections import namedtuple
from ansible.config.manager import ConfigManager
from ansible.parsing.dataloader import DataLoader
from ansible.inventory.manager import InventoryManager
from ansible.vars.manager import VariableManager
from ansible.executor.playbook_executor import PlaybookExecutor


def lambda_handler(event=None, context=None):
    Options = namedtuple('Options',
                         ['listtags', 'listtasks', 'listhosts', 'syntax', 'connection', 'module_path', 'forks',
                          'remote_user', 'private_key_file', 'ssh_common_args', 'ssh_extra_args', 'sftp_extra_args',
                          'scp_extra_args', 'become', 'become_method', 'become_user', 'verbosity', 'check', 'diff'])
    options = Options(listtags=False, listtasks=False, listhosts=False, syntax=False, connection='ssh',
                      module_path=None, forks=100, remote_user='root', private_key_file=None,
                      ssh_common_args=None, ssh_extra_args=None, sftp_extra_args=None, scp_extra_args=None,
                      become=False,
                      become_method='sudo', become_user='root', verbosity=None, check=False, diff=None)

    config_manager = ConfigManager()
    inventory_path = config_manager.data.get_setting(name='DEFAULT_HOST_LIST').value
    loader = DataLoader()
    inventory = InventoryManager(loader=loader, sources=inventory_path)
    variable_manager = VariableManager(loader=loader, inventory=inventory)
    print(variable_manager.get_vars()['groups'])

    pbex = PlaybookExecutor(playbooks=['playbook.yml'],
                            inventory=inventory,
                            variable_manager=variable_manager,
                            loader=loader,
                            options=options,
                            passwords=None,
                            )

    results = pbex.run()


if __name__ == '__main__':
    lambda_handler()

Few things here, right? The options were a mix between a SO post and trial and error until I found all the arguments. It looks like nobody is trying to do this. Why? :P

The playbook.yml is quite simple:

---
- hosts: all

  gather_facts: True

  tasks:
    - debug:

Does it work?

➜  app (master) ✗ ./lambda.py 
{'all': ['sora', 'router'], 'ungrouped': ['sora', 'router']}

PLAY [all] ***********************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************
fatal: [router]: UNREACHABLE! => {"changed": false, "msg": "[Errno None] Unable to connect to port 22 on 192.168.1.1", "unreachable": true}
ok: [sora]

TASK [debug] *********************************************************************************************************
ok: [sora] => {
    "msg": "Hello world!"
}
	to retry, use: --limit @/Users/alex/Developer/src/github.com/alexsaezm/silver-train/app/playbook.retry

PLAY RECAP ***********************************************************************************************************
router                     : ok=0    changed=0    unreachable=1    failed=0   
sora                       : ok=2    changed=0    unreachable=0    failed=0   

Yeah! Now, let’s move this to AWS and see what happens. It won’t work, for sure.

Uploading Ansible to an AWS Lambda

Let’s zip the whole project folder and upload it to AWS.

➜  silver-train (master) ✗ ls
LICENSE          README.md        app              requirements.txt venv
➜  silver-train (master) ✗ cd venv/lib/python3.7/site-packages
➜  site-packages (master) ✗ zip -r9 ../../../../silver-train.zip .
➜  site-packages (master) ✗ cd ../../../../app
➜  app (master) ✗ zip -g ../silver-train.zip lambda.py playbook.yml 
  adding: lambda.py (deflated 65%)
  adding: playbook.yml (deflated 6%)
➜  app (master) ✗ cd ..
➜  silver-train (master) ✗ ls -hl silver-train.zip
-rw-r--r--  1 alex  staff    27M Dec 12 23:29 silver-train.zip
➜  silver-train (master) ✗ aws --profile Personal lambda update-function-code --function-name silver-train --zip-file fileb://silver-train.zip
{
    "FunctionName": "silver-train",
    "FunctionArn": "arn:aws:lambda:eu-west-1:784744605207:function:silver-train",
    "Runtime": "python3.6",
    "Role": "arn:aws:iam::784744605207:role/silver-train",
    "Handler": "app.lambda.lambda_handler",
    "CodeSize": 27852424,
    "Description": "A starter AWS Lambda function.",
    "Timeout": 3,
    "MemorySize": 128,
    "LastModified": "2018-12-12T22:30:28.785+0000",
    "CodeSha256": "fld/HSTMfO8IgXKPRhi+uYxaIKy54gJTrvjxcxVskFs=",
    "Version": "$LATEST",
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "RevisionId": "0c969d1e-3f72-417e-8223-abaf09d13856"
}

Let’s try it!

Crap!

{
  "errorMessage": "module initialization error"
}
START RequestId: be0bfb98-fe5d-11e8-bbcf-f131fe57ad67 Version: $LATEST
module initialization error: Invalid settings supplied for DEFAULT_LOCAL_TMP: Unable to create local directories(/home/sbx_user1060/.ansible/tmp): [Errno 30] Read-only file system: b'/home/sbx_user1060'
Traceback (most recent call last):
  File "/var/task/ansible/utils/path.py", line 76, in makedirs_safe
    os.makedirs(b_rpath, mode)
  File "/var/lang/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/var/lang/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/var/lang/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: b'/home/sbx_user1060'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/ansible/config/manager.py", line 487, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/var/task/ansible/config/manager.py", line 439, in get_config_value_and_origin
    value = ensure_type(value, defs[config].get('type'), origin=origin)
  File "/var/task/ansible/config/manager.py", line 91, in ensure_type
    makedirs_safe(value, 0o700)
  File "/var/task/ansible/utils/path.py", line 81, in makedirs_safe
    raise AnsibleError("Unable to create local directories(%s): %s" % (to_native(rpath), to_native(e)))
ansible.errors.AnsibleError: Unable to create local directories(/home/sbx_user1060/.ansible/tmp): [Errno 30] Read-only file system: b'/home/sbx_user1060'


END RequestId: be0bfb98-fe5d-11e8-bbcf-f131fe57ad67
REPORT RequestId: be0bfb98-fe5d-11e8-bbcf-f131fe57ad67	Duration: 290.69 ms	Billed Duration: 300 ms 	Memory Size: 128 MB	Max Memory Used: 33 MB	
module initialization error
Invalid settings supplied for DEFAULT_LOCAL_TMP: Unable to create local directories(/home/sbx_user1060/.ansible/tmp): [Errno 30] Read-only file system: b'/home/sbx_user1060'
Traceback (most recent call last):
  File "/var/task/ansible/utils/path.py", line 76, in makedirs_safe
    os.makedirs(b_rpath, mode)
  File "/var/lang/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/var/lang/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/var/lang/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: b'/home/sbx_user1060'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/ansible/config/manager.py", line 487, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/var/task/ansible/config/manager.py", line 439, in get_config_value_and_origin
    value = ensure_type(value

Oh! OK, that is easy. In a Lambda, you cannot write outside /tmp, but by default, Ansible uses the home of the user.

As we can use ansible.cfg files, we can use one here to change the behavior of the lambda:

[defaults]
remote_tmp     = /tmp
local_tmp      = /tmp
host_key_checking = False

Let’s add this to the zip and try again.

START RequestId: d89a7314-fe66-11e8-b37e-2f08a6cdede6 Version: $LATEST
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source
 [WARNING]: No inventory was parsed, only implicit localhost is available
{'all': [], 'ungrouped': []}
END RequestId: d89a7314-fe66-11e8-b37e-2f08a6cdede6
REPORT RequestId: d89a7314-fe66-11e8-b37e-2f08a6cdede6	Duration: 6534.99 ms	Billed Duration: 6600 ms 	Memory Size: 128 MB	Max Memory Used: 32 MB	

Awesome, it works. Well, almost:

{
  "errorMessage": "Unable to use multiprocessing, this is normally caused by lack of access to /dev/shm: [Errno 38] Function not implemented",
  "errorType": "AnsibleError",
  "stackTrace": [
    "  File \"/var/task/lambda.py\", line 35, in lambda_handler\n    passwords=None,\n",
    "  File \"/var/task/ansible/executor/playbook_executor.py\", line 60, in __init__\n    self._tqm = TaskQueueManager(inventory=inventory, variable_manager=variable_manager, loader=loader, options=options, passwords=self.passwords)\n",
    "  File \"/var/task/ansible/executor/task_queue_manager.py\", line 107, in __init__\n    raise AnsibleError(\"Unable to use multiprocessing, this is normally caused by lack of access to /dev/shm: %s\" % to_native(e))\n"
  ]
}

It’s been like 6 hours since I begin, and for an unknown reason, awscli is giving me connection errors. Sounds like it’s bed time.

Stay in tune for the next episode!


I'm Alex · I do things ·