CSV files on each host contain certain figures and I need to add them all up.
To that end, I find the matching files first, and invoke awk to parse the CSV content and output the per-host total.
Where I'm stuck is summing up the totals across all machines:
- hosts: all
gather_facts: False
tasks:
- name: Find input
find:
path: /somepath
pattern: 'logbatch*.txt'
recurse: True
register: found
- set_fact:
inputs: "{{ found.files | map(attribute = 'path') }}"
- command: >-
awk -F, '{ t += $NF } END { print t }' {{ inputs | join(' ') }}
register: total
- debug: var=total.stdout
- debug:
msg: >-
{{ hostvars | json_query('[].total.stdout') }}
run_once: True
The first debug outputs each machine's total, as expected.
I then expected to pipe the output of json_query into sum, but the json_query returns a None (which debug then outputs as empty string).
Extract the total.stdout from the hostvars
- debug:
var: sum
run_once: true
vars:
sum: "{{ ansible_play_hosts | map('extract', hostvars, ['total', 'stdout'])
| map('int')
| sum }}"
Example of a complete playbook for testing
- hosts: test
tasks:
- find:
path: /tmp/ansible
pattern: logbatch*.txt
register: found
- command: >-
awk -F, '{ t += NF } END { print t }' {{ inputs | join(' ') }}
register: total
vars:
inputs: "{{ found.files | map(attribute = 'path') }}"
- debug:
var: total.stdout
- debug:
var: sum
run_once: true
vars:
sum: "{{ ansible_play_hosts | map('extract', hostvars, ['total', 'stdout'])
| map('int')
| sum }}"
Q: Is the traversal of hostvars too expensive for the tricks like this?
A: No. A 100 remote nodes should be fine. For example, given 100 FreeBSD jails
(env) > ansible-inventory -i iocage.yml --graph | wc -l
102
The simplified playbook pb.yml takes 10s to connect 100 remote nodes on LAN and register each total. The summary is immediate.
- hosts: all
vars:
ansible_user: admin
ansible_python_interpreter: auto_silent
tasks:
- shell: cat /etc/passwd | wc -l
register: total
- debug:
var: sum
run_once: true
vars:
sum: "{{ ansible_play_hosts | map('extract', hostvars, ['total', 'stdout'])
| map('int')
| sum }}"
gives (dynamic inventory cached)
(env) > ANSIBLE_STDOUT_CALLBACK=community.general.timestamp ansible-playbook -i iocage.yml pb.yml
PLAY [all] ************************************************* 23:20:32
TASK [shell] *********************************************** 23:20:32
changed: [0dee0f47]
changed: [03082ef5]
changed: [07b8c5aa]
changed: [0c06dd70]
changed: [11f10dc9]
changed: [18a86a8b]
changed: [1a964e79]
changed: [0f1acecf]
changed: [1e61c2ea]
changed: [1da9116c]
changed: [11f39acc]
changed: [169fa0d4]
changed: [132835a4]
changed: [210db3aa]
changed: [29acbb98]
changed: [2b531944]
changed: [239b8f66]
changed: [306aa364]
changed: [2dbad49d]
changed: [309b6b14]
changed: [2faf02cd]
changed: [39bcba32]
changed: [310d85c6]
changed: [32d44e91]
changed: [3c898cc1]
changed: [2b6f9857]
changed: [4363f7ed]
changed: [3f7c1613]
changed: [43d3034e]
changed: [4826bdac]
changed: [51b6fdba]
changed: [52d19d22]
changed: [544838a2]
changed: [54c63e0b]
changed: [5552ae88]
changed: [56770bc1]
changed: [5a6d7f47]
changed: [44831111]
changed: [6036ac22]
changed: [5d5b6ef3]
changed: [642e40a8]
changed: [68053653]
changed: [6c54400e]
changed: [6ef2b93b]
changed: [6f347bd0]
changed: [718089eb]
changed: [73e4049f]
changed: [853d0e13]
changed: [8559596f]
changed: [91f2043c]
changed: [9723a725]
changed: [7f37f1df]
changed: [7adac137]
changed: [8c921bde]
changed: [9babf616]
changed: [9f32d23d]
changed: [9be86289]
changed: [92d9a51f]
changed: [a7e312ef]
changed: [b312ea85]
changed: [b1d15cba]
changed: [b81706ac]
changed: [9b75e733]
changed: [bde10ef8]
changed: [bbfd8cd7]
changed: [bff5f74a]
changed: [c2645edb]
changed: [c6ad222b]
changed: [c92b84b8]
changed: [d3113f35]
changed: [d341b47d]
changed: [d449072f]
changed: [d47ade5b]
changed: [d52ba107]
changed: [d5736544]
changed: [d6f33565]
changed: [db2f388a]
changed: [df2287d4]
changed: [ba54e1ac]
changed: [e26ccd71]
changed: [e47c100a]
changed: [e89ed83d]
changed: [eb51ec19]
changed: [ea40e89b]
changed: [e6213819]
changed: [edab4bf1]
changed: [eec72102]
changed: [f15871f2]
changed: [ef17800c]
changed: [ef341a9d]
changed: [f0528834]
changed: [f5653015]
changed: [fa1ebfe6]
changed: [f479a59b]
changed: [fa465a90]
changed: [fa6dda68]
changed: [fd67d48c]
changed: [fd7cdce7]
changed: [fe3f39e5]
changed: [f3e5c95b]
TASK [debug] *********************************************** 23:20:42
ok: [03082ef5] =>
sum: 2800
PLAY RECAP ************************************************* 23:20:42
03082ef5 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
07b8c5aa : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
0c06dd70 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
0dee0f47 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
0f1acecf : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
11f10dc9 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
11f39acc : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
132835a4 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
169fa0d4 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
18a86a8b : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
1a964e79 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
1da9116c : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
1e61c2ea : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
210db3aa : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
239b8f66 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
29acbb98 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
2b531944 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
2b6f9857 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
2dbad49d : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
2faf02cd : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
306aa364 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
309b6b14 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
310d85c6 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
32d44e91 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
39bcba32 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
3c898cc1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
3f7c1613 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
4363f7ed : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
43d3034e : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
44831111 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
4826bdac : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
51b6fdba : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
52d19d22 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
544838a2 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
54c63e0b : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
5552ae88 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
56770bc1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
5a6d7f47 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
5d5b6ef3 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
6036ac22 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
642e40a8 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
68053653 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
6c54400e : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
6ef2b93b : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
6f347bd0 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
718089eb : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
73e4049f : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
7adac137 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
7f37f1df : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
853d0e13 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
8559596f : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
8c921bde : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
91f2043c : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
92d9a51f : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
9723a725 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
9b75e733 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
9babf616 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
9be86289 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
9f32d23d : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
a7e312ef : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
b1d15cba : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
b312ea85 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
b81706ac : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ba54e1ac : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
bbfd8cd7 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
bde10ef8 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
bff5f74a : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
c2645edb : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
c6ad222b : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
c92b84b8 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
d3113f35 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
d341b47d : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
d449072f : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
d47ade5b : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
d52ba107 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
d5736544 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
d6f33565 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
db2f388a : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
df2287d4 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
e26ccd71 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
e47c100a : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
e6213819 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
e89ed83d : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ea40e89b : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
eb51ec19 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
edab4bf1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
eec72102 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ef17800c : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ef341a9d : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
f0528834 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
f15871f2 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
f3e5c95b : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
f479a59b : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
f5653015 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
fa1ebfe6 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
fa465a90 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
fa6dda68 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
fd67d48c : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
fd7cdce7 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
fe3f39e5 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
All jails are identical with 28 entries in /etc/passwd
(env) > ssh admin@fe3f39e5 cat /etc/passwd | wc -l
28