Commit a749d9e9 authored by hadret's avatar hadret

feat: initial commit.

parents
Pipeline #670 passed with stage
in 5 minutes and 48 seconds
---
stages:
- test
- cleanup
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache:
paths:
- .cache/pip
before_script:
- docker -v
- python3 -V
- python3 -m venv ..
- source ../bin/activate
molecule:
stage: test
tags:
- shell
script:
- pip install docker molecule
- molecule test
---
language: python
services: docker
env:
matrix:
- MOLECULE_DISTRO: ubuntu1804
- MOLECULE_DISTRO: ubuntu1604
- MOLECULE_DISTRO: debian10
install:
- pip install molecule docker
script:
- molecule test
notifications:
webhooks: https://galaxy.ansible.com/api/v1/notifications/
MIT License
Copyright (c) 2019 Filip Chabik
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
# Ansible Role: Rsyslog
[![Build Status](https://travis-ci.com/hadret/ansible-role-rsyslog.svg?branch=master)](https://travis-ci.com/hadret/ansible-role-rsyslog)
Installs and configures [rsyslog](https://rsyslog.com) on Debian/Ubuntu servers.
This role installs and configures the latest version of rsyslog from official APT repository (on Debian) or [official PPA](https://launchpad.net/~adiscon/+archive/ubuntu/v8-stable) (on Ubuntu). By default it will take over contents of the `/etc/rsyslog.conf` and the `/etc/rsyslog.d/50-default.conf`.
## Requirements
None.
## Role variables
Here are available variables with their default values (as in
[defaults/main.yml](defaults/main.yml)):
```yaml
rsyslog_rules: []
```
An array of rules for rsyslog. Each entry will create a separate config file named as `$priority-$rule_name.conf`. Be sure to check `defaults/main.yml` for commented out example with all of the available options.
```yaml
rsyslog_rules:
- rule_name: "remote-udp"
priority: 99
ruleset: |
module(load="omfwd")
action(type="omfwd" target="central.server.local" port="514" protocol="udp")
state: "present"
```
Here's an example of a fully-populated `rsyslog_rules` entry. Please note the `|` for the block declaration for the `ruleset`. From there it becomes _bare_ rsyslog config syntax.
Alternative route from defining the `rsyslog_rules` in a rule-by-rule manner would be to use the `rsyslog_extra_conf_options`. It then extends the main `/etc/rsyslog.conf` configuration file with extra options instead of creating new files in the `/etc/rsyslog.d`.
```yaml
rsyslog_extra_conf_options: |
module(load="imudp")
input(type="imudp" port="514")
```
Here too `|` is used for the block declaration and the code itself is a _bare_ rsyslog config syntax. It can also be combined with `rsyslog_remove_default_rules: true` that would ensure `/etc/rsyslog.d/` is empty.
Additionally there are currently three preconfigured rsyslog rules. All of them have special, dedicated templates (`templates/*.conf.j2`). Only one of them is enabled by default and is called, well, `default`. It takes over definition of `/etc/rsyslog.d/50-default.conf` file. It can be easily disabled by specifying `state: "absent"`.
```yaml
rsyslog_rule_default:
rule_name: "default"
priority: 50
template: "default.conf.j2"
```
Second one is `docker` that handles logs for the Docker containers running on the given host. It's being defined in the `/etc/rsyslog.d/20-docker.conf` file.
```yaml
rsyslog_rule_docker:
rule_name: "docker"
priority: 20
template: "docker.conf.j2"
rsyslog_rule_docker_tag_all: true
```
It will create `/var/log/docker` and put log files inside it with `$CONTAINER_NAME.log` naming scheme. It expects that the `$syslogtag` will have `docker/` in the name (check the example below), otherwise it will push all the logs into `/var/log/docker/no_tag.log`. Additionally there's a `rsyslog_rule_docker_tag_all` that can be turned on when there's more than one container running on the given host and allows for single file with logs aggregated from all of them in `/var/log/docker/all.log` (*Note: this will double the space needed for the container logs*). You may check my [hadret.containers](https://github.com/hadret/ansible-role-containers) role for an example of the container definition with syslog support enabled.
```yaml
containers:
- name: cadvisor
image: "google/cadvisor:latest"
state: started
log_driver: journald
log_options:
tag: docker/cadvisor
```
`journald` is these days picked up automatically by rsyslog.
Last but not least is the `remote` handling. I wanted to create a turn-key solution for handling both client and server parts. Remote logging is currently very raw and basic, but it does work out of the box with a minimal configuration.
```yaml
rsyslog_rule_remote:
rule_name: "remote"
role: server
priority: 99
template: "remote.conf.j2"
ruleset_name: "remote"
```
At least one remote protocol (`relp`/`tcp`/`udp`) has to be specified (*Note: there's no default and specifying `rsyslog_rule_remote` alone will fail*). The way server-side is handled requires defined `ruleset_name` as it is `ruleset` that is executing the actual action of writing down the logs (via `omfile`) and applying predefined templates. These are configured to be as similar to the "ordinary" rules as possible with following outputs predefined: `auth.log`, `syslog.log`, `rsyslog.log`, `kern.log` and `mail.log`.
```yaml
rsyslog_rule_remote_relp:
port: 514
```
Currently only `relp` supports TLS setup.
```yaml
rsyslog_rule_remote_relp:
address: 0.0.0.0
port: 514
tls: true
tls_cacert: "/tls-certs/ca.pem"
tls_mycert: "/tls-certs/cert.pem"
tls_myprivkey: "/tls-certs/key.pem"
tls_authmode: "fingerprint"
```
Both `tcp` and `udp` currently allow only for specifying `address` (optional in server mode), `target` (required in client mode) and `port` (required in both modes).
```yaml
rsyslog_rule_remote_tcp:
address: 0.0.0.0
port: 514
rsyslog_rule_remote_udp:
address: 0.0.0.0
port: 514
```
Please note that you **can** define all three of them, with different addresses and ports (but each of them only once). All of the configuration will by default land in `/etc/rsyslog.d/99-remote.conf` (both server and client). It is currently impossible to have a single machine acting as both server and client solely with `rsyslog_rule_remote_relp` usage, but it is doable to specify additional rule with either `rsyslog_extra_conf_options` or `rsyslog_rules`.
```yaml
rsyslog_rule_remote:
rule_name: "server"
role: server
priority: 99
template: "remote.conf.j2"
ruleset_name: "server"
rsyslog_rule_remote_udp:
port: 514
rsyslog_rules:
- rule_name: "client"
priority: 99
ruleset: |
module(load="omfwd")
action(type="omfwd" target="central.server.local" port="514" protocol="tcp")
```
*Note: all three of these additionally preconfigured rsyslog rules are dictionaries, not arrays. Only the `rsyslog_rules` allow for multiple rule definitions.*
## Extending and replacing templates
I realize that not everything is covered with variables and there are tons of different possible configuration options out there. That's the reason why I'm using templates for all of the rules which allows for easy extending, block replacing (via [Jinja2 template inheritance](http://jinja.pocoo.org/docs/2.9/templates/#template-inheritance)) or full template exchanging to match the needs I haven't thought of.
```yaml
rsyslog_conf_template: "rsyslog.conf.j2"
rsyslog_rules_template: "rules.conf.j2"
```
It can also be changed on per-rule basis.
```yaml
rsyslog_rule_default:
rule_name: "default"
priority: 50
template: "{{ playbook_dir }}/templates/custom-default.conf.j2"
rsyslog_rule_docker:
rule_name: "docker"
priority: 20
template: "{{ playbook_dir }}/templates/custom-docker.conf.j2"
rsyslog_rules:
- rule_name: "remote-udp"
priority: 90
template: "{{ playbook_dir }}/templates/custom-udp.conf.j2"
- rule_name: "remote-tcp"
priority: 91
template: "{{ playbook_dir }}/templates/custom-tcp.conf.j2"
```
### Example: extending modules block in main config file
`rsyslog_conf_template` has to be set to point to the new file in your playbook directory.
```yaml
rsyslog_conf_template: "{{ playbook_dir }}/templates/custom-rsyslog.conf.j2"
```
Custom template file has to be placed relative to your `playbook.yml`.
```
{% extends 'roles/external/hadret.rsyslog/templates/rsyslog.conf.j2' %}
{% block modules %}
$ModLoad imuxsock
$ModLoad imklog
$ModLoad immark
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
{% endblock %}
```
The above example replaces/extends `modules` block in the main rsyslog config file.
## Dependencies
None.
## Example playbook
```
hosts: all
roles:
- hadret.rsyslog
```
## License
MIT.
## Authors
This role was somewhat assembled in 2019 by [Filip Chabik](https://chabik.com).
---
# Use the official rsyslog PPA for Ubuntu (v8-devel/v8-stable)
rsyslog_ppa_use: true
rsyslog_ppa_version: v8-stable
# Name of the package to install
# rsyslog_package_name: "rsyslog"
rsyslog_service_state: started
rsyslog_service_enabled: true
rsyslog_conf_template: "rsyslog.conf.j2"
rsyslog_rules_template: "rules.conf.j2"
rsyslog_action_file_default_template: "RSYSLOG_TraditionalFileFormat"
rsyslog_repeated_msg_reduction: 'off'
# Example extra config for the main rsyslog config file
# rsyslog_extra_conf_options: |
# module(load="imudp" ruleset="udp")
# input(type="imudp" port="1514")
# Preconfigured default rsyslog rules
rsyslog_rule_default:
rule_name: "default"
priority: 50
template: "default.conf.j2"
# Preconfigured docker rsyslog rules
# rsyslog_rule_docker: # /etc/rsyslog.d/20-docker.conf
# rule_name: "docker"
# priority: 20
# template: "docker.conf.j2"
# rsyslog_rule_docker_path: /var/log/docker
# rsyslog_rule_docker_template: "DockerLogFileName"
# rsyslog_rule_docker_tag_all: true
# Preconfigured remote logging rsyslog rules
# rsyslog_rule_remote: # /etc/rsyslog.d/99-remote.conf
# rule_name: "remote"
# role: "server" # server/client
# priority: 99
# template: "remote.conf.j2"
# ruleset_name: "remote"
# rsyslog_rule_remote_path: /var/log/remote
# rsyslog_rule_remote_relp:
# address: 0.0.0.0
# target: central.server.local
# port: 514
# tls: true
# tls_cacert: "/tls-certs/ca.pem"
# tls_mycert: "/tls-certs/cert.pem"
# tls_myprivkey: "/tls-certs/key.pem"
# tls_authmode: "fingerprint" # fingerprint/name
# rsyslog_rule_remote_relp_pkg: rsyslog-relp
# rsyslog_rule_remote_tcp:
# address: 0.0.0.0
# target: "central.server.local"
# port: 514
# rsyslog_rule_remote_udp:
# address: 0.0.0.0
# target: "central.server.local"
# port: 514
rsyslog_remove_default_rules: false
rsyslog_rules: []
# - rule_name: "remote-relp" # name of the rule
# priority: 99 # order of the rules may be important
# ruleset: |
# module(load="omudp")
# action(type="omudp" port="1514")
# # Properties that are only added if defined:
# filename: "99-remote-relp.conf" # default: "$priority-$rule_name.conf"
# template: "" # override the `rsyslog_template` per ruleset
# state: "absent" # remove the ruleset configuration
---
- name: restart rsyslog
service: name=rsyslog state=restarted
---
dependencies: []
galaxy_info:
role_name: rsyslog
author: Filip Chabik (hadret)
description: Rsyslog installation and configuration for Ubuntu/Debian.
license: MIT
min_ansible_version: 2.8
platforms:
- name: Debian
versions:
- buster
- name: Ubuntu
versions:
- xenial
- bionic
galaxy_tags:
- rsyslog
- syslog
- system
---
dependency:
name: galaxy
driver:
name: docker
lint:
name: yamllint
options:
config-file: molecule/default/yaml-lint.yml
platforms:
- name: instance
image: geerlingguy/docker-${MOLECULE_DISTRO:-ubuntu1804}-ansible
command: ${MOLECULE_DOCKER_COMMAND:-""}
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:ro
privileged: true
pre_build_image: true
provisioner:
name: ansible
lint:
name: ansible-lint
playbooks:
converge: ${MOLECULE_PLAYBOOK:-playbook.yml}
scenario:
name: default
verifier:
name: testinfra
lint:
name: flake8
---
- name: test all the things
hosts: all
vars:
rsyslog_ppa_use: true
rsyslog_rule_default:
rule_name: "default"
priority: 50
template: "default.conf.j2"
rsyslog_rule_docker:
rule_name: "docker"
priority: 20
template: "docker.conf.j2"
rsyslog_rule_docker_tag_all: true
rsyslog_rule_remote:
rule_name: "remote"
role: server
priority: 99
template: "remote.conf.j2"
ruleset_name: "remote"
rsyslog_rule_remote_relp:
port: 514
rsyslog_rules:
- rule_name: "remote-udp"
priority: 99
ruleset: |
module(load="imudp")
input(type="imudp" port="514")
pre_tasks:
- name: update apt cache
apt: update_cache=yes cache_valid_time=600
when: ansible_os_family == 'Debian'
changed_when: false
- name: define rsyslog_user
set_fact:
rsyslog_user: "root"
when: ansible_distribution == 'Debian'
roles:
- role: ansible-role-rsyslog
---
extends: default
rules:
line-length:
max: 120
level: warning
---
- name: ensure rsyslog_rule_docker_path exists
file:
path: "{{ rsyslog_rule_docker_path|default('/var/log/docker') }}"
owner: "{{ rsyslog_user }}"
group: "{{ rsyslog_group }}"
mode: 0755
state: directory
when: rsyslog_rule_docker is defined
- name: ensure rsyslog_rule_remote_path exists
file:
path: "{{ rsyslog_rule_remote_path|default('/var/log/remote') }}"
owner: "{{ rsyslog_user }}"
group: "{{ rsyslog_group }}"
mode: 0755
state: directory
when:
- rsyslog_rule_remote is defined
- rsyslog_rule_remote.role == "server"
---
- name: include OS-specific variables
include_vars: "{{ ansible_os_family }}.yml"
- name: define rsyslog_user
set_fact:
rsyslog_user: "{{ __rsyslog_user }}"
when: rsyslog_user is not defined
- name: define rsyslog_group
set_fact:
rsyslog_group: "{{ __rsyslog_group }}"
when: rsyslog_group is not defined
- name: define rsyslog_file_create_mode
set_fact:
rsyslog_file_create_mode: "{{ __rsyslog_file_create_mode }}"
when: rsyslog_file_create_mode is not defined
- name: define rsyslog_dir_create_mode
set_fact:
rsyslog_dir_create_mode: "{{ __rsyslog_dir_create_mode }}"
when: rsyslog_dir_create_mode is not defined
- name: define rsyslog_umask
set_fact:
rsyslog_umask: "{{ __rsyslog_umask }}"
when: rsyslog_umask is not defined
- include_tasks: setup-Debian.yml
when: ansible_distribution == 'Debian'
- include_tasks: setup-Ubuntu.yml
when: ansible_distribution == 'Ubuntu'
# rules configuration
- import_tasks: rules.yml
# extras configuration
- include_tasks: extras.yml
when: (rsyslog_rule_remote is defined) or
(rsyslog_rule_docker is defined)
# rsyslog setup
- name: copy main rsyslog config file
template:
src: "{{ rsyslog_conf_template }}"
dest: "{{ rsyslog_conf_file_path }}"
owner: root
group: "{{ root_group }}"
mode: 0644
validate: /usr/sbin/rsyslogd -N1 -f %s
notify:
- restart rsyslog
- name: ensure rsyslog is up and enabled on boot
service:
name: rsyslog
state: "{{ rsyslog_service_state }}"
enabled: "{{ rsyslog_service_enabled }}"
---
- name: remove default rsyslog rules
file:
path: "{{ rsyslog_conf_path }}"
state: absent
when: rsyslog_remove_default_rules | bool
notify: restart rsyslog
- name: ensure rsyslog_conf_path exists
file:
path: "{{ rsyslog_conf_path }}"
owner: root
group: "{{ root_group }}"
mode: 0755
state: directory
- name: add managed rsyslog rules
template:
src: "{{ item.template|default(rsyslog_rules_template) }}"
dest: "{{ rsyslog_conf_path }}/{{ item.filename|default(item.priority ~ '-' ~ item.rule_name ~ '.conf') }}"
force: true
owner: root
group: "{{ root_group }}"
mode: 0644
when: item.state|default('present') != 'absent'
with_items:
- "{{ rsyslog_rule_default|default([]) }}"
- "{{ rsyslog_rule_docker|default([]) }}"
- "{{ rsyslog_rule_remote|default([]) }}"
- "{{ rsyslog_rules|default([]) }}"
notify: restart rsyslog
tags: ['skip_ansible_lint']
- name: remove managed rsyslog rules
file:
path: "{{ rsyslog_conf_path }}/{{ item.filename|default(item.priority ~ '-' ~ item.rule_name ~ '.conf') }}"
state: absent
when: item.state|default('present') == 'absent'
with_items:
- "{{ rsyslog_rule_default|default([]) }}"
- "{{ rsyslog_rule_docker|default([]) }}"
- "{{ rsyslog_rule_remote|default([]) }}"
- "{{ rsyslog_rules|default([]) }}"
notify: restart rsyslog
tags: ['skip_ansible_lint']
---
- name: ensure rsyslog is installed
apt:
name: "{{ rsyslog_package_name|default('rsyslog') }}"
state: present
cache_valid_time: 3600
- name: ensure rsyslog-relp is installed
apt:
name: "{{ rsyslog_rule_remote_relp_pkg|default('rsyslog-relp') }}"
state: present
when:
- rsyslog_rule_remote is defined
- rsyslog_rule_remote_relp is defined
---
- name: add official PPA for rsyslog
apt_repository:
repo: 'ppa:adiscon/{{ rsyslog_ppa_version }}'
state: present
update_cache: true
register: rsyslog_ppa_added
when: rsyslog_ppa_use | bool
- name: upgrade rsyslog to version from PPA
apt:
name: "{{ rsyslog_package_name|default('rsyslog') }}"
state: latest
when: rsyslog_ppa_added.changed
tags: ['skip_ansible_lint']
- name: ensure rsyslog-relp is installed
apt:
name: "{{ rsyslog_rule_remote_relp_pkg|default('rsyslog-relp') }}"
state: present
when:
- rsyslog_rule_remote is defined
- rsyslog_rule_remote_relp is defined
# {{ ansible_managed }}
# Default rules for rsyslog.
#
# For more information see rsyslog.conf(5) and {{ rsyslog_conf_file_path }}
{% block default_rules %}
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none /var/log/syslog
syslog.* /var/log/rsyslog.log #rsyslog error messages
#cron.* /var/log/cron.log
#daemon.* /var/log/daemon.log
kern.* /var/log/kern.log
#lpr.* /var/log/lpr.log
mail.* /var/log/mail.log
#user.* /var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info /var/log/mail.info
#mail.warn /var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice /var/log/news/news.notice
#
# Some "catch-all" log files.
#
#*.=debug;\
# auth,authpriv.none;\
# news.none;mail.none /var/log/debug
#*.=info;*.=notice;*.=warn;\
# auth,authpriv.none;\
# cron,daemon.none;\
# mail,news.none /var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
#
# As this functionality is almost never needed, it is commented out. If you
# need it, be sure to remove the comment characters below.
#daemon.*;mail.*;\
# news.err;\
# *.=debug;*.=info;\
# *.=notice;*.=warn |/dev/xconsole
{% endblock %}
# {{ ansible_managed }}
{% block docker_rules %}
template(name="{{ rsyslog_rule_docker_template|default('DockerLogFileName') }}" type="list") {
constant(value="{{ rsyslog_rule_docker_path|default('/var/log/docker') }}/")
property(name="syslogtag" securepath="replace" \
regex.expression="docker/\\(.*\\)\\[" regex.submatch="1")
constant(value=".log")
}
{% if rsyslog_rule_docker_tag_all is defined -%}
if $programname == 'docker' then {{ rsyslog_rule_docker_path|default('/var/log/docker') }}/all.log
{% endif -%}
if $programname == 'docker' then \
if $syslogtag contains 'docker/' then \
?{{ rsyslog_rule_docker_template|default('DockerLogFileName') }}
else
{{ rsyslog_rule_docker_path|default('/var/log/docker') }}/no_tag.log
& stop
{% endblock %}
# {{ ansible_managed }}
{% if rsyslog_rule_remote.role == "server" %}
{% block server_protocol %}
{% if rsyslog_rule_remote_relp is defined %}
module(load="imrelp" ruleset="{{ rsyslog_rule_remote.ruleset_name }}")
input(type="imrelp" port="{{ rsyslog_rule_remote_relp.port }}"
{%- if rsyslog_rule_remote_relp.address is defined %} address="{{ rsyslog_rule_remote_relp.address }}"{% endif -%}
{%- if rsyslog_rule_remote_relp.tls is defined %} tls="on"
{%- if rsyslog_rule_remote_relp.tls_cacert is defined %} tls.cacert="{{ rsyslog_rule_remote_relp.tls_cacert }}"{% endif -%}
{%- if rsyslog_rule_remote_relp.tls_mycert is defined %} tls.mycert="{{ rsyslog_rule_remote_relp.tls_mycert }}"{% endif -%}
{%- if rsyslog_rule_remote_relp.tls_myprivkey is defined %} tls.myprivkey="{{ rsyslog_rule_remote_relp.tls_myprivkey }}"{% endif -%}
{%- if rsyslog_rule_remote_relp.tls_authmode is defined %} tls.authmode="{{ rsyslog_rule_remote_relp.tls_authmode }}"{% endif -%}
{% endif -%})
{% endif %}
{% if rsyslog_rule_remote_tcp is defined %}
module(load="imtcp" ruleset="{{ rsyslog_rule_remote.ruleset_name }}")
input(type="imtcp" port="{{ rsyslog_rule_remote_tcp.port }}"
{%- if rsyslog_rule_remote_tcp.address is defined %} address="{{ rsyslog_rule_remote_tcp.address }}"{% endif -%})
{% endif %}
{% if rsyslog_rule_remote_udp is defined %}