forked from sadsfae/ansible-elk
Folders and files
Ansible Playbook for setting up the ELK/EFK Stack and Filebeat client on remote hosts
- Automated deployment of a full ELK or EFK stack (Elasticsearch, Logstash/Fluentd, Kibana)
- 5.5+ and 2.4 ELK versions are maintained.
- Uses Nginx as a reverse proxy for Kibana
- Generates SSL certificates for Filebeat or Logstash-forwarder
- Adds either iptables or firewalld rules if firewall is active
- Tunes Elasticsearch heapsize to half your memory, to a max of 32G
- Deploys ELK clients using SSL and Filebeat for Logstash (Default)
- Deploys rsyslog if Fluentd is chosen over Logstash, picks up the same set of OpenStack-related logs in /var/log/*
- All service ports can be modified in install/group_vars/all.yml
- Optionally install curator
- Optionally install Elastic X-Pack Suite
- This is also available on Ansible Galaxy
- RHEL7 or CentOS7+ server/client with no modifications
- ELK/EFK server with at least 8G of memory (you can try with less but 5.x series is quite demanding - try 2.4 series if you have scarce resources).
- Fedora 23 or higher needs to have yum python2 python2-dnf libselinux-python packages.
- You can run this against Fedora clients prior to running Ansible ELK:
- ansible fedora-client-01 -u root -m shell -i hosts -a "dnf install yum python2 libsemanage-python python2-dnf -y"
- Fedora 23 or higher needs to have yum python2 python2-dnf libselinux-python packages.
- You may want to modify vm.swappiness as ELK/EFK is demanding and swapping kills the responsiveness.
- I am leaving this up to your judgement.
echo "vm.swappiness=10" >> /etc/sysctl.conf sysctl -p
- Current ELK version is 5.5.x but you can checkout the 2.4 branch if you want that series
- Sets the nginx htpasswd to admin/admin initially
- nginx ports default to 80/8080 for Kibana and SSL cert retrieval (configurable)
- Uses OpenJDK for Java
- It's fairly quick, takes around 3minutes on test VM
- Filebeat templating is focused around OpenStack service logs
- Fluentd can be substituted for the default Logstash
- Set logging_backend: fluentd in group_vars/all.yml
- Install curator by setting install_curator_tool: true in install/group_vars/all.yml
- Install Elastic X-Pack Suite for Elasticsearch, LogStash or Kibana via:
- install_elasticsearch_xpack: true
- install_kibana_xpack: true
- install_logstash_xpack: true
- Note: Deploying X-Pack will wrap your ES with additional authentication and security, Kibana for example will have it's own credentials now - the default is username: elastic and password: changeme
- Clone repo and setup your hosts file
git clone https://github.com/sadsfae/ansible-elk cd ansible-elk sed -i 's/host-01/elkserver/' hosts sed -i 's/host-02/elkclient/' hosts
- If you're using a non-root user for Ansible, e.g. AWS EC2 likes to use ec2-user then set the follow below, default is root.
ansible_system_user: ec2-user
- Run the playbook
ansible-playbook -i hosts install/elk.yml
- (see playbook messages)
- Navigate to the ELK at http://host-01:80
- Default login is:
- username: admin
- password: admin
- Run the client playbook against the generated elk_server variable
ansible-playbook -i hosts install/elk-client.yml --extra-vars 'elk_server=X.X.X.X'
- Once this completes return to your ELK and you'll see log results come in from ELK/EFK clients via filebeat
- The 2.4 series of ELK/EFK is also available, to use this just clone the 2.4 branch
git clone https://github.com/sadsfae/ansible-elk cd ansible-elk git checkout 2.4
- You can view a deployment video here:
.
├── hosts
├── install
│ ├── elk-client.yml
│ ├── elk.yml
│ ├── group_vars
│ │ └── all.yml
│ └── roles
│ ├── curator
│ │ ├── files
│ │ │ └── curator.repo
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ ├── curator-action.yml.j2
│ │ └── curator-config.yml.j2
│ ├── elasticsearch
│ │ ├── files
│ │ │ ├── elasticsearch.in.sh
│ │ │ └── elasticsearch.repo
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ └── elasticsearch.yml.j2
│ ├── elk_client
│ │ ├── files
│ │ │ └── elk.repo
│ │ └── tasks
│ │ └── main.yml
│ ├── filebeat
│ │ ├── meta
│ │ │ └── main.yml
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ ├── filebeat.yml.j2
│ │ └── rsyslog-openstack.conf.j2
│ ├── firewall
│ │ └── tasks
│ │ └── main.yml
│ ├── fluentd
│ │ ├── files
│ │ │ ├── filebeat-index-template.json
│ │ │ └── fluentd.repo
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ ├── openssl_extras.cnf.j2
│ │ └── td-agent.conf.j2
│ ├── heartbeat
│ │ ├── meta
│ │ │ └── main.yml
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ └── heartbeat.yml.j2
│ ├── instructions
│ │ └── tasks
│ │ └── main.yml
│ ├── kibana
│ │ ├── files
│ │ │ ├── filebeat-dashboards.zip
│ │ │ ├── kibana.repo
│ │ │ └── logstash.repo
│ │ └── tasks
│ │ └── main.yml
│ ├── logstash
│ │ ├── files
│ │ │ ├── filebeat-index-template.json
│ │ │ └── logstash.repo
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ ├── 02-beats-input.conf.j2
│ │ ├── logstash.conf.j2
│ │ └── openssl_extras.cnf.j2
│ ├── metricbeat
│ │ ├── meta
│ │ │ └── main.yml
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ └── metricbeat.yml.j2
│ ├── nginx
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ ├── kibana.conf.j2
│ │ └── nginx.conf.j2
│ ├── packetbeat
│ │ ├── meta
│ │ │ └── main.yml
│ │ ├── tasks
│ │ │ └── main.yml
│ │ └── templates
│ │ └── packetbeat.yml.j2
│ └── xpack
│ └── tasks
│ └── main.yml
└── meta
└── main.yml
51 directories, 49 files