Logging software including Splunk and ELK typically require resources in excess of typical Unix system defaults set in ulimits.conf (user limits). These Unix system defaults are designed to prevent a program from taking over the resources of a server. However, in the case of SIEM software you want it to maximize the resources on the system for performance.
A proper Splunk install will require customizing and changing ulimits so the Splunk daemon can have open file handles in excess of the system defaults.
In a distributed Splunk environment, the easiest way to check your ulimits is by utilizing the Health Check in the Monitoring Console.
If the Monitoring Console comes back with a finding on improper ulimits. Please see the following reference guide by Elastic on setting proper ulimits.
https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#ulimit
ulimits can be adjusted in the following places in
- /etc/security/limits.conf
- /etc/init.d/functions
In Ubuntu you also need to uncomment (please see elastic documentation for further details)
# session required pam_limits.so
You will need to restart the Splunk daemon under the user that you modified the ulimits. If the modified ulimits were set in the limits.conf for the Splunk user and the init.d scripts were modified for the Splunk user, then you will need to switch into the Splunk user with sudo and restart Splunk.
(If you restart as root but the higher ulimits were set for another user then the ulimits for Splunk user will not get picked up)
References:
https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#ulimit
http://www.georgestarcher.com/splunk-ulimits-and-you/
https://www.splunk.com/blog/2011/11/21/whats-your-ulimit.html
No comments:
Post a Comment