SIEM, Splunk, ELK and ulimits

Logging software including Splunk and ELK typically require resources in excess of typical Unix system defaults set in ulimits.conf (user limits). These Unix system defaults are designed to prevent a program from taking over the resources of a server. However, in the case of SIEM software you want it to maximize the resources on the system for performance.

A proper Splunk install will require customizing and changing ulimits so the Splunk daemon can have open file handles in excess of the system defaults.

In a distributed Splunk environment, the easiest way to check your ulimits is by utilizing the Health Check in the Monitoring Console.




If the Monitoring Console comes back with a finding on improper ulimits. Please see the following reference guide by Elastic on setting proper ulimits.

https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#ulimit

ulimits can be adjusted in the following places in 

- /etc/security/limits.conf 

- /etc/init.d/functions

In Ubuntu you also need to uncomment (please see elastic documentation for further details)

# session    required   pam_limits.so

You will need to restart the Splunk daemon under the user that you modified the ulimits. If the modified ulimits were set in the limits.conf for the Splunk user and the init.d scripts were modified for the Splunk user, then you will need to switch into the Splunk user with sudo and restart Splunk.
(If you restart as root but the higher ulimits were set for another user then the ulimits for Splunk user will not get picked up)

References:
https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#ulimit
http://www.georgestarcher.com/splunk-ulimits-and-you/
https://www.splunk.com/blog/2011/11/21/whats-your-ulimit.html

Preparing for the AWS SysOps Exam 2019

Introduction

I had held the AWS Associate Architect Certification and wanted to continue to test my learning by pursuing the AWS SysOps Certificate. I found the SysOps exam to be the harder than the Associate Architect exam.

Overview of Exam

The exam is 130 minutes, 65 questions.

Exam Preparation

It took me roughly 6 months to prepare for the certification. A majority of this time spent was reading through the AWS Certified SysOps Administrator Official Study Guide. The book is slightly dated since it came out in 2017, but roughly ~80% of the material is still relevant. The most helpful sections in the book are the practice questions that come at the end of each chapter.

https://www.amazon.com/Certified-SysOps-Administrator-Official-Study/dp/1119377420

The exam guide on the website is helpful in understanding the overall format and weighting of each section.
https://d1.awsstatic.com/training-and-certification/docs-sysops-associate/AWS_Certified_SysOps_Associate-Exam_Guide_EN_1.4.pdf

The sample questions provided by Amazon are helpful in getting a feel for the test. Furthermore, you can pay $20 to get a mock exam that will get a good feel of the exam interface along with more sample questions. Prior to taking the mock exam, I recommend getting pencil and paper because unfortunately you are only given a final score without feedback.

I also recommend going through the free training provided. The video is only 90 minutes but covers a lot of ground and goes through some additional sample questions.
https://www.aws.training/learningobject/video?id=27486

Guidelines

Day of the exam
Make sure you get plenty of rest and remember to bring the proper ID.
Protip: You can use the keys 1, 2, 3, 4, 5 to select the appropriate answer then Tab and Enter to submit.

Topics to study for the SysOps exam

Domain 1: Monitoring and Reporting 22%
Cost explorer

AWS Budgets allows to set alerts once you approach or exceed budget

Personal Health Dashboard
Cloudwatch metrics (What are the 6 metrics?)
Cloudwatch metrics, total error rate used to troubleshoot 4xx and 5xx.

How to monitor for memory usage within an EC2 (Hint: Requires a custom metric)

Domain 2: High Availability 8%
Difference between Read Replica vs Multi-AZ in RDS

Domain 3: Deployment and Provisioning 14%
Stack Policy's - Assign a stack policy to prevent updates to stack resources.
Stack changes
Enable termination protection to prevent users from deleting the stack from the AWS CloudFormation console or AWS Command Line Interface (AWS CLI).
https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-accidental-updates/
- Set the DeletionPolicy attribute to prevent the deletion of an individual resource at the stack level.

Domain 4: Storage and Data Management 12%
AWS EBS Elastic Volumes can expand a volume

Bucket policies
Elastic File Shares, EFS can be Multi-AZ

Elastic cache 

AWS Auora
S3
- principal can be a user or role
- Origin Access Identity

Domain 5: Security and Compliance 18%
KMS events recorded by cloudtrail
AWS Config - typically used for governance
AWS WAF
AWS Identity and Access Management (IAM) policies to restrict the ability of users to delete or update a stack and its resources.

VPC Endpoints Communicating with AWS services outside VPC without going over internet. 

Domain 6: Networking 14%
Direct connect

Internet Gateways
NAT Gateway
Egress Only Internet Gateway
Customer Gateway 

Vpc route
Dual Stack mode
Customer Gateway - If behind NAT must use publically routable IP address

Domain 7: Automation and Optimization
AWS Cache Hit Ratio - Only forward required cookies, headers, queryStep up auto scale

Autoscaling
- Troubleshooting
- Scaling
- Does autoscale work across regions or only within a region? 

Experiences moving away from SMS based 2nd factor authentication

NIST Special Publication 800-63B was released 2017.

Why 2fa using SMS is good enough for most people. Default settings in security are extremely important since a majority of users do not change these settings. For many sites, the default is to allow users access using the standard username and password. Users need to opt in order to get their 2fa enabled.

The NIST publication recommends moving away from SMS based 2 factor authentication.
- SMS text messages can be intercepted
- Users can be susceptible to SMS based phishing

The recommendation is for the implementation of OTP based hardware/software tokens that are based on asymmetric cryptography. While it is still possible for an attacker to get the codes they are now reduced to either an attack
- compromise the second factor
- phish the second factor from the user
- physical access

Based on my personal experiences, before implementing software tokens and moving away from SMS based 2fa make sure you have a backup plan for when you lost those tokens.

A couple months ago I had lost my phone and as a result, I lost all software tokens from my device. As a result, I was nearly locked out from some of my accounts.

Getting LTE in China, phone incompatible with TD-LTE

While traveling in China, I noticed that I was only getting edge. I ran the following steps in order to get LTE.

1. Choose Cellular Networks

2. Choose Network operators

3. Search Networks and choose China Unicom


My mobile phone didn't support TD-LTE, as I result I needed to force my phone to choose China Unicom instead of China Mobile. China Unicom supports FDD-LTE while China Mobile only supports TD-LTE. My phone choose China Mobile automatically.


*Credit to reddit where I found the solution

Recovering from a search head cluster where some members have broken kvstore

The following is some pseudocode / ansible for fixing your search head cluster when you have bad kvstores or orphaned members.

- name: Restart splunk
  command: "sudo -H -u splunk /opt/splunk/bin/splunk restart"

- name: back up folders
  command: "cp -Rp {{ splunk_home }} {{ splunk_backup }}"

- name: Stop splunk
  command: {{ splunk_home }}/bin/splunk stop

- name: splunk clean raft
  command: "{{ splunk_home }}/bin/splunk clean raft --answer-yes -auth admin:changeme"

- name: clean kvstore clustering
  command: "{{ splunk_home }}/bin/splunk clean kvstore --cluster --answer-yes -auth admin:changeme"

- name: start search head with search head with good kvstore
  command: {{ splunk_home }}/bin/splunk start
  when: "good_search_head"

- name: bootstrap shc with good_search_head
  command: "{{ splunk_home }}/bin/splunk bootstrap shcluster-captain -servers_list "https://<member_ip>:8089" -auth admin:changeme"
  when: "good_search_head"

- name: verify status
  command: "{{ splunk_home }}/bin/splunk show shcluster-status -auth admin:changeme"
  when: "good_search_head"

- name: Stop splunk
  command: {{ splunk_home }}/bin/splunk stop

- name: splunk clean raft on good_search_head
  command: "{{ splunk_home }}/bin/splunk clean raft --answer-yes -auth admin:changeme"
  when: "good_search_head"

- name: start all search heads in cluster
  command: {{ splunk_home }}/bin/splunk start

- name: bootstrap shc with good_search_head

  command: "{{ splunk_home }}/bin/splunk bootstrap shcluster-captain -servers_list "https://<member_ip>:8089,https://<member_ip>:8089,https://<member_ip>:8089,https://<member_ip>:8089" -auth admin:changeme"

References:
https://answers.splunk.com/answers/482209/why-is-the-kv-store-status-is-showing-as-starting.html
*note splunk clean raft is a better way to clear it

Using ansible to make API requests to Splunk

I used the following format to make a REST API call against Splunk using Ansible.

- name: REST API call to Encrypt passwords
  uri:
    url: https://localhost:8089/servicesNS/nobody/{{ item.app_directory }}/storage/passwords
    user: admin
    password: changeme
    method: POST
    body:
      app_username:
      app_password: "{{ item.app_password }}"
      app_realm: (optional)
    body_format: form-urlencoded

Note: form-urlencoded is only supported on later versions of ansible.

References:
http://www.georgestarcher.com/splunk-stored-encrypted-credentials/

Experience taking GREM certification

My experience taking the GREM certification

Tips and tricks

The GREM certification is one of the harder certifications that I have attempted.

There wasn't too much assembly in the exam. There are certain patterns in assembly that you should recall and identify quickly. Since this is a timed exam you could easily run out of time attempting to slowly work out the assembly code.

Study your javascript

Create an index
@hacks4pancakes has written an excellent blog post on this topic.
https://tisiphone.net/2015/08/18/giac-testing/

Study, learn, and attempt to understand the material.