Recovering from a search head cluster where some members have broken kvstore

The following is some pseudocode / ansible for fixing your search head cluster when you have bad kvstores or orphaned members.

- name: Restart splunk
  command: "sudo -H -u splunk /opt/splunk/bin/splunk restart"

- name: back up folders
  command: "cp -Rp {{ splunk_home }} {{ splunk_backup }}"

- name: Stop splunk
  command: {{ splunk_home }}/bin/splunk stop

- name: splunk clean raft
  command: "{{ splunk_home }}/bin/splunk clean raft --answer-yes -auth admin:changeme"

- name: clean kvstore clustering
  command: "{{ splunk_home }}/bin/splunk clean kvstore --cluster --answer-yes -auth admin:changeme"

- name: start search head with search head with good kvstore
  command: {{ splunk_home }}/bin/splunk start
  when: "good_search_head"

- name: bootstrap shc with good_search_head
  command: "{{ splunk_home }}/bin/splunk bootstrap shcluster-captain -servers_list "https://<member_ip>:8089" -auth admin:changeme"
  when: "good_search_head"

- name: verify status
  command: "{{ splunk_home }}/bin/splunk show shcluster-status -auth admin:changeme"
  when: "good_search_head"

- name: Stop splunk
  command: {{ splunk_home }}/bin/splunk stop

- name: splunk clean raft on good_search_head
  command: "{{ splunk_home }}/bin/splunk clean raft --answer-yes -auth admin:changeme"
  when: "good_search_head"

- name: start all search heads in cluster
  command: {{ splunk_home }}/bin/splunk start

- name: bootstrap shc with good_search_head

  command: "{{ splunk_home }}/bin/splunk bootstrap shcluster-captain -servers_list "https://<member_ip>:8089,https://<member_ip>:8089,https://<member_ip>:8089,https://<member_ip>:8089" -auth admin:changeme"

References:
https://answers.splunk.com/answers/482209/why-is-the-kv-store-status-is-showing-as-starting.html
*note splunk clean raft is a better way to clear it

Using ansible to make API requests to Splunk

I used the following format to make a REST API call against Splunk using Ansible.

- name: REST API call to Encrypt passwords
  uri:
    url: https://localhost:8089/servicesNS/nobody/{{ item.app_directory }}/storage/passwords
    user: admin
    password: changeme
    method: POST
    body:
      app_username:
      app_password: "{{ item.app_password }}"
      app_realm: (optional)
    body_format: form-urlencoded

Note: form-urlencoded is only supported on later versions of ansible.

References:
http://www.georgestarcher.com/splunk-stored-encrypted-credentials/

Experience taking GREM certification

My experience taking the GREM certification

Tips and tricks

The GREM certification is one of the harder certifications that I have attempted.

There wasn't too much assembly in the exam. There are certain patterns in assembly that you should recall and identify quickly. Since this is a timed exam you could easily run out of time attempting to slowly work out the assembly code.

Study your javascript

Create an index
@hacks4pancakes has written an excellent blog post on this topic.
https://tisiphone.net/2015/08/18/giac-testing/

Study, learn, and attempt to understand the material.

Credit Freezes

When freezing your credit file remember to place it at the following places.

Major three credit bureaus

- Experian
- Equifax
- Transunion

In addition, the following places
- Innovis
- Sage Stream

References:

https://krebsonsecurity.com/2015/06/how-i-learned-to-stop-worrying-and-embrace-the-security-freeze/

Rewriting indexes.conf for volume based definitions

At some point when you move from a one tiered storage to a two tiered storage in Splunk where hot/warm buckets are on fast storage (SSD), and cold on slow storage (HDD), you may need to rewrite your indexes.conf

Rewriting your indexes.conf is a fairly easy exercise, but can go disastrously wrong. Just remember that thawed storage can not reference volumes. So double check that the same data locations are still referenced. I also recommend that you rework the Splunk internal indexes ie _internal, _telemetry to reference volumes.

When rewriting your indexes.conf. I recommend placing the index cluster into maintenance mode to prevent buckets from moving in the event there is a bug in your conf file. Then monitor your index cluster for unusual activity once the modified indexes.conf file is deployed. Anomalous activity can include the number of tasks / fixup tasks increasing drastically. In our situation, we saw the number of tasks go beyond 10,000 tasks.

If you see the following, likely something has gone wrong.

- Data may not be searchable temporarily
- Search factor may not be met
- Replication factor may not be met
- High number of fixups to meet search and replication factor

In my situation, somehow at the OS level, our symlink mapped to a different location with our new indexes.conf. Resulting in a high number of fixups and Splunk not seeing the data.

Equifax and credit freezes

Equifax is one of the biggest pains when it comes to credit freezes.

When you have a credit freeze in place with equifax, pulling a credit report via annualcreditreport.com becomes a chore. The site will require offline verification in order to view and audit your report.

Furthermore, when lifting credit freezes, I have encountered the same hassle where it requires an offline submission to lift the freezes. Luckily, I found a hotline 888-298-0045 where you can call to get your freeze lifted.

My experience taking FOR610 - Malware Analysis Training

I have recently taken SANS FOR610 with Lenny Zeltser

SANS courses are typically very expensive, I would not recommend paying the full price if it is out of pocket. There are other resources online, and books that deliver more bang for the buck. But many employers will pay for the course because there are few other places that will deliver as much practical knowledge within the time span of 1 week, and many of the concepts you learn can be applied immediately in your job. 

The FOR610 course has been recently been rewritten and the authors have done a good job with updating the material to keep up to date. Fundamental reverse engineering and malware analysis techniques have stayed the same. However, I was surprised at the amount of changes made in the number of available tools. A major change was the use of x64dbg as a debugger over the classic Ollydbg. 

Prereq's for the course

- Laptop

- Ethernet Jack

- VMware Pro

- 8gb Ram

- 128GB ssd

You do not need to have prior programming experience to take this course. However, I recommend people taking the course should have at least taken 1 semester of computer science / programming course. Knowing basic programming concepts such as switches, if statements, arrays, API calls, will help. To get the most out of the course you should also be able to read basic assembly. Know what registers are EDI, EAX, ESP, AL, EBP.... would greatly help on certain days.

I went into this course not coming from a developer or programming background nor doing malware or reverse engineering as part of my job. 

Day 1

Overview of knowledge domain of malware analysis, and goes into dynamic analysis. There's some basic static analysis such as using strings or pestudio.

Day 2

Intensive day into Assembly. What does control flow, conditional statements look like in assembly. 

Day 3

Analyzing pdf and word attachments, deobfuscating javascript

Day 4

Unpacking malware and using debugger to dump from memory

Day 5

Anti-analysis techniques implemented by malware. Different tricks malware authors will utilize to detect virtual environments. 

Day 6

Capture the flag. This was a good exercise in putting the concepts we learned into practice. Not a necessary day since new material isn't covered. However, highly recommended. 

Things missing from the course

This is not a hard core reverse engineering course. Kernel debugging isn't covered nor arm reverse engineering and analysis. Importing symbols also isn't covered. 

Overall
I recommend this course to anyone that does Incident Response, reverse engineering, or malware analysis. The course has a good mix of dynamic and static techniques that will help improve your skills. I do recommend that people should have a basic programming background and/or have done malware analysis before. The learning doesn't stop once the course ends. The course authors have packed more material than there is time to present, there are exercises in the appendix to practice on your own time. I would have liked SANS modify the course to have 1 or 2 days of extended hours There is simply more material than there is time to cover on some days.

Resources for additional learning

Malware Unicorn's RE 101 course on her website provides very good material to learn and great graphics. 

Practical Malware Analysis

Mandiant's Flare challenges