Recovering from a search head cluster where some members have broken kvstore

The following is some pseudocode / ansible for fixing your search head cluster when you have bad kvstores or orphaned members.

- name: Restart splunk
  command: "sudo -H -u splunk /opt/splunk/bin/splunk restart"

- name: back up folders
  command: "cp -Rp {{ splunk_home }} {{ splunk_backup }}"

- name: Stop splunk
  command: {{ splunk_home }}/bin/splunk stop

- name: splunk clean raft
  command: "{{ splunk_home }}/bin/splunk clean raft --answer-yes -auth admin:changeme"

- name: clean kvstore clustering
  command: "{{ splunk_home }}/bin/splunk clean kvstore --cluster --answer-yes -auth admin:changeme"

- name: start search head with search head with good kvstore
  command: {{ splunk_home }}/bin/splunk start
  when: "good_search_head"

- name: bootstrap shc with good_search_head
  command: "{{ splunk_home }}/bin/splunk bootstrap shcluster-captain -servers_list "https://<member_ip>:8089" -auth admin:changeme"
  when: "good_search_head"

- name: verify status
  command: "{{ splunk_home }}/bin/splunk show shcluster-status -auth admin:changeme"
  when: "good_search_head"

- name: Stop splunk
  command: {{ splunk_home }}/bin/splunk stop

- name: splunk clean raft on good_search_head
  command: "{{ splunk_home }}/bin/splunk clean raft --answer-yes -auth admin:changeme"
  when: "good_search_head"

- name: start all search heads in cluster
  command: {{ splunk_home }}/bin/splunk start

- name: bootstrap shc with good_search_head

  command: "{{ splunk_home }}/bin/splunk bootstrap shcluster-captain -servers_list "https://<member_ip>:8089,https://<member_ip>:8089,https://<member_ip>:8089,https://<member_ip>:8089" -auth admin:changeme"

References:
https://answers.splunk.com/answers/482209/why-is-the-kv-store-status-is-showing-as-starting.html
*note splunk clean raft is a better way to clear it

Using ansible to make API requests to Splunk

I used the following format to make a REST API call against Splunk using Ansible.

- name: REST API call to Encrypt passwords
  uri:
    url: https://localhost:8089/servicesNS/nobody/{{ item.app_directory }}/storage/passwords
    user: admin
    password: changeme
    method: POST
    body:
      app_username:
      app_password: "{{ item.app_password }}"
      app_realm: (optional)
    body_format: form-urlencoded

Note: form-urlencoded is only supported on later versions of ansible.

References:
http://www.georgestarcher.com/splunk-stored-encrypted-credentials/

Experience taking GREM certification

My experience taking the GREM certification

Tips and tricks

The GREM certification is one of the harder certifications that I have attempted.

There wasn't too much assembly in the exam. There are certain patterns in assembly that you should recall and identify quickly. Since this is a timed exam you could easily run out of time attempting to slowly work out the assembly code.

Study your javascript

Create an index
@hacks4pancakes has written an excellent blog post on this topic.
https://tisiphone.net/2015/08/18/giac-testing/

Study, learn, and attempt to understand the material.