Experiences moving away from SMS based 2nd factor authentication

NIST Special Publication 800-63B was released 2017.

Why 2fa using SMS is good enough for most people. Default settings in security are extremely important since a majority of users do not change these settings. For many sites, the default is to allow users access using the standard username and password. Users need to opt in order to get their 2fa enabled.

The NIST publication recommends moving away from SMS based 2 factor authentication.
- SMS text messages can be intercepted
- Users can be susceptible to SMS based phishing

The recommendation is for the implementation of OTP based hardware/software tokens that are based on asymmetric cryptography. While it is still possible for an attacker to get the codes they are now reduced to either an attack
- compromise the second factor
- phish the second factor from the user
- physical access

Based on my personal experiences, before implementing software tokens and moving away from SMS based 2fa make sure you have a backup plan for when you lost those tokens.

A couple months ago I had lost my phone and as a result, I lost all software tokens from my device. As a result, I was nearly locked out from some of my accounts.

Getting LTE in China, phone incompatible with TD-LTE

While traveling in China, I noticed that I was only getting edge. I ran the following steps in order to get LTE.

1. Choose Cellular Networks

2. Choose Network operators

3. Search Networks and choose China Unicom


My mobile phone didn't support TD-LTE, as I result I needed to force my phone to choose China Unicom instead of China Mobile. China Unicom supports FDD-LTE while China Mobile only supports TD-LTE. My phone choose China Mobile automatically.


*Credit to reddit where I found the solution

Recovering from a search head cluster where some members have broken kvstore

The following is some pseudocode / ansible for fixing your search head cluster when you have bad kvstores or orphaned members.

- name: Restart splunk
  command: "sudo -H -u splunk /opt/splunk/bin/splunk restart"

- name: back up folders
  command: "cp -Rp {{ splunk_home }} {{ splunk_backup }}"

- name: Stop splunk
  command: {{ splunk_home }}/bin/splunk stop

- name: splunk clean raft
  command: "{{ splunk_home }}/bin/splunk clean raft --answer-yes -auth admin:changeme"

- name: clean kvstore clustering
  command: "{{ splunk_home }}/bin/splunk clean kvstore --cluster --answer-yes -auth admin:changeme"

- name: start search head with search head with good kvstore
  command: {{ splunk_home }}/bin/splunk start
  when: "good_search_head"

- name: bootstrap shc with good_search_head
  command: "{{ splunk_home }}/bin/splunk bootstrap shcluster-captain -servers_list "https://<member_ip>:8089" -auth admin:changeme"
  when: "good_search_head"

- name: verify status
  command: "{{ splunk_home }}/bin/splunk show shcluster-status -auth admin:changeme"
  when: "good_search_head"

- name: Stop splunk
  command: {{ splunk_home }}/bin/splunk stop

- name: splunk clean raft on good_search_head
  command: "{{ splunk_home }}/bin/splunk clean raft --answer-yes -auth admin:changeme"
  when: "good_search_head"

- name: start all search heads in cluster
  command: {{ splunk_home }}/bin/splunk start

- name: bootstrap shc with good_search_head

  command: "{{ splunk_home }}/bin/splunk bootstrap shcluster-captain -servers_list "https://<member_ip>:8089,https://<member_ip>:8089,https://<member_ip>:8089,https://<member_ip>:8089" -auth admin:changeme"

References:
https://answers.splunk.com/answers/482209/why-is-the-kv-store-status-is-showing-as-starting.html
*note splunk clean raft is a better way to clear it

Using ansible to make API requests to Splunk

I used the following format to make a REST API call against Splunk using Ansible.

- name: REST API call to Encrypt passwords
  uri:
    url: https://localhost:8089/servicesNS/nobody/{{ item.app_directory }}/storage/passwords
    user: admin
    password: changeme
    method: POST
    body:
      app_username:
      app_password: "{{ item.app_password }}"
      app_realm: (optional)
    body_format: form-urlencoded

Note: form-urlencoded is only supported on later versions of ansible.

References:
http://www.georgestarcher.com/splunk-stored-encrypted-credentials/

Experience taking GREM certification

My experience taking the GREM certification

Tips and tricks

The GREM certification is one of the harder certifications that I have attempted.

There wasn't too much assembly in the exam. There are certain patterns in assembly that you should recall and identify quickly. Since this is a timed exam you could easily run out of time attempting to slowly work out the assembly code.

Study your javascript

Create an index
@hacks4pancakes has written an excellent blog post on this topic.
https://tisiphone.net/2015/08/18/giac-testing/

Study, learn, and attempt to understand the material.

Credit Freezes

When freezing your credit file remember to place it at the following places.

Major three credit bureaus

- Experian
- Equifax
- Transunion

In addition, the following places
- Innovis
- Sage Stream

References:

https://krebsonsecurity.com/2015/06/how-i-learned-to-stop-worrying-and-embrace-the-security-freeze/

Rewriting indexes.conf for volume based definitions

At some point when you move from a one tiered storage to a two tiered storage in Splunk where hot/warm buckets are on fast storage (SSD), and cold on slow storage (HDD), you may need to rewrite your indexes.conf

Rewriting your indexes.conf is a fairly easy exercise, but can go disastrously wrong. Just remember that thawed storage can not reference volumes. So double check that the same data locations are still referenced. I also recommend that you rework the Splunk internal indexes ie _internal, _telemetry to reference volumes.

When rewriting your indexes.conf. I recommend placing the index cluster into maintenance mode to prevent buckets from moving in the event there is a bug in your conf file. Then monitor your index cluster for unusual activity once the modified indexes.conf file is deployed. Anomalous activity can include the number of tasks / fixup tasks increasing drastically. In our situation, we saw the number of tasks go beyond 10,000 tasks.

If you see the following, likely something has gone wrong.

- Data may not be searchable temporarily
- Search factor may not be met
- Replication factor may not be met
- High number of fixups to meet search and replication factor

In my situation, somehow at the OS level, our symlink mapped to a different location with our new indexes.conf. Resulting in a high number of fixups and Splunk not seeing the data.