Rewriting indexes.conf for volume based definitions

At some point when you move from a one tiered storage to a two tiered storage in Splunk where hot/warm buckets are on fast storage (SSD), and cold on slow storage (HDD), you may need to rewrite your indexes.conf

Rewriting your indexes.conf is a fairly easy exercise, but can go disastrously wrong. Just remember that thawed storage can not reference volumes. So double check that the same data locations are still referenced. I also recommend that you rework the Splunk internal indexes ie _internal, _telemetry to reference volumes.

When rewriting your indexes.conf. I recommend placing the index cluster into maintenance mode to prevent buckets from moving in the event there is a bug in your conf file. Then monitor your index cluster for unusual activity once the modified indexes.conf file is deployed. Anomalous activity can include the number of tasks / fixup tasks increasing drastically. In our situation, we saw the number of tasks go beyond 10,000 tasks.

If you see the following, likely something has gone wrong.

- Data may not be searchable temporarily
- Search factor may not be met
- Replication factor may not be met
- High number of fixups to meet search and replication factor

In my situation, somehow at the OS level, our symlink mapped to a different location with our new indexes.conf. Resulting in a high number of fixups and Splunk not seeing the data.