dvvy 2.0.x Installation & Configuration

 

This document provides information relating to the installation and configuration of the dvvy Chargeback app for Splunk Enterprise.

For technical support, please email support@redfactorapps.com.

Define customers and implement data source tagging

Create an organization identifier (“Org ID”) structure for all of the teams, departments, lines of business, etc. This structure should reflect the way you intend to report on utilization and charges. Below is a simple example:

Group

Org ID

Network Operations Center (NOC)

noc

Security Operations Center (SOC)

soc

Application Development

appdev

Systems Engineering

syseng

 Once you have defined your Org IDs, add a stanza to your inputs.conf files (and any HEC sources) throughout your environment to create the index-time field (pipeline key). This field will establish data ownership for dvvy dashboards and reports.

Any data source that you want to track in dvvy must have an Org ID assigned. The dvvy app will automatically track data sources that contain the field.

inputs.conf Example:

NOC Data Source:

SOC Data Source:

1 2 3 4 5 [monitor:///var/log/syslog] _meta = org_id::noc disabled = false index = linux sourcetype = syslog
1 2 3 4 5 [monitor:///var/log/syslog] _meta = org_id::soc disabled = false index = linux sourcetype = syslog

 

The example above creates an index-time field of org_id with values of “noc” and “soc” respectively.

You are not required to use the field name org_id – any field name can be used. If you use a field other than org_id, you will need to update the field name in the KV – Track Data and Utilization Summary - Events reports.

Install the dvvy app

The dvvy app is installed on your search tier only (standalone or search head cluster). It relies upon the KV store and summary indexes. It is recommended that you follow Splunk best practices, to include forwarding your summary indexes from your search heads to your indexers.

Set global configuration

All global configuration is stored in the dvvyConfig KV store collection. Below is an explanation of the configuration parameters:

  • chargeModeLicenseGB: Controls how data ingest (GB volume) charges are calculated. Possible values: simple, over_entitlement

  • chargeModeRTSearchRuntime: Controls how real-time search runtime charges are calculated. Possible values: simple, over_entitlement

  • chargeModeSearchRuntime: Controls how search runtime charges are calculated. Possible values: simple, over_entitlement

  • chargeModeStorageTB: Controls how storage charges are calculated. Possible values: simple, over_entitlement

  • chargeModeVCPU: Controls how VCPU charges are calculated. Possible values: simple, over_entitlement

Charge Modes:

simple: The entitlement values in the customer configuration are used only for reporting and do not impact charge calculations (a customer with 100 GB of ingest and a 50 GB entitlement is charged for 100 GB). This was the only option prior to dvvy 1.5.0.

over_entitlement: The entitlement values in the customer configuration change how charges are calculation (a customer with 100 GB of ingest and a 50 GB entitlement is charged for 50 GB).

  • currencyUnit: The currency symbol to use in dashboards. Possible values: $, €,£, etc.

  • currencyUnitPosition: The position of the currency symbol in dashboards. Possible values: before, after

  • dvvyAdmin: A comma-separated list of Splunk usernames that have access to all dvvy customer data in dashboards.

  • indexerTargetGB: The daily indexing target per indexer in GB based on your architecture and sizing. Used for Indexer Load (IDXL) calculation, a simple methodology to represent indexer consumption as a whole.

  • splunkAdminCost: The daily cost of a Splunk Administrator/Engineer to use in admin chargeback calculations (ex: $150,000 yearly salary = $410.95 daily)

  • splunkAdminFlat: A flat daily fee to apply to all customers as part of admin chargeback.

  • splunkAdminForwarder: The quantity of forwarders a single Splunk admin can support (used to calculate Splunk Admin Load)

  • splunkAdminLicenseGB: The amount of license (GB) a single Splunk admin can support (used to calculate Splunk Admin Load)

  • splunkAdminStorage: The amount of storage (TB) a single admin can support (used to calculate Splunk Admin Load)

  • splunkAdminUsers: The number of users a single admin can support (used to calculate Splunk Admin Load)

See Recommended Configuration Workflow below for recommendations of working with data in KV store collections.

Configure customers

 Customers are defined in the dvvyGroups KV store collection. Below are the available parameters:

  • costForwarder: The daily rate per 100 forwarders

  • costIndexer: The daily rate for indexer load

  • costLicenseGB: The daily rate for 1 GB of data ingest

  • costRTSearchTime: The daily rate for 1 second of real-time search runtime

  • costSearchTime: The daily rate for 1 second of search runtime

  • costStorageColdTB: The daily rate for 1 TB of cold storage

  • costStorageHotWarmTB: The daily rate for 1 TB of hot/warm storage

  • costVCPU: The daily rate for 1 VCPU

  • group: The org_id value for the customer (i.e., “noc” or “soc” from the earlier example)

  • groupDisplay: The customer display name (i.e., friendly name) for dashboard presentation

  • licenseEntitlementGB: The amount of data ingest (GB/day) allocated to the customer for chargeback and reporting purposes

  • rtSearchTimeEntitlement: The amount of real-time search runtime (seconds) allocated to the customer for chargeback and reporting purposes

  • searchTimeEntitlment: The amount of search runtime (seconds) allocated to the customer for chargeback and reporting purposes

  • splunkRole: A comma-separated list of Splunk roles associated with the customer (used for Splunk Admin Load charge calculation)

  • storageEntitlementGB: The total amount of storage allocated to a given group for reporting purposes

  • techContact: A comma-separated list of Splunk usernames that are affiliated with the group and can access dvvy

  • vcpuEntitlement: The quantity of virtual CPUs (vCPU) allocated to the customer for chargeback and reporting purposes

See Recommended Configuration Workflow below for recommendations of working with data in KV store collections.

Update dvvy data tracking & populate environment details

Data Tracking

Once data is flowing with your Org ID defined and the other app configuration is in place, run the KV – Track Data report. This will write all data to the dvvyData collection.

This report is scheduled to run daily out of the box. Running it manually is only required to accelerate the implementation process.

The KV – Track Data report will run daily and append the following information to the dvvyData collection for any untracked data source:

  • group: The Org ID associated with the source type

  • idx: The index

  • st: The source type

  • costCenter: The cost center will be set to ‘default’ unless search is modified with lookup.

  • timestamp: Timestamp of when the data source was added to dvvy

Splunk Infrastructure

Run the KV – Splunk Infrastructure report to capture system specs of indexers and search heads in your environment. This search is not scheduled by default – please schedule to effectively capture any changes in your environment whether weekly, monthly, or yearly.

If your environment changes (e.g., add/remove an indexer or search head) and you do not run the report, VCPU reporting will not be accurate.

Review and update dvvy summary index configurations

The indexes.conf that ships with dvvy contains the Splunk default index configuration. Adjust the index configuration to meet your needs in terms of retention and many other considerations.

The dvvy_* summary indexes contain utilization data and *_cost_summary indexes contain summarized utilization and calculated charges. All dashboards reference the cost summary indexes.  

Indexes highlighted in the tables below were introduced in dvvy 2.0.

Utilization Summaries

Summary Name

Description

dvvy_admin_summary

Admin workload metrics

dvvy_event_summary

Event counts and customer distribution

dvvy_forwarder_summary

Forwarder counts

dvvy_infrastructure_summary

Infrastructure utilization

dvvy_license_summary

Data ingest

dvvy_search_summary

Search runtime

dvvy_storage_summary

Storage usage for hot/warm and cold buckets


Chargeback Summaries 

Summary Name

Description

dvvy_admin_cost_summary

Splunk administration load charges

dvvy_cpu_cost_summary

VCPU charges

dvvy_forwarder_cost_summary

Forwarder charges

dvvy_indexer_cost_summary

IDXL charges

dvvy_license_cost_summary

Ingest charges

dvvy_search_cost_summary

Search runtime charges

dvvy_storage_cost_summary

Daily storage charges

Add cost center information (optional)

Optionally add accounting cost centers to the dvvyCostCenters collection to further partition the data for reporting.

  • costCenter: The cost center code

  • costCenterDescription: The cost center display name for dashboards

If cost centers aren’t required, consider using this field as an internal identifier or reference number to enhance reporting. 

Validate & schedule searches

When you install dvvy, all of the necessary utilization and calculation reports are enabled and scheduled. By default, all jobs are scheduled to run overnight for the prior day’s events with the exception of the “Utilization Summary - License GB” search that runs every five (5) minutes by default.

Saved searches in Splunk run as the owner by default. When dvvy is installed, ownership will not be set (‘nobody’ ownership in meta) . This will cause the saved searches to interpret the cron schedules as UTC and then apply the timezone offset (i.e., -4 for EDT, etc.). This breaks the schedule by causing the searches to span two days.

Set ownership on all saved searches to an admin’s account with the correct local time zone via Reassign Knowledge Objects and restart Splunk. After restart, confirm that the saved searches starting with Utilization Summary – Events are scheduled to run at 12:03 AM.

Scheduled Search Sequence

Although the schedule, time ranges, etc. can be changed to meet your requirements, daily search jobs must execute in a specific order due to dependencies.

  1. Utilization Summary – Events

  2. Utilization Summary - Storage

  3. Utilization Summary - Forwarders

  4. Utilization Summary - Search

  5. Utilization Summary - CPU

  6. Cost Summary - License GB

  7. Cost Summary – Storage

  8. Cost Summary – Search

  9. Cost Summary - RT Search

  10. Cost Summary – CPU

  11. Cost Summary – Forwarders

  12. Cost Summary - Indexer Simple

  13. Utilization Summary - Admin - License GB

  14. Utilization Summary - Admin – Storage

  15. Utilization Summary - Admin – Users

  16. Utilization Summary - Admin – Forwarders

  17. Cost Summary – Admin

Consider removing the | collect command when testing searches to avoid writing data to summaries prematurely.

Contact us for scripts and other assets to streamline dvvy testing before going live.

Confirm dashboards are populating

 If you run the searches in the sequence above (manually or via our handy shell script), dashboards should populate. All dashboards only search against the *_cost_summary indexes.

 Dashboard filters are restricted to customer affiliation as defined by techContact in dvvyGroups. Any users listed in dvvyAdmin in the dvvyConfig collection will have access to all app data in the dashboards.

Once everything is populated to your liking, to include the charge calculations, clean (or | delete) all of dvvy indexes so that reporting is not skewed.

All RedFactor customers with a dvvy app license automagically have a zero-cost kvkit license. Kvkit is a lightweight Node/Express application that sits outside of Splunk and interacts with the the KV store via REST API. It can be used to create web-based forms that can be used to easily update dvvy configuration. 

In lieu of kvkit, please us the Lookup File Editor app.

KV Store Collections 

The dvvy app leverages KV store collections for all request operations. The table below lists the collections and their role.

Collections highlighted in the tables below were introduced in dvvy 2.0.

Collection Name

Description

dvvyConfig

Global app configuration options

dvvyCostCenters

Cost center information

dvvyData

Data sources tracked by dvvy

dvvyGroups

Group (OU) configuration

dvvySplunkInfrastructure

Splunk environment information

KV Store Backup

Since some of the app data is stored in KV store collections and collections are susceptible to accidental deletion or overwrite (e.g., unintentional outputlookup by an admin), it is highly recommended that you regularly backup all dvvy collections to prevent data loss.

 If you are using the kvkit application, you can easily make on-demand backups or schedule automated backups based on crontab syntax.

 If you are not using kvkit, you can perform backups via the CLI:

$SPLUNK_HOME/bin/splunk backup kvstore [-archiveName <archive>] [-collectionName <collection>] [-appName <app>]

 Using the above syntax as a guide, running this command:

/opt/splunk/bin/splunk backup kvstore -archiveName dvvyData -collectionName dvvyData -appName dvvy

will generate a .tar.gz archive in /opt/splunk/var/lib/splunk/kvstorebackup.

The archive will contain a JSON file with the contents of the dvvyData collection which could be used to restore.  

Consider scheduling this command with cron (or equivalent) or creating a simple shell script to streamline the process of backing up each dvvy collection. Please see Backup and Restore the KV store in the Admin Manual for official Splunk guidance on the topic.

Upgrading dvvy

Version 2.0 of dvvy is considered a major release. Changes were made throughout the app, to include new dashboards, saved searches, navigation, indexes, and KV store collection configuration.

Any changes made to dashboards, search logic, or other app assets will need to be manually integrated with the dvvy 2.0.

  • Backup all of dvvy’s KV store collections using kvkit, the Lookup File Editor, or the command line process.

  • Backup $SPLUNK_HOME/etc/apps/dvvy.

  • Upgrade the app via Splunk Web or install through other means appropriate for your environment.

  • Note that dvvy has four (4) additional summary indexes that are included in the indexes.conf.

  • If you made changes to the app and those changes have been addressed, remove the $SPLUNK_HOME/etc/apps/dvvy/local directory.

  • Update the KV – Track Data report with your Org ID (replace org_id with your field name).

  • Add vCPU, search, and charge mode configuration to the dvvyConfig and dvvyGroups collections.

  • Run the KV – Splunk Infrastructure report to capture details about your indexers and search heads.

  • Restart Splunk

Help = Available

If you would like more information or if you would like an assist with installation or configuration, please email support@redfactorapps.com.