Installing Splunk agent for log collection

Overview:

This page describes the initial process for Splunk log collection.  The only server-side requirements  are 1) installation of the Splunk agent 2) configuration of that agent to ask the Splunk deployment server for its configuration.   A necessary follow-up step is a list of servers (hostname/DNS and IP), and the logs (paths to files, event logs, etc.) to collect from each system: this should be sent to security@.example.com or via a ServiceIT ticket routed to the Security Office.

Please note: Splunk can be configured on a server-by-server basis, because the applications on each server typically differ in a significant way along with the log sources that need to be collected. Additionally, because the Information Security Office provides financially for the license, we must whitelist newly incoming servers to ensure that we stay within the license limits and plan for growth. Finally, as a general rule, our office reviews only security-sensitive logs at a high level and, due to the significant churn in population across 1500 servers, each administrator as a subject-matter-expert must verify whether or not data being logged is as expected.

For any new system, configuration change on an old system, etc. this process will need to be repeated to:

1. install the agent if necessary
2. update hostname/IP if necessary
3. clear out the defunct previous configuration
4. adjust for any changes made or new applications that are logging.

Download the relevant agent for the target system:

Windows Server 2008, 2008 R2, 2012, and 2012 R2 (64-bit)
https://securityapps.example.com/splunk-deployment/6.4.2/windows/splunkforwarder-6.4.2-00f5bb3fa822-x64-release.msi

2.6+ kernel Linux distributions (64-bit x86)
https://securityapps.example.com/splunk-deployment/6.4.2/linux/splunkforwarder-6.4.2-00f5bb3fa822-linux-2.6-amd64.deb
https://securityapps.example.com/splunk-deployment/6.4.2/linux/splunkforwarder-6.4.2-00f5bb3fa822-linux-2.6-x86_64.rpm
https://securityapps.example.com/splunk-deployment/6.4.2/linux/splunkforwarder-6.4.2-00f5bb3fa822-Linux-x86_64.tgz

2.6+ kernel Linux distributions (32-bit x86)
https://securityapps.example.com/splunk-deployment/6.4.2/linux/splunkforwarder-6.4.2-00f5bb3fa822-linux-2.6-intel.deb
https://securityapps.example.com/splunk-deployment/6.4.2/linux/splunkforwarder-6.4.2-00f5bb3fa822.i386.rpm
https://securityapps.example.com/splunk-deployment/6.4.2/linux/splunkforwarder-6.4.2-00f5bb3fa822-Linux-i686.tgz

AIX (Power)

https://securityapps.example.com/splunk-deployment/6.4.2/aix/splunkforwarder-6.4.2-00f5bb3fa822-AIX-powerpc.tgz

Agent installation

Install the agent on any/every system from which you wish to forward logs, or may wish to in the future.  OS-specific details are below. The agent is very self-contained; all libraries and support files are included.  Windows will install into %ProgramFiles%. Unix-like operating systems can unpack in any location that is convenient; in this case, /opt is recommended.  Only minimal configuration is required; it is fully described below.

Lastly, send security@.example.com a list of all hosts upon which the client will be installed, including paths to log files. The agent communicates in an outbound-only manner (no inbound firewall rules are required). If you are doing egress filtering, the destination is IP_Address and 137.99.26.103, tcp ports 8089 and 40000.

The log intake is centrally configured, so no specific input configuration is needed on the server.

Windows instructions

1. Windows instructions: Run the MSI and, when prompted, enter splunkdeploy.example.com:8089 for the deployment server.  (Ignore all other config options).
2. Entering the following will allow you to install the agent silently without an installation wizard (avoiding the need to manually enter the deployment server):
3. msiexec.exe /i %YOUR PATH%\splunk-forwarder.msi DEPLOYMENT_SERVER=splunkdeploy.example.com:8089 AGREETOLICENSE=Yes /quiet
4. Start the service

Linux instructions

1. Install the package (choose appropriate command depending upon the package type downloaded):
   1. rpm -i $THE_RPM
   2. (within /opt) tar xf $THE_TARBALL
   3. dpkg -i $THE_DEB
2. Create $INSTALL_LOCATION/etc/system/local/deploymentclient.conf, with the following:
   [deployment-client]
   [target-broker:deploymentServer]
   targetUri = splunkdeploy.example.com:8089
3. Create an init script suitable for your distribution
   1. $INSTALL_LOCATION/bin/splunk enable boot-start
      To avoid running Splunk as root, you can pass in the -user parameter to specify which user to start Splunk as. For example, if Splunk runs as the user bob, then as root you would run:$INSTALL_LOCATION/bin/splunk enable boot-start -user bobMake sure it’s set to execute during runlevel initialization:

Redhat/CentOS/Fedora and derivatives: chkconfig splunk on

Debian/Ubuntu and derivatives: update-rc.d splunk defaults

4. Start the daemon : /etc/init.d/splunk start (or service splunk start).

Next steps

A list of servers (hostname/DNS and IP) along with the logs (paths to files, event logs, etc.) to be collected from each system should be sent to security@example.com with an optional CC to Splunk Admin. Each should be labelled in manner which will help locate the data (by you) rapidly without searching via a path or file name. Any convenient format is acceptable (plaintext, spreadsheet, etc).

For each server, list all log sources that should be collected (file path, Windows event log, etc.) as well as a tag that means something to you as an admin. E.g., for a custom webapp called VMprovisioning on server XYZ, the log file path might be /var/log/webapps/vmprov.log and the tag you supply might be “vmprov”. This is simply an aid to locate log data quickly.  The tag can be reused across multiple servers.  In splunk terminology, this is referenced as a “sourcetype.”

Log data collected by default :

all Linux default distribution logs in /var/log/* are collected. For specific applications also logging in /var/log, please specify.

all Windows event logs (Security, System, Application) and windows update logs. If logging of an item in the “Applications and Services” hierarchy is desired, please specify in the provided list of servers. The specific information present in each event log is highly system-specific (many 3rd party applications can emit something that lands in the Security or Application logs, for example).