How can I configure splunk with log files residing on remote unix servers?

Normally I log into putty to a linux server, from there I ssh into another company server and I navigate through directories and perform my operations mainly as cat, zcat etc., with grep filters. Ex:

  1. login to example_server with putty
  2. ssh to ssh_server
  3. cd to req dir
  4. perform cat etc.,

ssh_server will not directly allow to be logged in from putty, I have to first login to example_server and then to ssh_server.

How can I configure these log files to be used by splunk to search for a string, similarly to how I use grep. I've installed splunk on my laptop, and by clicking add data > files and dir > add new it shows full path to the data field, what path should I fill it with?

 1) I can't do any modification to server(have no rights), so i couldn't use splunk    universal forwarder
 2)There's no way i can open port to write to and make splunk listen to it, as i said              my higher-ups won't allow, if something goes wrong, it will cost my job
 if there's anyway i can write to port securely ,that might help.
 3) I can run shell scripts FYI
 4)It will help if anyone suggest how i can securely connect to log servers, download data files(not manually) a python script or .bat file, and will use this local directory in splunk :)

Assuming that you already have a Splunk instance set up somewhere there are several ways to forward your log data to a central location.

First and foremost the easiest method is to send your syslog messages to the remote log server. This only applies if you have the necessary rights and permissions to do so and it is possible to get your desired logs to write to syslog.

If you have installation rights you can set up a Splunk forwarder instance on your remote box. This is supported on Linux, Unix and Windows, possibly more.

There are also additional tools that you can use to forward the logs including Epilog and forwarding the system logs.

Failing installing anything on the server I've set up scripts that effectively tail an output file and dump the results into a TCP or UDP connection opened to the Splunk server.

EDIT - since you responded that you don't have rights to install to this box you can look at copying your files via SSH and adding the destination folder for the copy to your Splunk index. Schedule this job to execute via cron. This isn't ideal - it allows a degree of tampering before your logs are sent off - but it is functional.

EDIT - In your case I would recommend using the nc command if you have it available. This is a great way for a no installation monitoring of logs.

tail /your/log/file -f | nc <your.server.ip.addr> <yourport>
  • I don't have any rights to do such kind of installations, need to get approval from higher-ups-so no Splunk forwarder instance soln and finally as u said i could dump results to TCP or UDP connection, i'm afraid i cannot open such connections too so, but is there a way to make splunk login to server and download the files, index them etc., ? – cypronmaya Dec 15 '11 at 13:58
  • @LAST EDIT as u said i would try using nc command, make it used by splunk to listen to.But i'm not sure i can use that nc even. as we are of support team, as i said could get required info using cat,zcat,grep,sort,etc., Anyway for now i will download logs manually, and point it to splunk. But it will be a heavy tiresome work, as we have nearly 25 logs servers ,which writes logs into. so i have to find a better option of it. – cypronmaya Dec 15 '11 at 14:52
  • as u said nc <your.server.ip.addr> <yourport> wouldn't this make any issue? as i've opened a port ,which might show lot of info ,if it got into wrong hands kind of thin ? – cypronmaya Dec 15 '11 at 14:53
  • The nc command is clear text. There are ways to use SSL to send the information (the splunk forwarder, among others). You need to analyze your requirements further. If you have additional questions please open a more detailed question. – Tim Brigham Dec 15 '11 at 15:05

With Linux systems and most log management solutions, you simply need to modify the syslog or rsyslog configuration on the Linux server to send all data to the logging server.

sudo vi /etc/syslog.conf . @X.X.X.X sudo /sbin/service syslog restart

Having this configuration will send all logs via port 514 to the Splunk logging server you have set up. You just have to ensure that you have set up Splunk to allow that traffic inbound. Most of the time this is done by setting up various hosts which allows X IP to send traffic into the machine.

Other logging implementations such as OSSEC allows you to install an agent on the machine so you don't have to modify syslog.

  • I'd like solutions which doesn't involve installing or configuring for port forward things and all. Simple way to download from the log server,index,analyze and store it locally :) – cypronmaya Dec 15 '11 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.