Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Question
I've a log-File which display all console-logs of my website in the range of 10.Oct to 1.Nov, start with the logs of 10.Oct.
I need all logs from the range of 25.Oct until today.

The problem is that there is a huge exception. Every day I get an IOException over 21 lines and that for 6030 users, so there are about 126'600 lines of code which I won't see; per day.

So I need to filter the output to be able to read the logs normal again.

What I already tried:

less catalina.out | grep -v "java.io.IOException: Server returned HTTP response code: 401 for URL"  

The log file is called catalina.out. But it don't my code don't hide the Exception, and I'm not able to scroll through the output.
So what i'm doing wrong? How should the code be?


INFO about me
I'm NOT familiar with Linux. I work in a support group, but today i'm alone and a website make some trouble. So I need to fix it by myself.
I google it now by 45 minutes but don't get it and the problem hurrys... So please be patient with me...


INFO's about the problem
Here all the facts and informations I know about my problem:
LOG-File Name: catalina.out
The whole IOException which I want to hide and get repeated 6030 times per day:

java.io.IOException: Server returned HTTP response code: 401 for URL: http://secure.intern.webpage/userpicture/u117054.jpg
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
        at java.net.URL.openStream(URL.java:1037)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.downloadUrl(PersonCleanupNightlyJob.java:270)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.downloadPic(PersonCleanupNightlyJob.java:245)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.doCleanup(PersonCleanupNightlyJob.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

The only thing which change in every exception is the picture-name. Everything else is always the same.

share|improve this question
    
You'd better start with cat catalina.out | grep -v "..." | less which will restore scrolling ability. (First do the filtering, then aplpy less for viewing) –  Kwaio Nov 1 '13 at 10:45
add comment

3 Answers

up vote 3 down vote accepted

OK, genuinely interesting problem, given that it's got to function at the end of a pipe. There may be easier ways to do this, but I found this one worked (and it uses the sysadmin's Swiss Army Chainsaw, perl):

tail -f catalina.out | perl -n -e '{ if (/^java.io.IOException: Server returned HTTP response code: 401/) { $ignore=1;} elsif ($ignore>=1 and $ignore<20) {$ignore++;} else {$ignore=0; print $_;}  }'

Basically, it steps through STDIN one line at a time, looking for the search string ("^java.io.IOException: Server returned HTTP response code: 401"). Until it's found, it prints each line as it comes in; once it's found, it starts counting to 20, once for each new line in, printing nothing while it counts; once the count gets to 20, it resets the count to zero and resumes printing each new line in.

Edit: No problem. Start with tail -100000f catalina.out | ..., or indeed a larger number yet if the logfile has more lines than that. If you don't want to see the new stuff as it's added, try

perl -n -e '{ if (/^java.io.IOException: Server returned HTTP response code: 401/) { $ignore=1;} elsif ($ignore>=1 and $ignore<20) {$ignore++;} else {$ignore=0; print $_;}  }' < catalina.out.

It's designed for use in a pipeline, so you can feed it whatever you want stripped, and you can send the output onto whatever needs it. It's the UNIX way!

share|improve this answer
    
Why not ignoring all lines that foillow and begin with ` at ` instead of counting to 20 ? if the java call stack is different, your code breaks –  Kwaio Nov 1 '13 at 11:04
    
Because the problem as specified by the OP was to ignore 21 lines which started with a given string. If (s)he'd specified it differently, I'd've coded it differently, and that would be valid in that case. I tend not to override the spec without good reason (ie, the spec is dumb); in this case, there's no reason to assume that the next line might not also begin with " at", and be very important. –  MadHatter Nov 1 '13 at 11:06
    
K.I.S.S. indeed. –  Kwaio Nov 1 '13 at 11:07
    
When I run your code, don't read the whole log file. I just open it and get all NEW added lines. But I need the lines of the past 5 days, not these which are just added. –  dTDesign Nov 1 '13 at 12:09
    
+1. It's a huge code. I think with your answer, I learned most :) –  dTDesign Nov 1 '13 at 12:26
show 3 more comments

grep knows to take files as arguments. Please check the man. You could use egrep, instead of grep -e (which is the same):

# egrep -v '^java.io|^\ {8}at' catalina.out | less
share|improve this answer
    
+1 & accept. Works perfectly. –  dTDesign Nov 1 '13 at 12:27
    
Other question. With grep "2013-10-25" I only get all the entrys of this day. How can I add this to your code? so using the opposite of -v ? –  dTDesign Nov 1 '13 at 12:55
1  
Since -v is applied for all grep command, you should pipe it into another grep. If you want to extend to more days: ...| egrep '2013-10-2[5-9]' | less. –  Mihai Nov 1 '13 at 13:48
add comment

If you want to totally ignore exceptions you may want to try this :

cat catalina.out | grep -v -e '^java.io.IOExc' -e '       at ' | less

(replace the 7 spaces by a tab [Ctrl+V then Tab] if that doesn't work)

If need more removal, we'll start perl-grepping.

share|improve this answer
    
I ran it but it donesn't work. And I can't add a tab... the output is the same like the output with only less catalina.out. Do you need more information...? –  dTDesign Nov 1 '13 at 12:06
    
+1 With your code, I found the (for me) simplest way to solve my issue. –  dTDesign Nov 1 '13 at 12:27
    
The working version of your code is cat catalina.out | grep -v "java.io.IOException" | grep -v "at" | less. Maybe you could edit your answer. –  dTDesign Nov 1 '13 at 12:35
    
Using -einsterad of grep twice works the same. Plus, I wouldn't remove all lines containing "at" you might miss a lot of them. –  Kwaio Nov 1 '13 at 12:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.