3

Question
I've a log-File which display all console-logs of my website in the range of 10.Oct to 1.Nov, start with the logs of 10.Oct.
I need all logs from the range of 25.Oct until today.

The problem is that there is a huge exception. Every day I get an IOException over 21 lines and that for 6030 users, so there are about 126'600 lines of code which I won't see; per day.

So I need to filter the output to be able to read the logs normal again.

What I already tried:

less catalina.out | grep -v "java.io.IOException: Server returned HTTP response code: 401 for URL"  

The log file is called catalina.out. But it don't my code don't hide the Exception, and I'm not able to scroll through the output.
So what i'm doing wrong? How should the code be?


INFO about me
I'm NOT familiar with Linux. I work in a support group, but today i'm alone and a website make some trouble. So I need to fix it by myself.
I google it now by 45 minutes but don't get it and the problem hurrys... So please be patient with me...


INFO's about the problem
Here all the facts and informations I know about my problem:
LOG-File Name: catalina.out
The whole IOException which I want to hide and get repeated 6030 times per day:

java.io.IOException: Server returned HTTP response code: 401 for URL: http://secure.intern.webpage/userpicture/u117054.jpg
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
        at java.net.URL.openStream(URL.java:1037)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.downloadUrl(PersonCleanupNightlyJob.java:270)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.downloadPic(PersonCleanupNightlyJob.java:245)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.doCleanup(PersonCleanupNightlyJob.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

The only thing which change in every exception is the picture-name. Everything else is always the same.

1
  • You'd better start with cat catalina.out | grep -v "..." | less which will restore scrolling ability. (First do the filtering, then aplpy less for viewing)
    – mveroone
    Nov 1, 2013 at 10:45

3 Answers 3

3

OK, genuinely interesting problem, given that it's got to function at the end of a pipe. There may be easier ways to do this, but I found this one worked (and it uses the sysadmin's Swiss Army Chainsaw, perl):

tail -f catalina.out | perl -n -e '{ if (/^java.io.IOException: Server returned HTTP response code: 401/) { $ignore=1;} elsif ($ignore>=1 and $ignore<20) {$ignore++;} else {$ignore=0; print $_;}  }'

Basically, it steps through STDIN one line at a time, looking for the search string ("^java.io.IOException: Server returned HTTP response code: 401"). Until it's found, it prints each line as it comes in; once it's found, it starts counting to 20, once for each new line in, printing nothing while it counts; once the count gets to 20, it resets the count to zero and resumes printing each new line in.

Edit: No problem. Start with tail -100000f catalina.out | ..., or indeed a larger number yet if the logfile has more lines than that. If you don't want to see the new stuff as it's added, try

perl -n -e '{ if (/^java.io.IOException: Server returned HTTP response code: 401/) { $ignore=1;} elsif ($ignore>=1 and $ignore<20) {$ignore++;} else {$ignore=0; print $_;}  }' < catalina.out.

It's designed for use in a pipeline, so you can feed it whatever you want stripped, and you can send the output onto whatever needs it. It's the UNIX way!

8
  • Why not ignoring all lines that foillow and begin with ` at ` instead of counting to 20 ? if the java call stack is different, your code breaks
    – mveroone
    Nov 1, 2013 at 11:04
  • Because the problem as specified by the OP was to ignore 21 lines which started with a given string. If (s)he'd specified it differently, I'd've coded it differently, and that would be valid in that case. I tend not to override the spec without good reason (ie, the spec is dumb); in this case, there's no reason to assume that the next line might not also begin with " at", and be very important.
    – MadHatter
    Nov 1, 2013 at 11:06
  • K.I.S.S. indeed.
    – mveroone
    Nov 1, 2013 at 11:07
  • When I run your code, don't read the whole log file. I just open it and get all NEW added lines. But I need the lines of the past 5 days, not these which are just added. Nov 1, 2013 at 12:09
  • +1. It's a huge code. I think with your answer, I learned most :) Nov 1, 2013 at 12:26
3

grep knows to take files as arguments. Please check the man. You could use egrep, instead of grep -e (which is the same):

# egrep -v '^java.io|^\ {8}at' catalina.out | less
3
  • +1 & accept. Works perfectly. Nov 1, 2013 at 12:27
  • Other question. With grep "2013-10-25" I only get all the entrys of this day. How can I add this to your code? so using the opposite of -v ? Nov 1, 2013 at 12:55
  • 1
    Since -v is applied for all grep command, you should pipe it into another grep. If you want to extend to more days: ...| egrep '2013-10-2[5-9]' | less.
    – Mihai
    Nov 1, 2013 at 13:48
1

If you want to totally ignore exceptions you may want to try this :

cat catalina.out | grep -v -e '^java.io.IOExc' -e '       at ' | less

(replace the 7 spaces by a tab [Ctrl+V then Tab] if that doesn't work)

If need more removal, we'll start perl-grepping.

4
  • I ran it but it donesn't work. And I can't add a tab... the output is the same like the output with only less catalina.out. Do you need more information...? Nov 1, 2013 at 12:06
  • +1 With your code, I found the (for me) simplest way to solve my issue. Nov 1, 2013 at 12:27
  • The working version of your code is cat catalina.out | grep -v "java.io.IOException" | grep -v "at" | less. Maybe you could edit your answer. Nov 1, 2013 at 12:35
  • Using -einsterad of grep twice works the same. Plus, I wouldn't remove all lines containing "at" you might miss a lot of them.
    – mveroone
    Nov 1, 2013 at 12:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .