0

I'm trying to make a shell script which filters http request URL by mime type. (ex: image/jpg image)

I used tcpdump to sniff packets and grep to filter http headers. This is my current command:

echo <password> | sudo -S tcpdump -vvAtp -i en5 tcp port 80 | grep -E GET\|Content-Type:.*image.*

This filters http URL and content-type header. I want some improvement, but I can't figure out how to do these:

  • Prints URL when only Content-Type is image.
  • There is no newline character before GET, so dirty characters prefixed before GET. I wanna remove those characters. Including "GET" or "HTTP 1.1" itself if possible.

3 Answers 3

1

Not exactly what you're looking for (which would require some sed/awk/perl fanciness), but I think you'll like this:

echo <password> | sudo -S tcpdump -vvAtp -i en5 tcp port 80 | grep -oE 'GET.*|Content-Type:.*image.*'

Well, here's a first try. Completely untested:

echo <password> | sudo -S tcpdump -vvAtp -i en5 tcp port 80 | grep -oE 'GET|Content-Type:.*image.*' | perl -npe 's/\n/#####/ if /GET/;' | grep -oE '#####[^#][^#]*$' 
1
  • Thanks. I got a hint this requires some kind of programming. I'll dig it later.
    – Eonil
    Jun 27, 2010 at 8:46
1

Okay, this is enough of a departure to warrant another answer, especially since, this time, I could actually test it before posting it.

Here's my test string generator:

TESTSTRING='GET /foo/bar\nX-Random-Header: true\nContent-Type: text/html\nGET /foo/baz.jpg\nContent-Type: image/jpeg\nGET /index.html\nContent-Type: text/html\nGET /one/two.png\nContent-Type: image/png\nX-Another-Random-Header: 42\nGET /some.gif\nContent-Type: image/gif'
/bin/echo -e $TESTSTRING

It produces the following output:

GET /foo/bar
X-Random-Header: true
Content-Type: text/html
GET /foo/baz.jpg
Content-Type: image/jpeg
GET /index.html
Content-Type: text/html
GET /one/two.png
Content-Type: image/png
X-Another-Random-Header: 42
GET /some.gif
Content-Type: image/gif

Now here's the evolution of the output:


First step: Filter out lines that don't contain GET or Content-Type, and while we're at it, use the "-o" flag to filter out any strangeness at the start of those lines.

/bin/echo -e $TESTSTRING | \
 grep -oE 'GET.*|Content-Type:.*image.*'

Produces:

GET /foo/bar
GET /foo/baz.jpg
Content-Type: image/jpeg
GET /index.html
GET /one/two.png
Content-Type: image/png
GET /some.gif
Content-Type: image/gif

Step two: Remove newlines from all GET lines. This will "stack up" the GETs. Since the only lines that aren't "GETs" are the Content-Type lines, this means that we end up with one Content-Type per line.

/bin/echo -e $TESTSTRING | \
 grep -oE 'GET.*|Content-Type:.*image.*' | \
 perl -npe 's/\n/#####/ if /GET/;'

Produces:

GET /foo/bar#####GET /foo/baz.jpg#####Content-Type: image/jpeg
GET /index.html#####GET /one/two.png#####Content-Type: image/png
GET /some.gif#####Content-Type: image/gif

Step three: Get rid of everything except that last GET/Content-Type pair.

/bin/echo -e $TESTSTRING | \
 grep -oE 'GET.*|Content-Type:.*image.*' | \
 perl -npe 's/\n/#####/ if /GET/;' | \
 grep -oE '[^#]*#####[^#][^#]*$'

Produces:

GET /foo/baz.jpg#####Content-Type: image/jpeg
GET /one/two.png#####Content-Type: image/png
GET /some.gif#####Content-Type: image/gif

Step four: We can now strip off the Content-Type.

/bin/echo -e $TESTSTRING | \
 grep -oE 'GET.*|Content-Type:.*image.*' | \
 perl -npe 's/\n/#####/ if /GET/;' | \
 grep -oE '[^#]*#####[^#][^#]*$' | \
 sed 's/#####.*//; s/GET //;'

Produces:

/foo/baz.jpg
/one/two.png
/some.gif

Which, as I understand it, is your desired output.

1

Use ngrep, it brings the raw power of the grep to the network traffic. Here are some examples how to use it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .