How to filter http request URL by mime type?

Question

I'm trying to make a shell script which filters http request URL by mime type. (ex: image/jpg image)

I used tcpdump to sniff packets and grep to filter http headers. This is my current command:

echo <password> | sudo -S tcpdump -vvAtp -i en5 tcp port 80 | grep -E GET\|Content-Type:.*image.*

This filters http URL and content-type header. I want some improvement, but I can't figure out how to do these:

• Prints URL when only Content-Type is image.
• There is no newline character before GET, so dirty characters prefixed before GET. I wanna remove those characters. Including "GET" or "HTTP 1.1" itself if possible.

Answer 1

Not exactly what you're looking for (which would require some sed/awk/perl fanciness), but I think you'll like this:

echo <password> | sudo -S tcpdump -vvAtp -i en5 tcp port 80 | grep -oE 'GET.*|Content-Type:.*image.*'

Well, here's a first try. Completely untested:

echo <password> | sudo -S tcpdump -vvAtp -i en5 tcp port 80 | grep -oE 'GET|Content-Type:.*image.*' | perl -npe 's/\n/#####/ if /GET/;' | grep -oE '#####[^#][^#]*$' 

Answer 2

Okay, this is enough of a departure to warrant another answer, especially since, this time, I could actually test it before posting it.

Here's my test string generator:

TESTSTRING='GET /foo/bar\nX-Random-Header: true\nContent-Type: text/html\nGET /foo/baz.jpg\nContent-Type: image/jpeg\nGET /index.html\nContent-Type: text/html\nGET /one/two.png\nContent-Type: image/png\nX-Another-Random-Header: 42\nGET /some.gif\nContent-Type: image/gif'
/bin/echo -e $TESTSTRING

It produces the following output:

GET /foo/bar
X-Random-Header: true
Content-Type: text/html
GET /foo/baz.jpg
Content-Type: image/jpeg
GET /index.html
Content-Type: text/html
GET /one/two.png
Content-Type: image/png
X-Another-Random-Header: 42
GET /some.gif
Content-Type: image/gif

Now here's the evolution of the output:

---

First step: Filter out lines that don't contain GET or Content-Type, and while we're at it, use the "-o" flag to filter out any strangeness at the start of those lines.

/bin/echo -e $TESTSTRING | \
 grep -oE 'GET.*|Content-Type:.*image.*'

Produces:

GET /foo/bar
GET /foo/baz.jpg
Content-Type: image/jpeg
GET /index.html
GET /one/two.png
Content-Type: image/png
GET /some.gif
Content-Type: image/gif

---

Step two: Remove newlines from all GET lines. This will "stack up" the GETs. Since the only lines that aren't "GETs" are the Content-Type lines, this means that we end up with one Content-Type per line.

/bin/echo -e $TESTSTRING | \
 grep -oE 'GET.*|Content-Type:.*image.*' | \
 perl -npe 's/\n/#####/ if /GET/;'

Produces:

GET /foo/bar#####GET /foo/baz.jpg#####Content-Type: image/jpeg
GET /index.html#####GET /one/two.png#####Content-Type: image/png
GET /some.gif#####Content-Type: image/gif

---

Step three: Get rid of everything except that last GET/Content-Type pair.

/bin/echo -e $TESTSTRING | \
 grep -oE 'GET.*|Content-Type:.*image.*' | \
 perl -npe 's/\n/#####/ if /GET/;' | \
 grep -oE '[^#]*#####[^#][^#]*$'

Produces:

GET /foo/baz.jpg#####Content-Type: image/jpeg
GET /one/two.png#####Content-Type: image/png
GET /some.gif#####Content-Type: image/gif

---

Step four: We can now strip off the Content-Type.

/bin/echo -e $TESTSTRING | \
 grep -oE 'GET.*|Content-Type:.*image.*' | \
 perl -npe 's/\n/#####/ if /GET/;' | \
 grep -oE '[^#]*#####[^#][^#]*$' | \
 sed 's/#####.*//; s/GET //;'

Produces:

/foo/baz.jpg
/one/two.png
/some.gif

Which, as I understand it, is your desired output.

Answer 3

Use ngrep, it brings the raw power of the grep to the network traffic. Here are some examples how to use it.