0

I was given a list of urls that are generating 404 errors as reported to us by google.

I can test a url with curl (from the command line ) like this:

curl -k --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)" https://MYURLHERE

which works exactly as I expected. I wanted to put this in a script so I can run through a list of them here is what I have.

#!/usr/bin/bash
url=$1

curlcmd="curl -k --user-agent \"Googlebot/2.1 (+http://www.google.com/bot.html)\""
$curlcmd $url

but its not working. I keep getting

curl: (1) Protocol "(+http" not supported or disabled in libcurl

I'm not sure how to escape this to make it work. any suggestions ?

0

put quotes around your variable $1, or you can use something like this:

$ touch $$
$ echo 'http://www.google.com' >> $$
$ echo 'http://www.yahoo.com' >> $$
$ for url in $(cat $$); do curl -I $url ; done
HTTP/1.1 200 OK
Date: Wed, 22 Nov 2017 15:57:19 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2017-11-22-15; expires=Fri, 22-Dec-2017 15:57:19 GMT; path=/; domain=.google.com
Set-Cookie: NID=117=CaOUCOyr9TPjs64tqyz1MuqHsASzL_3eO5n-NE4ubqAikITGbs7QY0aegNByOWX1Vaf9SsUVQDJ1wdaIOZwXoiqfVZ9ISLtta7tvcDH6LFM52OGFKRH4J5Clde2EX8oG; expires=Thu, 24-May-2018 15:57:19 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Age: 0
Transfer-Encoding: chunked
Via: 1.1 localhost.localdomain

HTTP/1.1 200 OK
Date: Wed, 22 Nov 2017 15:57:19 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2017-11-22-15; expires=Fri, 22-Dec-2017 15:57:19 GMT; path=/; domain=.google.com
Set-Cookie: NID=117=VRrA0-bCESlSCoerEK0n1hxXfldwpQI4cisiKrEgnKVph9HkfQJu-tbur3ZBiLh3-RFKZ0kbWUWsBwJKzsi_aPUuJzztM1rCuDfljZLxqjaHanZxiCx7qch4P2WCoDDC; expires=Thu, 24-May-2018 15:57:19 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Age: 0
Transfer-Encoding: chunked
Via: 1.1 localhost.localdomain

HTTP/1.1 200 OK
Date: Wed, 22 Nov 2017 15:57:19 GMT
Via: http/1.1 media-router-fp56.prod.media.ne1.yahoo.com (ApacheTrafficServer [c s f ]), 1.1 localhost.localdomain
Server: ATS
Cache-Control: no-store, no-cache, max-age=0, private
Content-Type: text/html
Content-Language: en
Expires: -1
X-Frame-Options: SAMEORIGIN
Content-Length: 12
Age: 0

$ 
  • I"m sorry i do not understand what you are telling me. I would like to run curl as a googlebot and pull down a url passed to the script so i can see what googlebot sees when it tries to crawl this url – General Foch Nov 22 '17 at 16:21
  • @GeneralFoch assuming your list of URLs inside of file, you can run against that list, or as I mentioned earlier you can put double quotes around your parameter, that should do the trick for you as well. – alexus Nov 22 '17 at 16:32
  • i placed the $url in double quotes, but still get the message curl: (1) Protocol "(+http" not supported or disabled in libcurl – General Foch Nov 22 '17 at 17:42
0

You may modify it like this:

#!/usr/bin/bash
url="$1"

curlcmd='curl -k --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)"'
$curlcmd "$url"

The message, you're getting, says that http (default) is not supported. Use https instead:

./test.sh https://www.somepage.com
  • I tried the the change above using single quotes for the curlcmd and then placing the $url inside double quotes. the url i pass is https . still get the same message as before. note when i run directly from the command line, i do not get that message. – General Foch Nov 22 '17 at 16:17
  • but you have to use https protocol (test.sh somepage.com) – Jaroslav Kucera Nov 22 '17 at 16:18
  • test.sh getspassed MYURLHERE ( all passed urls are https) i may not understand what you are telling me but the only http is in the curlcmd – General Foch Nov 22 '17 at 16:25
  • It works for me fine. I get your message only in case I try to access https only page via http. Is there a chance, that you have to use proxy at your site? – Jaroslav Kucera Nov 23 '17 at 7:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.