0

I am working on a web crawler, so I parse HTML pages. My problem is sometime the page encoding is not UTF8 (ISO, exotic Windows[0-9] etc..) and my analyser failled.

I tried many solution in PHP/Java/NodeJS to convert the content but there is always a problem.

Is exist a proxy module (nginx, squid, varnish ....) to convert automatically the content charset to UTF8?

2
  • 1
    Can't you analyse the html headers (for example with a regex) and convert them individually?
    – Jeredepp
    Jan 7, 2014 at 9:36
  • Yes, I am doing this and with HTTP headers as well, but I would like to separate this part of job Jan 7, 2014 at 10:26

1 Answer 1

1

The charset should be declared in the header - if it's not utf-8 then convert it - iconv is available on most flavours of Linux and Unix. If you're building a web crawler then it'd be easier to integrate in your code than in a proxy.

1
  • I want a proxy because I prefer split my project components, I already tried iconv and auto-detect but it doesnt work at 100%, I have many crawlers thats why I prefer isolate the encoding part. Jan 7, 2014 at 10:25

Not the answer you're looking for? Browse other questions tagged .