Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
I am working on a web crawler, so I parse HTML pages. My problem is sometime the page encoding is not UTF8 (ISO, exotic Windows[0-9] etc..) and my analyser failled.
I tried many solution in PHP/Java/NodeJS to convert the content but there is always a problem.
Is exist a proxy module (nginx, squid, varnish ....) to convert automatically the content charset to UTF8?
The charset should be declared in the header - if it's not utf-8 then convert it - iconv is available on most flavours of Linux and Unix. If you're building a web crawler then it'd be easier to integrate in your code than in a proxy.
I want a proxy because I prefer split my project components, I already tried iconv and auto-detect but it doesnt work at 100%, I have many crawlers thats why I prefer isolate the encoding part.