4

I host a webpage that has 'project²' in the URL, matching an on-disk directory project² from where static files are hosted.

This page is used by a java-based client to load data from URLs (bioinformatics software IGV). My page lists URLs in the form of http://localhost:60151/load?file=http://example.org/project²/some/data/file.bam. Clicking these links in the browser will cause the IGV client (running on localhost) to request GET http://example.org/project²/some/data/file.bam from my server.

✅ IGV on Linux/Mac responds by requesting this URL as UTF-8 encoded ²=%C2%B2, and everything is happily working.
❌ My newly gained Win-10 user's client requests ²=%B2 (windows-1252 encoded), resulting in a 404-not-found.

After trying dozens of things, I am at wits end how to help this user.

I have the impression that I should be able to dynamically rewrite the wrongly-encoded URLs on the server-side, so that they still end up serving the desired data, but I do not know the magic character combinations to make the rule-patterns match escaped characters.


Things I've already tried

  • Doublechecking that the 404s are not network issues; I see the GET %B2 in my ssl_access_log with 404 as the returned statuscode, so it really is the server doing it.
  • 'Proper' way: UrlEncoding the URL before giving it into the client. Perl's URI::Encode encode_uri turns the ² into %C3%82%C2%B2 (apparently ò?) which is even more wrong somehow?
  • triple-checked that the webpage providing the load-URLs is served as utf-8
    • it provides header Content-Type: text/html; charset=UTF-8
    • Set AddDefaultCharset UTF-8 in httpd.conf
    • It seems the encoding info is not transferred through from webbrowser API-link-click into the Java program
  • 'doubled' the directory by symlinking projectª -> project² and project%B2 -> project² (ª is the UTF8-match for %B2) edit: ª is in no way related; no idea where I got that from
  • Tried to mod_rewrite 'bad' URLs into good ones in several different ways, none of which seem to catch:
RewriteEngine on
# RewriteRule Pattern Substitution [flags]
RewriteRule (.*)project%B2/(.*) $1project²/$2 [NE] # encoded 'bad' request, unencoded redirect
RewriteRule (.*)²(.*) $1%C2%B2$2 [B,NE]            # config file is utf-8 encoded, so this is senseless.      
RewriteRule (.*)%B2(.*) $12$2 [B,NE]               # doesn't match?        
RewriteRule (.*)TZZT(.*) $1test$2                  # works, so RewriteEngine is working

The RewriteRule and RewriteRuleFlags docs also do not help me understand how I should encode the Pattern-part so that it'll work :-(


Similar questions on here

3
  • 1
    "ª is the UTF8-match for %B2" - How did you derive this? This does not appear to be correct (which is part of the problem). ª (U+00AA "feminine ordinal indicator", not strictly "superscript-a") as UTF-8 would URL encode as %C2%AA. %B2 itself does not decode to anything printable in a UTF-8 encoding. – MrWhite Oct 6 '20 at 16:38
  • "ª": I found this in an online table that listed encoded values side by side for utf-8 and win-1252, and followed that in reverse direction. From what you describe, I massively misinterpreted that source, or the source is wrong. I can't for the life of me recreate the search terms that lead me to it, so I'll have to check the browser history of my work-pc, which I can't do until next week (currently work-from-home). I'll get back to it; for now I've editted the question to make it clear it's probably wrong. – Jules Kerssemakers Oct 7 '20 at 18:55
  • 1
    Finally at my work PC again; found the encoding source I used: teixeira-soft url encoding reference (has both windows-1252 and utf-8) It doesn't support my reasoning, so that seems to have been a brain-fart of me, probably delusional after too long stuck in the debugging hole... Question updated to reflect this and prevent confusion! – Jules Kerssemakers Oct 16 '20 at 7:09
2

You can't "convert encodings" as such using only mod_rewrite, however, you can search for that specific sequence of characters in the requested URL and "correct it".

http://localhost:60151/load?file=http://example.org/project²/some/data/file.bam
RewriteRule (.*)project%B2/(.*) $1project²/$2 [NE]

Note that project² appears as part of the query string in the example URL you posted, however, the RewriteRule pattern (which you are using above) matches against the %-decoded URL-path only (which excludes the query string). To match against the query string you need to use an additional RewriteCond directive and match against the QUERY_STRING (or THE_REQUEST) server variable instead.

Note that the QUERY_STRING (and THE_REQUEST) server variable is %-encoded (or rather, as sent from the client) - they have not been %-decoded.

Try the following instead:

RewriteCond %{QUERY_STRING} (.+)/project%B2/(.*)
RewriteRule ^(load)$ $1?%1/project%C2%B2/%2 [NE,L]

The backreferences %1 and %2 in the substitution string refer to the preceding CondPattern - the parts before and after the troublesome /project%B2/ part.

$1 is simply a backreference to the URL-path (to save repetition), which I assume is always load.

The NE flag prevents the % itself (when used as part of the URL-encoded characters) being URL encoded.

UPDATE: I'm afraid my original question was unclear about who GETs which URL, so the "query-string" part of your answer doesn't apply...

If you need to match the %-encoded URL-path then you should match against THE-REQUEST server variable instead. THE_REQUEST contains the first line of the HTTP request header and is not %-decoded. It contains the full URL-path (and query string) as sent from the client (as well as the request method and protocol version). For example, in the case of the malformed request, a string of the form:

GET /project%B2/some/data/file.bam HTTP/1.1

Which you could match and correct as follows:

RewriteCond %{THE_REQUEST} ^[A-Z]{3,7}\s(/project)%B2([^\s]+)
RewriteRule ^/?project %1%B2%C2%2 [NE,L]

%1 and %2 are backreferences to the captured subpatterns in the preceding CondPattern.

The RewriteRule pattern, on the other hand, matches against a pre-processed %-decoded URL-path only (as mentioned above). So, %B2 is whatever that decodes to; assuming a UTF-8 encoding. Unfortunately, this is a non-printable character so would need to be represented by the hex character sequence in the regex, ie. \xb2 (this is PCRE syntax representing a single byte sequence).

6
  • Thank you very much for thinking along! I'm afraid my original question was unclear about who GETs which URL, so the "query-string" part of your answer doesn't apply :-( I've updated my question to make this (hopefully) more clear. (What happens: Webpage has link to localhost:60151/load?/load?file=example.org/foo; clicking that will trigger IGV (listening on localhost) to do GET example.org/foo from my server, thereby taking the troublesome part 'back out' of the query-string) – Jules Kerssemakers Oct 2 '20 at 8:12
  • Second point: The fact that RewriteRule operates on the 'raw'/non-decoded URL is highly valuable information, which was previously unclear to me. I'll try if this helps me further and report back. – Jules Kerssemakers Oct 2 '20 at 8:17
  • 1
    I've updated my answer addressing the invalid char in the URL-path (as opposed to the query string). The solution is very similar - just using THE_REQUEST instead of QUERY_STRING. – MrWhite Oct 6 '20 at 1:12
  • 1
    "The fact that RewriteRule operates on the 'raw'/non-decoded URL is highly valuable information" - this is not true. (That's not what I stated above?) In fact, the opposite is true... the URL-path that the RewriteRule pattern matches against is %-decoded and pre-processed. I've also addressed this in my answer. – MrWhite Oct 6 '20 at 1:14
  • Ah, seems I misunderstood "they have not been %-decoded"; great clarification. And thank you for explaining why \x works; It seems obvious looking back, but it didn't occur to me in this context, despite the fact I eat Regex for breakfast... (so..many.. layers.. of processing, escaping, unescaping, and re-unescaping-for-printing.. argh!); %B2 decoded becomes a superscript-a, but when I wrote that into the rewrite-rule, it also didn't catch, even though it should be the same byte. – Jules Kerssemakers Oct 6 '20 at 7:53
2

Solution

RewriteRules must use \x instead of % in order to match %-encoded URLs! (PCRE syntax for byte sequences)

mod_rewrite-config uses PCRE regex syntax, and operates on decoded URLs, so typing a %-encoding in a RewriteRule pattern causes it to look for the literal %-character, not an encoded value.
The correct escape-character in RewriteRules is \x, so the URLencoded value %B2 can be matched using \xb2 (or \xB2, it's case-insensitive).

Note that RewriteRule is a hacky solution for character encoding issues, that only works when there is exactly one specific wrong-encoded character is in a specific, predictable place.

For a general solution for multiple wrong-encoded characters in arbitrary places, please see Can Apache .htaccess convert the percent-encoding in encoded URIs from Win-1252 to UTF-8? , which suggests a general solution using RewriteMap coupled to an external program in a full-featured programming language.

The proper solution is still to prevent this from the source, using explicit %-encoding throughout the entire chain. This avoids OS-dependent encoding accidentally happening 'somewhere in the middle', outside of your control. (assuming no client along the paths does double-encoding, which should be a punishable offense..)


How I got here

Getting desperate, I upped the server-wide logging using LogLevel Warn rewrite:trace3 as suggested in mod_rewrite docs. This is warned to (heavily) impact server performance, but was manageable because this is a low-traffic server, and there were no pre-existing rewrites.

The additional logging is emitted into (ssl_)error_log. This gave me insight into how exactly the matching was attempted, and what the internal representations for rules and URIs are in mod_rewrite.

excerpt from ssl_error_log (many columns ommitted for brevity), with rule RewriteRule (.*)project%B2/(.*) $1project²/$2 [NE,L]

[rewrite:trace3] applying pattern '(.*)project%B2/(.*)' to uri 'project\xb2/'
[rewrite:trace1] pass through /var/www/html/example.org/project\xb2

Note that the request-uri from client is written \xb2, but my pattern uses %B2.

Matching the rule-syntax to the uri-syntax, with rule RewriteRule (.*)project\xB2/(.*) $1project²/$2 [NE,L]

[rewrite:trace3] applying pattern '(.*)project\\xb2/(.*)' to uri 'project\xb2/'
[rewrite:trace2] rewrite 'project\xb2/' -> 'project%c2%b2/'
[rewrite:trace1] internal redirect with /auth-test/project\xc2\xb2/ [INTERNAL REDIRECT]

🎉 success! 🎉 As we can see, we are now matching!


Why no [R]/[R=302] flag?

As this is a character-encoding issue, I do not think doing an extra HTTP-round-trip will add value; Every link fed into the client will run into the same issue again, unless I fix the encoding issue before feeding it into the client-side java-program.


Don't forget RewriteBase

Please note that this shortened version omits setting the correct RewriteBase, which can screw up the rewritten path, depending on where in your conf it is written (e.g. <Directory> vs <Location>). Without RewriteBase I accidentally redirected to ❌https://example.org/var/www/html/rewrite-testing/project² instead of ✅https://example.org/rewrite-testing/project²)

3
  • 1
    \x is regex syntax (not particular to mod_rewrite) for representing any character using a hex code. So, \xB2 matches the literal character that %B2 (URL encoding) decodes to. Since the RewriteRule pattern matches against the %-decoded URL-path. Just as the regex \x68\x65\x6c\x6c\x6f matches the literal string "hello". %B2 is a non-printable character when UTF-8 decoded, so can only be represented by the hex code in the regex. – MrWhite Oct 6 '20 at 1:26
  • 1
    My windows user has confirmed that "it works now" (yay), so I'm selfishly accepting my own answer. @MrWhite; You've invested quite a lot of time into my question, and not even getting the covetted green-check, sorry! I'm not sure what the proper etiquette is for thanking you, other than thanking you. (Upvote of your answer seems wrong somehow, since even though well-researched, it didn't apply due to my unclear phrasing; sorry!) – Jules Kerssemakers Oct 6 '20 at 7:16
  • 1
    @MrWhite I Just realised you've massively updated your answer, you definitely deserve the upvote now! – Jules Kerssemakers Oct 6 '20 at 7:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.