2

I'm using nginx (and OpenResty) and trying to hash the $request_body for caching purposes (with srcache).

When trying to just echo the $request_body, it works flawlessly:

# actually echoes the request body
location /works {
    # force loading the body
    echo_read_request_body;

    echo $request_body;
}

But when trying to calculate the md5, I get the MD5 for an empty string, although I ensure the $request_body is loaded via echo_read_request_body;.

# echoes d41d8cd98f00b204e9800998ecf8427e, which is the MD5 for empty input.
location /doesnt-work-md5 {
    # force loading the body
    echo_read_request_body;

    set_md5 $digest $request_body;
    echo $digest;
}

The weirdest phenomenon is that it doesn't even work when just trying to copy the variable:

# echoes nothing - empty data
location /doesnt-work {
    # force loading the body
    echo_read_request_body;

    set $temp $request_body;
    echo $temp;
}

By the way, these latest snippets don't work when using $echo_request_body instead of $request_body.

Thanks in advance!

5

With the help of the developers of OpenResty, I realized that the set_XYZ (i.e. set, set_md5) are evaluated during nginx's rewrite phase, while the $request_body/$echo_request_body are available only through the content phase.

So, when evaluating set_md5 $digest $request_body;, $request_body variable is empty, which explains the constant MD5 result.

Ultimately I implemented the actual cache key generation in my own API application (see example below), and accessed it with access_by_lua block.

The block runs in the access phase, before srcache_fetch and srcache_store are evaluated (they evaluate in post-access and output-filter, respectively).

Implementing it in my own API allowed greater control over the cache key generation logic, which would be hard to do with nginx alone (since I didn't want to become a full-pledged lua programmer).

For instance, I wanted to be able to deterministically cache POST requests with Json bodies. Json serialization is not deterministic since keys can be in any order. In my API, I sort the keys so the generated cache key is constant for the same data.

Also, it simplified handling of the $request_body, since the lua-issued subrequst just forwards it to the API, regardless of phase or disk buffering status.

The final config looks like

location /api {
    # proxy_pass ...
    # Force normal responses (no deflate, etc.) See https://github.com/openresty/srcache-nginx-module#srcache_ignore_content_encoding
    proxy_set_header  Accept-Encoding  "";

    set $cache_key "";
    access_by_lua_block {
        local res = ngx.location.capture('/generate-key' .. ngx.var.request_uri, {
                method=ngx.HTTP_POST,
                -- forwards the entire request body,
                -- regardless of disk buffering!
                always_forward_body=true,
                args=ngx.var.args
            })

        if res then
            ngx.var.cache_key = res.body
        end
    }

    # ... srcache options ...
}

location /generate-key {
    # proxy_pass ...
}

Example key generation API is as follows:

import flask
import json
import hashlib
import urllib


app = flask.Flask(__name__)

@app.route('/generate', defaults={'path': ''}, methods=['POST'])
@app.route('/generate/<path:path>', methods=['POST'])
def generate_cache_key(path):
    return urllib.quote('{}_{}_{}'.format(path,
        digest(stable_body()),
        digest(stable_json_dumps(flask.request.args))))


def stable_body():
    if flask.request.json:
        return stable_json_dumps(flask.request.json)

    return flask.request.data


def stable_json_dumps(data):
    return json.dumps(data, sort_keys=True)


def digest(data):
    return hashlib.md5(data).hexdigest()


if __name__ == '__main__':
    app.run()
1
  • I think md5_hash of $request_body should be its own lightweight NGINX plugin. This seems like a prime requirement to enable sensible caching for microservices, especially when they form any kind of pipeline. Sep 9 '19 at 16:14
-1

I think what is happening is that you're having a problem with multiple modules running in the locations you have setup. Without the rest of your config I am not sure though.

Can you move your last case to up in the config and test?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.