How to verify signature on a file using OpenSSL with custom engine

Question

Update Dec 28, 2017 – 3:

The author of OpenSSL DSTU module kindly provided patch to OpenSSL+DSTU implementation with a fix for the issue, and assisted further.

I was able to accomplish what I need first with this command:

./apps/openssl smime -verify -noverify -in my_message.txt.p7s -engine dstu -inform DER
engine "dstu" set.
Hello, world!
Verification successful

And later after concatenating a chain of certificates into a bundle.pem, I was able to do this:

./apps/openssl smime -verify -CAfile bundle.pem -in /yo/my_message.txt.p7s -engine dstu -inform DER
engine "dstu" set.
Hello, world!
Verification successful

Update Dec 28, 2017 – 2:

The author of OpenSSL DSTU module confirmed that the module is not working properly at the moment – https://github.com/dstucrypt/openssl-dstu/issues/2#issuecomment-354288000.

I guess I'll have to look elsewhere to find a proper DSTU4145 implementation. I've just learned about a BountyCastle project, and it's specification includes DSTU-4145. I guess there's no options left but to write some Java code to do perform signature verification.

Update Dec 28, 2017 – 1:

Here are my files:

• the message to be verified: https://www.dropbox.com/s/pt7ms096lygz8es/my_message.txt.p7s?dl=0
• the message to be verified, in ASN.1 format: https://gist.github.com/gmile/a9bb5cb57fc8195d74029251eb3946ba
• certificate(s) I'm trying to verify with: https://acsk.privatbank.ua/arch/docs/PrivatBank.zip

---

I have a file, signed by someone with his private key: signed_content.txt. I also have a certificate from CA. The private key and certificate are somehow related to each other.

How do I verify the signature on a file?

This is what I'm doing:

1. Extract the public key from certificate (obtained from authority):

   openssl x509 -pubkey -inform der -in PrivateCerts/CA-3004751DEF2C78AE010000000100000049000000.cer -noout -engine dstu > public_key.txt
   

2. Attempt to verify the contents of the file:

   openssl rsautl -verify -in my_message.txt.p7s -inkey public_key.txt -pubin -engine dstu
   engine "dstu" set.
   openssl (lock_dbg_cb): already locked (mode=9, type=18) at md_rand.c:387
   openssl (lock_dbg_cb): not locked (mode=10, type=18) at dstu_rbg.c:87
   Error getting RSA key
   139964169291424:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:p_lib.c:288:
   

Also, how do I extract the actual contents of the signed file?

---

Is the file I have is incorrect somehow? I can view it's ASN.1 contents:

openssl asn1parse -inform DER -in my_message.txt.p7s -i

The asn.1 structure seems to look OK (honestly, I know too little about ASN.1): I can see some fields about organization and stuff.

I'm using a DSTU engine (Ukrainian crypto standard), similar to GOST (Russian crypto standard).

Answer 1

openssl rsautl handles only the RSA algorithm, not any other algorithm: not DSA, not ECDSA, not GOST, not DSTU, etc. openssl pkeyutl -sign/-verify can handle any algorithm available through the standard EVP interface(s), which your engine presumably should.

However, most signature algorithms actually sign a hash of the data not the original data. In particular I see BouncyCastle has several signature schemes using GOST3411 (a hash) with DSTU4145 (and with or without LE aka Little-Endian encoding). For that, you either need to explicitly hash and then use openssl pkeyutl, or more easily use openssl dgst -$hashname -sign/-verify which combines them for you. For builtin hashes you can abbreviate this to openssl $hashname -sign/-verify but I don't know if that works for an engine hash.

In any case you almost certainly don't want to treat all of signed_content.txt as the data, much less as the hash of the data. If it has ASN.1 structure it probably includes the signed data (as only part of the structure) plus the signature value, and likely metadata or even other data. If it is a common structure and you post the asn1parse result, with any data values that you consider sensitive suppressed but all metadata like OIDs intact, I or someone else here might recognize it and advise. If it is the most common structure, CMS/PKCS7, OpenSSL commandline can handle that directly.

Answer 2

How do I verify the signature on a file?

The simple openssl smime -verify ... should work even with dstu engine:

$ openssl1.0 smime -verify -noverify -engine dstu -inform DER -in my_message.txt.p7s

engine "dstu" set.

Hello, world!

Verification successful

Is that what you need? When I remove the option -noverify, I get the verification failure Verify error:unable to get local issuer certificate, but it's related to certificate self verification, not the message. My openssl1.0 tool is like regular openssl 1.0.2o with gost and dstu patches applied.