Both Postfix and Dovecot use Certificate-Key pair that has same format with Apache Certificate-Key pair. Because of that, you can steal use solution from this thread: Converting a Java Keystore into PEM Format.
Convert it pkcs12 format
keytool -importkeystore -srckeystore foo.jks \
-destkeystore foo.p12 \
-srcstoretype jks \
-deststoretype pkcs12
To extract the certificate:
openssl pkcs12 -in foo.p12 -clcerts -nokeys -out servercert.pem
To extract the private key:
umask 0077
openssl pkcs12 -in foo.p12 -nocerts -nodes -out serverkey.pem
umask 0022
The last step is configure postfix/dovecot to use that pair. There are tons of tutorial on the internet to configure it.
Main parameter for postfix
# Server side TLS
smtpd_use_tls = yes
# private key file
smtpd_tls_key_file = /path/to/serverkey.pem
# key you generated at cacert.org
smtpd_tls_cert_file = /path/to/servercert.pem
Main parameter for dovecot
ssl = yes
# Preferred permissions: root:root 0444
ssl_cert = </path/to/servercert.pem
# Preferred permissions: root:root 0400
ssl_key = </path/to/serverkey.pem