Enable secure email communication in Tomcat

When you want to use secure email communication in Nexus, you need to tell Java where to find the appropriate key.
In my case, I was using a non-official certificate to my mail host (I use dreamhost).
So what needed to be done is obtain and 'trust' the server's public key.

In order to get the public key, you need to make a connection to it from a terminal:
$ openssl s_client -connect mail.example.org:imaps

which resulted in the following output:
CONNECTED(00000003)
depth=0 /C=US/ST=California/L=Brea/O=Dreamhost.com/OU=Security/CN=*.mail.dreamhost.com/emailAddress=support@dreamhost.com
....
---
Certificate chain
0 s:/C=US/ST=California/L=Brea/O=Dreamhost.com/OU=Security/CN=*.mail.dreamhost.com/emailAddress=support@dreamhost.com
i:/C=US/ST=California/L=Los Angeles/O=New Dream Network, LLC/OU=Security/CN=New Dream Network Certificate Authority/emailAddress=support@dreamhost.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEXzCCAkcCCQDoyJJ40AXOXzANBgkqhkiG9w0BAQQFADCBxDELMAkGA1UEBhMC
VVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFDASBgNVBAcTC0xvcyBBbmdlbGVzMR8w
HQYDVQQKExZOZXcgRHJlYW0gTmV0d29yaywgTExDMREwDwYDVQQLEwhTZWN1cml0
eTEwMC4GA1UEAxMnTmV3IERyZWFtIE5ldHdvcmsgQ2VydGlmaWNhdGUgQXV0aG9y
aXR5MSQwIgYJKoZIhvcNAQkBFhVzdXBwb3J0QGRyZWFtaG9zdC5jb20wHhcNMDcw
NDEyMDA0ODU3WhcNMTcwNDA5MDA0ODU3WjCBoTELMAkGA1UEBhMCVVMxEzARBgNV
BAgTCkNhbGlmb3JuaWExDTALBgNVBAcTBEJyZWExFjAUBgNVBAoTDURyZWFtaG9z
dC5jb20xETAPBgNVBAsTCFNlY3VyaXR5MR0wGwYDVQQDFBQqLm1haWwuZHJlYW1o
b3N0LmNvbTEkMCIGCSqGSIb3DQEJARYVc3VwcG9ydEBkcmVhbWhvc3QuY29tMIGf
MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDoSVivPL+2qhBgKI6D4bl891MSue68
GUxx6xo6R3cGdGE038rYU5rKoWEoTKJ8BOcN5Hnbgl4OwdXg9HnDgDFfQtRjBL6N
BkRYBzmjPYU6G3DO9zxjibCeMaooa5Ax7Jzmi++7imhG4fQKpvNKIYtacvYsClK3
snaZi5CeNEFi+wIDAQABMA0GCSqGSIb3DQEBBAUAA4ICAQAg7LgbSoH+y81q3kDd
eBafzyXv6Zn4MJdy7Vhyg9EQUAPaJUPrfprMdl4v+y1WThFU2QfhXXeD2w5fPKSl
N+x8/QSh3wI2r1IINT34PHz1+z0+LArED7KZeNCS/8KxbjABo0WSTvMh9cg/qKkD
LuG4oL6kIX5GjCITGn+F1aL4BDvHABZ+Z/YM8XAQI4E7DBPLdVNnZLSXCme4tf8q
voc5bmrwFfBtwr9sRsG/jO2DeqmNoOEcUfoY/7Z+zhakaaGj1C63adrc64tJOp5t
/JFnr8Dtmf5LCXXXsNeuHc+0iNN8VJtc+WgQHDTjEu09/jKWVKJCYa/h/D4x661X
9nIBnPhHIbcHK29HIfC+lbryFFlgXxpwfYe/b2+VqiQ7ntLB1NA/aLC2kzHaNHWy
gC9P0CK0hvsFUwqd2eXm8I7C1v1DHJOp45IpxLDhIf6IT/S1807a2Fki2uea0RlE
0Lgock6hoQ/RvRR/fWU8466hLtEa4OLotRLwgbagbeVz/+2IWGKLEmlznl/nrB6a
reGTSdCMpArad40c2vvvL7zak6EyLPgBnNpj+gDRkMSXaV66gaSDVR7d2k/LvTsM
/COyEkqDJHjxzrv/IvWl+eLpF6EgAcVbKz49OlVOZ+VOex5WNujsRVIRepXexLgr
4l8gunP+gHaHyzYjXoMfKxAOZg==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Brea/O=Dreamhost.com/OU=Security/CN=*.mail.dreamhost.com/emailAddress=support@dreamhost.com
issuer=/C=US/ST=California/L=Los Angeles/O=New Dream Network, LLC/OU=Security/CN=New Dream Network Certificate Authority/emailAddress=support@dreamhost.com
...

The important part here is the server certificate starting with -----BEGIN CERTIFICATE----- and ending with -----END CERTIFICATE-----
Save that part, including the BEGIN and END part, to a file. I saved it as dreamhost_cert.pem.

Now that we have the server's public key, we need to add that key to the keystore and mark it as trusted as follows:
# keytool -import -alias mail.example.org -keystore /etc/java-6-sun/security/cacerts -file dreamhost_cert.pem
and then you get asked for the password, which if you haven't changed it, is changeit.

Then you get the following output:
Owner: EMAILADDRESS=support@dreamhost.com, CN=*.mail.dreamhost.com, OU=Security, O=Dreamhost.com, L=Brea, ST=California, C=US
Issuer: EMAILADDRESS=support@dreamhost.com, CN=New Dream Network Certificate Authority, OU=Security, O="New Dream Network, LLC", L=Los Angeles, ST=California, C=US
Serial number: e8c89278d005ce5f
Valid from: Thu Apr 12 02:48:57 CEST 2007 until: Sun Apr 09 02:48:57 CEST 2017
Certificate fingerprints:
MD5: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
SHA1: 6B:8C:79:AB:96:6D:70:27:7B:A8:6E:6F:82:08:59:A2:B5:B8:CC:C0
Signature algorithm name: MD5withRSA
Version: 1
Trust this certificate? [no]: yes
Certificate was added to keystore

So now that Java is aware of this new trusted key, we need to restart tomcat to enable secure email from Nexus.
Then login to Nexus as an administrator and go to the Administration - Server and use the following settings in the SMTP Settings:
hostname: mail.example.org
port: 587
TLS enabled: checked

Fill in the rest of the fields and then verify your settings by clicking on Test SMTP settings and you should see a message that it succeeded and that you need to check your email.