Creating Secure Transactions on Mac OS X server Using SSL


	Log In \| Not a Member?	Contact ADC

Creating
Secure Transactions on Mac OS X server Using SSL

In most common Internet applications, information is sent entirely in the clear. Mail clients carefully hide password data from other users of the same physical machine, but when they make a POP request to retrieve mail, they send strings like “USER bob PASS b4tm4n” unencrypted. Any miscreant listening anywhere along the line can find out Bob’s password. The same goes for information sent via Web forms, LDAP data, and so forth. This is a fine, low-overhead way to do things most of the time, but for certain critical applications—say, sending credit card information—a bit of encryption on the data would be very handy indeed.

Enter the protocol known as Secure Sockets Layer, or SSL.

The SSL protocol provides a layer of public-key encryption on top of any TCP connection. This allows for encrypted data and authentication on both ends. Most Web browsers have SSL capability built in, and Apple’s Mail client has supported SSL for POP and IMAP since 10.1.3. With Mac OS X Server v10.3, using SSL is easier and more powerful than ever.

This article is the second in a series on using Mac OS X Server for a small business local area network (LAN); an earlier overview article looked at Using Xserve for a Small Business LAN; this article addresses how to add security functionality to your Xserve or other Mac OS X Server configuration.

How It Works: Public-Key Cryptography

Public-key cryptography involves a pair of keys for each user: one public and one private. Data that is encrypted with the private key can only be decrypted with the public key, and vice versa. While the private key is closely guarded and never shared, the public key is distributed so that anyone can use it. This asymmetrical approach solves a traditional problem: when both parties share a single secret key, it is vital that they find a way of initially sharing the key without revealing it to anyone else. As more users are added to the system, this becomes more and more difficult.

By encrypting a message with a particular public key, it can be ensured that that message is only readable by the holder of the corresponding private key. Sender authentication can be accomplished by creating a digital signature with an algorithm that combines the sender’s private key with the content of the message. By evaluating this signature with the sender’s public key, the recipient can tell if the message actually came from the sender, and if it has been altered.

To establish that a particular key pair actually belongs to a particular entity, certificates are created by a trusted third party, known as a certificate authority (CA). These certificates, signed with the CA’s public key, attest that entity X has public key Y.

In this article, as we set up SSL for various services, we will be working with a few distinct pieces of data. There is the key, which is the secret snippet that is used to encrypt and authenticate data. The certificate, which accompanies the key, is proof that the key has been certified by a CA; that is, that a trusted third party vouches that I am who I claim to be. The CSR, or Certificate Signing Request, is just that —an encoded request for signing that is sent to the CA. Lastly, we have to deal with the CA’s certificate, proving that they are who they claim.

Setting Up SSL

As of version 10.3, Mac OS X Server provides built-in SSL encryption and authentication for POP, IMAP, SMTP, and LDAP, as well as for remote server management via servermgrd. The first step in getting SSL running on Mac OS X Server is to get a certificate. This can be done in two ways: the official way, by getting an official CA to sign your key pair; or the unofficial way, by acting as your own CA. The latter method will raise errors on users’ systems until they accept your authority, so it is not ideal for transactions with the public, but it’s just fine for testing and internal company use, and it is free to generate as many as desired, whereas official certificates start in the hundreds of dollars.

In this article, we’ll go over how to create keys and keychains, how to get keys signed, and how to use a signed key to implement SSL protection for various protocols.

First, we’ll set up SSL for mail—POP, IMAP, and SMTP. The first step is to create a keychain using CA.pl, the very convenient built-in command-line certificate management tool.

Working in the Terminal as root (sudo zsh), first issue the following command to simultaneously create a key and a CSR:

/System/Library/OpenSSL/misc/CA.pl -newreq

CA.pl prompts you for a secure passphrase. You may want your passphrase to be secure but memorable, so you can use DiceWare, a free method involving using dice to choose words from a list. Then CA.pl asks for Distinguished Name information—various pieces of information about my DNS, company, and name. Note that the “Common Name” must be identical to the hostname of the server on which the key will ultimately be used: i.e., your mail server.

The CSR and key are generated in the current directory, in a file called newreq.pem. When you enter:

cat newreq.pem

the system displays the file, which looks something like this:

Now, you can take this CSR to a Certificate Authority (CA) such as Thawte and Verisign. Using the CSR, you can purchase an SSL certificate from one of these CAs, and then use it to authenticate your email server.

Alternately, you can act as your own CA and sign my own certificate. As discussed above, this has the advantage of being free (the name-brand CAs charge hundreds of dollars per year for certificate signing), but users of your server will have to OK a message that the certificate authority is not recognized. We explain below how to sign a key.

Once the signed certificate is in hand, the next step is to place the certificate in a keychain, so the mail service can find it. This can be done using a tool called certtool.

First create the keychain, which we call “certkc”:

certtool c k=/Users/root/Library/Keychains/certkc

Note that “c” indicates that you want to create a new keychain; “k=certkc” gives the name of the keychain to create.

With the file saved on the server, the “i” function of certtool imports it into the keychain:

certtool i /path/to/certfile k=certkc

Finally, we need to configure a file containing the passphrase we assigned to our CSR in the first step, so that mail service can automatically view the key when it needs to verify a transaction. To do this, simply create a plain text file containing the passphrase. The file should be called “certkc.pass” and should be placed in the /private/var/root/Library/Keychains directory. The following command creates the file:

echo 'my very secure pass phrase' > /private/var/root/Library/Keychains/certkc.pass

Now, secure the file against prying eyes by removing unnecessary permissions:

chmod go= /private/var/root/Library/Keychains/certkc.pass

All that remains is to tell the mail service to use SSL. In Server Admin, go to Mail, and click on Settings. Then open the Advanced tab, and use the drop-down menus for SMTP SSL and IMAP and POP SSL. There are three options: Don’t Use, which is the default and disables SSL for that service; Use, which causes the mail service to use SSL if it’s available, but to drop back to an unencrypted connection if SSL support is not available; and finally, Require, which tells the mail service to permit only SSL connections, not unencrypted ones. Choose Require in both menus, for maximum security. This means that all mail clients that wish to connect to your server have to support SSL encryption for SMTP, IMAP, and POP connections.

Note: POP over SSL operates on port 995; IMAP over SSL on 993. These ports must be available and not blocked by a firewall in order for the service to work. If SSL is required for these services, then the unencrypted POP and IMAP ports, 110 and 143, can be closed. The Mac OS X version of SMTP over SSL operates not on the traditional port 465, but on port 25, the standard SMTP port. Mail clients should be made aware of that fact.

Acting as Your Own CA

OpenSSL (included in Mac OS X Server) provides the means not only to generate keys and CSRs, but also to sign keys and generate certificates. These self-signed certificates aren’t as widely accepted as those signed by a well-known CA, but they are perfect for some applications. The first step is to create a CA. Working at the command line as the superuser, as above, in our home directory, we issue the following command:

/System/Library/OpenSSL/misc/CA.pl -newca

It prompts for a CA certificate filename. To create a new one, just press Return. You are asked for a passphrase (different from the one above), and Distinguished Name information. It then generates a CA hierarchy in the directory demoCA, which is located in demoCA/private/cakey.pem. This will be used later.

Now this CA can be used to sign certificates. With your CSR is in the current directory, in the file called newreq.pem, enter the command:

/System/Library/OpenSSL/misc/CA.pl -signreq

It prompts for the CA passphrase, and outputs a signed certificate in the file newcert.pem. This certificate should be kept in a safe place.

Using SSL with the Web Server

The Mac OS X built-in Web server software, Apache, uses a module called mod_ssl to provide SSL security for Web sites. Mac OS X Server makes it easy to set up SSL with Apache, and here is a step-by-step guide for doing so.

The same SSL certificate obtained in the section above can be used for securing Apache; or a new one can be obtained for use with Web service via the same method. Below, we set up a secured site using Apache and SSL.

First, we place a copy of the certificate file, which for this purpose we call server.crt, into the directory /etc/httpd/ssl.crt.

We also place a copy of the key, which was generated using CA.pl, into a file called server.key and place the file in the directory /etc/httpd/ssl.key.

Finally, we need a copy of my CA’s certificate, which is available from their web site. We save this in a file called ca.crt in the /etc/httpd/ssl.crtdirectory.

Next, we go to the Web section of Server Admin, and select the Settings tab. We double-click the site for which we want to provide SSL, to bring up the Editing window. This site should be available on port 443, the port designated for secure HTTP traffic.

Click the Security tab, and check Enable Secure Sockets Layer, then type the passphrase for the certificate. In the three boxes labeled “Certificate File,” “Key File,” and “CA File,” we place, respectively, the locations of the server certificate (/etc/httpd/ssl.crt/server.crt), the server key (/etc/httpd/ssl.key/server.key) and the CA certificate (/etc/httpd/ssl.crt/ca.crt). We could also simply paste in copies of these keys if we preferred. Save the settings. When we restart Web service, SSL connections are enabled.

Setting up SSL for LDAP

The LDAP service in Mac OS X Server v10.3 has built-in SSL encryption. It’s easy to activate, too.

First we create the /etc/openldap/ssl directory, and place three files in it: the SSL certificate, the SSL key, and the CA certificate. Following the naming format above (though this is flexible), we call them respectively server.key, server.crt, and ca.crt.

In Server Admin, click Open Directory, and select an Open Directory Master or Replica machine. Then choose Protocols, and select LDAP Settings from the pull-down menu. To enable SSL, check the Use SSL checkbox, and enter the locations of the three SSL files. Then restart the LDAP service.

Note: It is also necessary to make sure that port 636 is open, as that is the port for SSL-secured LDAP traffic.

Posted: 2004-01-12