Debilitating NTLM proxy

It is interesting to note how a corporate NTLM proxy can cause untold misery.

The saviour in this case is cntlm but I had to set it up in the following ways.

Environment

admin@LINUX:~/graphite/graphite-web-0.9.9/webapp/graphite$ env | grep "proxy"
http_proxy=http://127.0.0.1:3129
ftp_proxy=http://127.0.0.1:3129
https_proxy=http://127.0.0.1:3129

~/.bashrc

export http_proxy=http://127.0.0.1:3129
export https_proxy=http://127.0.0.1:3129
export ftp_proxy=http://127.0.0.1:3129

After all this I had to set it up again for a ruby gem installation.

sudo gem install fpm --http-proxy=http://127.0.0.1:3129

Sometimes I had to watch the verbose output from cntlm to be sure that I am not
being fooled by a wrong setup.

Painful SSL experience

I have dabbled in activities related to SSL for a long time but each time it starts with a painful experience. Sometimes it also ends painfully. After reading articles, the Java API and using various tools and creating CSR’s and coding Symmetric encryption routines one would feel like a security expert.

Actually there is no reason to feel like an expert because we are only dabbling in security. So my explanations could be fraught with inaccuracies.

This whole affair is amateurish but I can at least blog about this.

Standards and RFC’s

I feel that they are very tricky. The first time I established Two-way SSL I felt that I have understood the difficult SSL protocol. I couldn’t have been more wrong about that because the procedure I followed then was only one of several ways to accomplish this. There are various formats like .pfx, .DER, .crt, .pem etc. and tools like OpenSSL help you convert one into the other.

In addition to that there seems to be various key store formats.

The bottom-line is that if one could read RFC 2246 and understand and code SSL libraries using Java API’s or BouncyCastle then everything would seem to be clear. But this information is quite complex and will be forgotten quickly.

CSR’s, TrustStores and KeyStores

The second time I generated a CSR I was tripped by a quirk which I haven’t understood fully. When I tried to import a CA-signed public key and the certificate chain back into my keystore I faced this error.

java.security.UnrecoverableKeyException: Cannot recover key. 

This exception may result from the fact that you had provided a key password that was different from the keystore password when you generated the self-signed certificate

I realized that in fact the keystore and key passwords were different but I could not locate any such restriction anywhere.

Now the CA has sent the signed certificates already and I am not in a position to generate a new CSR and start the process all over again because some worthless managers are breathing down my neck.

Fortunately some coders have published libraries to do tricky security stuff. One such library is found at http://juliusdavies.ca/commons-ssl/download.html and the author recommended that I build my key store again and use a common password for the key as well as the store.

java -cp not-yet-commons-ssl-0.3.11.jar org.apache.commons.ssl.KeyStoreBuilder 'password' rsa.key

How do I get the private key from the Java KeyStore ?

It looked like J2SE 6 keytool allows us to export the contents including the private key from the store. These commands fail in J2SE 7.

"C:\Program Files\Java\jdk1.6.0_18\bin\keytool" -importkeystore -srckeystore keystore.jks -destkeystore mystore.p12 -srcstoretype JKS -deststoretype PKCS12 -srcstorepass srcstorepass -deststorepass deststorepass -srcalias srcalias  -destalias destalias -srckeypass srckeypass -destkeypass destkeypass -noprompt

I had Cygwin and OpenSSL and I could execute this next step to convert pkcs12 the keystore that was created to a pem format using openssl

 openssl pkcs12 -in mystore.p12 -out mystore.pem -passin pass:mysecret -passout pass:mysecret

So after this all the contents of the original key store are exported and ready to be built into a fresh KeyStore using not-yet-commons-ssl-0.3.11.jar and the procedure shown above.

Renewal of Certificates

The procedure for this is also different for IIS and simple Java Key stores. I received a new intermediate certificate from the CA because the old one expired. Now I tried to refresh my KeyStore and TrustStore but that was not quite easy till I understood how the certificates are stacked and sent.

This is RFC 2246

   certificate_list
       This is a sequence (chain) of X.509v3 certificates. The sender's
       certificate must come first in the list. Each following
       certificate must directly certify the one preceding it. Because
       certificate validation requires that root keys be distributed
       independently, the self-signed certificate which specifies the
       root certificate authority may optionally be omitted from the
       chain, under the assumption that the remote end must already
       possess it in order to validate it in any case.

So the original certificate generated along with the public/private key pair was
signed by the sub root certificate which was in turn signed by the CA’s root. The CA had sent me a new sub root or intermediate certificate.

Another painful experience because I had to build the KeyStore again using not-yet-commons-ssl-0.3.11.jar and the same procedure shown above.

What if someone hadn’t published a library for this ? What is the alternative ? OpenSSL ?

I am not sure but there are a few libraries like this for manipulating Java KeyStores and viewing and converting certificates. These libraries complement each other in some cases and the desired result can be achieved without fully understanding ‘Security’.

Update :

It seems to me that still I have not understood enough about certificate renewal. Now there is a simpler procedure which I have missed earlier.

If we just export each old certificate – public key certificate, intermediate, root- and then just replace the old intermediate with the new one and import, the chain is refreshed in the KeyStore.
The order of certificates in the .pem file should be

1. Public key certificate
2. New Intermediate
3. Root

Software Assurance Maturity models

I came across links to Software Assurance maturity models in the current issue of
Crosstalk.

There are so many maturity models to choose from but to the uninitiated they might
seem to be too theoretical.

1. “Build Security In Maturity Model (BSIMM): www.bsimm2.com
2. Capability Maturity Model Integration (CMMI): www.sei.cmu.edu/cmmi/index.cfm
3. Open Software Assurance Maturity Model (OSAMM): www.opensamm.org
4. Assurance Focus for CMMI: https://buildsecurityin.us-cert.gov/swa/downloads/Assurance_for_CMMI_Pilot_version_March_2009.pdf
5. CERT Resilience Management Model (RMM): www.cert.org/resilience/rmm.html
6. Build Security In website: https://buildsecurityin.us-cert.gov/bsi/home.html
7. SwA Community Resources and Information Clearninghouse: https://buildsecurityin.us-cert.gov/swa/
8. Making Security Measurable website: measurablesecurity.mitre.org/“

They serve as good points of reference and cover governance. As far as practical technical guidance is concerned I would look elsewhere.

They drive home the point that security has to be built into the product and have suggestions to address different compliance requirements like PCI-DSS using a process.

Bouncy Castle – Encrypt a stream

I should be forgiven for thinking that encrypting a stream of data instead of a disk file is more difficult than it is supposed to be. Actually when somebody like me who has no experience with writing secure code uses encryption/decryption libraries like Bouncy Castle, it is always hard. So even after the PCI auditor has paid us a visit many times I can still point out many loopholes in our security code like the AES key that is hard-coded. Split key technology is really hard to grok. So I have not attempted it yet. So the AES key is still inside the java class.

This example uses Bouncy Castle to decrypt a file using a GPG key and encrypt it again using another key on the fly. Some parts of the code that are supposed to be obvious are not shown but as I mentioned secure code is hard to write. The missing piece can be reconstructed from the Bouncy Castle examples.

I am planning to fix the code formatting in the blog soon.

Decrypt a file and return a stream

public InputStream decryptToStream( String file ) throws KeyNotFoundException{

PGPEncryptedDataList enc = null;

PGPObjectFactory objectFactory = null;

InputStream in = null;

try {

objectFactory = new PGPObjectFactory( new FileInputStream( file ) );

Object o = objectFactory.nextObject();

// What is a PGP marker packet ?

if ( o instanceof PGPEncryptedDataList ) {

enc = ( PGPEncryptedDataList ) o;

} else {

enc = ( PGPEncryptedDataList ) objectFactory.nextObject();

}

Iterator it = enc.getEncryptedDataObjects();

PGPPublicKeyEncryptedData pbe = null;

while ( it.hasNext()){

pbe = (PGPPublicKeyEncryptedData)it.next();

}

InputStream clear = pbe.getDataStream( getPGPSecretKey( pbe.getKeyID() ), "BC" );

PGPObjectFactory plainFact = new PGPObjectFactory( clear );

Object message = plainFact.nextObject();

if (message instanceof PGPCompressedData) {

PGPCompressedData cData = (PGPCompressedData) message;

PGPObjectFactory pgpFact = new PGPObjectFactory(cData.getDataStream());

message = pgpFact.nextObject();
}

if (message instanceof PGPLiteralData) {

PGPLiteralData literal = (PGPLiteralData) message;

in = literal.getInputStream();

}
} catch (FileNotFoundException e) {

} catch (IOException e) {

} catch (NoSuchProviderException e) {

} catch (PGPException e) {

}
return in;
}

Encrypt the stream and write to a file

It should be possible to return a stream from this method instead of writing it to a file.

public void encrypt( PGPPublicKey key,
                               String compressedfile,
                               String encryptedfile,
                               InputStream in ) throws IOException,
PGPException,
NoSuchProviderException {
byte[] buf = new byte[ 1024 ],
literalBuffer = new byte[ 1024 ],
encryption = new byte[ 1024 ];

ByteArrayOutputStream o = null;

OutputStream out = null;

FileOutputStream fos = new FileOutputStream( new File( encryptedfile ));

PGPCompressedDataGenerator compressedData = new PGPCompressedDataGenerator(
PGPCompressedDataGenerator.ZIP);
PGPLiteralDataGenerator lg =
new PGPLiteralDataGenerator();

try {

PGPEncryptedDataGenerator data =
new PGPEncryptedDataGenerator( PGPEncryptedData.CAST5,
true,
new SecureRandom(),
"BC");

data.addMethod( key );
int bytes = 0;

OutputStream writeToEncrypt = data.open( fos,
encryption );

OutputStream lgo = lg.open( compressedData.open( writeToEncrypt ),
PGPLiteralData.BINARY,
compressedfile,
new Date(),
literalBuffer
);

while ((bytes = in.read( buf )) >= 0) {
//As it is written it is encrypted.

lgo.write( buf, 0 , bytes );

}

lg.close();
compressedData.close();
writeToEncrypt.close();
lgo.close();

} catch (IOException e) {

} catch (NoSuchProviderException e) {

} catch (PGPException e) {

}
}

: : : : : : : : : : :

PCI requires good security

We generally add security after building the product. It is a serious issue because security is not built into the product and we do not know cryptography well enough to be able to write good code.

This Java class was coded quickly just to encrypt and decrypt using AES 256 without understanding the foundations or key security principles. Even this took a long time and still there are holes in the way I have understood it but it is a good starting point.
It is also a good idea to subscribe to forums like dev-crypto@bouncycastle.org and read. I am planning to write Java code using the GPG API in the Bouncy Castle library.

This code uses the Bouncy Castle Provider but I do not think it is needed. I also used the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction libraries.

package com.encryption;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.security.Security;

import javax.crypto.Cipher;
import javax.crypto.CipherInputStream;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;

import org.bouncycastle.jce.provider.BouncyCastleProvider;

/**
 * A AES 256 encryption routine. 
 */

public class AESKeyEncryption {
	
	 

	     public static void encrypt( File in,
	    		                     File out,
	    		                     Cipher cipher ) throws Exception { 
			byte[] buf = new byte[1024];

			FileInputStream fin = new FileInputStream( in );
	        
			final CipherOutputStream cout = 
	        	new CipherOutputStream( new FileOutputStream( out ), cipher );
	        
			try {

	            int bytes = 0;
	        
	            /* Assumption is that there is no guarantee that
	             * there will be a full buffer every time. Isn't that
	             * what is called a subtle bug ;-)
	             */
	            while ((bytes = fin.read( buf )) >= 0) {
	            	cout.write( buf, 0 , bytes );
	            }
	            
	            cout.close();
	            fin.close();
	        } catch ( IOException e ) {
				
	        	System.out.println( "IOException");
	       }finally{
	    	   if( null != cout ){
	    		   cout.close();
	    	   }
	    	   if( null != fin ){
	    		   fin.close();
	    	   }
	    }
	     }
	     

	     public static void main(String[] args) throws Exception { 

                Security.addProvider(new BouncyCastleProvider());

                /*
                 * Supposed to be picked up from the database
                 * and salted if possible.  It is a 32 byte(256-bit) Key
                 */
                String key ="11112444123123222444200012311111";
	            
                byte[] raw = key.getBytes("UTF8");
	            
                SecretKeySpec skeySpec = new SecretKeySpec(raw, "AES"); 
     	        
                System.out.println("SecretKeySpec = " + skeySpec.toString());

		       // Instantiate the cipher 
	
		       Cipher cipher = Cipher.getInstance ("AES/CBC/PKCS5Padding", "BC");
	
		       
	
		       byte iv[] = new byte[16]; //cipher.getIV(); 
	
		             
	
		       IvParameterSpec dps = new IvParameterSpec(iv); 
	
		       cipher.init(Cipher.ENCRYPT_MODE , skeySpec, dps);
	
		       /*
		        * File Encryption. Use Platform independent File separator API.
		        */
		       File in = new File( "D:\\tools\\Project Research\\PCI\\Clear Text.txt");
		       File out = new File( "D:\\tools\\Project Research\\PCI\\Cipher Text.txt");
		       
		       encrypt( in,
	                    out,
	                    cipher );	 
	
	     }
}