Additional Blog

I have started writing about my experience with Ubuntu and my interest in programming languages like Erlang here.

Actually it is difficult to learn distributed architectures and tools like REST, Hadoop etc. and scalability if you don’t have enough exposure. I work for the offshore industry and true innovation is hard here because people are wary of revenue and they play a safe game.

This new blog could be about my experiences with many of these open-source initiatives.

Debug logs

We have a Windows SAN drive that is used to store files. The JCIFS library that is
used to access this drive sometimes timed out. We wanted to look at the packets before
increasing the socket timeout.Packet capturing software which
would have shown these access failures were barred from our production systems.
So we had a nagging debugging problem.

We set -Djcifs.util.loglevel=10 but this information is copious and was difficult to
analyze when printed in the DOS console.

We could not understand how to gather low-level debugging information in a log file.
I asked the forums and found that this code

PrintStream ps = new PrintStream( new File( "C:\\jcifslog" ));

jcifs.util.LogStream.setInstance( ps );

can actually redirect the log based on the “jcifs.util.loglevel” setting to a log file.
Now we have ample packet information to understand where the problem is.

We have realized that sufficient logs are indispensable even in production environments and
we make quite an effort to gather and print logs. Sometimes even though the logs are GigaBytes
in size we enable them as they are very helpful.

Teaching Computer Science

I came across Teach Computer Science without a Computer! in Hot Scalability Links for April 30, 2010from High Scalability by Todd Hoff.

Inspite of my experience with offshore work 🙂 , I still have a plan to teach my kid some basic ideas about computers.

JoinPoint Matching Covariant methods

AspectJ JoinPoint matches methods with Covariant return types. The ‘declare warning’ construct seems to be a useful way of testing because the eclipse markers show the match.

Eclipse View

Aspect

declare warning: execution(Collection<? extends Test> *.test( .. )) : "Covariant";


package com.test;

import java.util.Collection;

public class CovariantSub extends CovariantSuper{

	public Collection<Test> test(){
		return null;
	}
	
	@SuppressWarnings("unchecked")
	public Class test1(){
		return null;
	}

}

package com.test;

import java.util.Collection;

public class CovariantSuper {

	public Collection<? extends Test> test(){
		return null;
	}

	public Class<?> test1(){
		return null;
	}

}

SEMAT

I started reading the position papers of SEMAT which I came across in Ivar Jacobson’s blog. I am hoping that there will be good guidance from the signatories of this initiative which could make my life easier 🙂

Commodity software or expensive tool

We use a scheduler/file transfer tool from a smaller vendor to avoid costs and huge upfront investments. This financial project has very strict SLA’s. Would a more expensive solution help ?

There are many open-source tools that can do the same thing. We used WebSphere portal that works but I have also seen a open-source stack built using open-source WSRP etc. that can be used to build portals.

Actually there is one more fundamental problem. That is the company’s technology culture. If the culture is not conducive to technical software development or testing then good quality or scalability cannot be ensured.

Basically I think that delivering good SLA’s means clusters, performance testing, HA etc.

Not an expensive tool. Even a very expensive tool needs strong testing teams and a test environment that matches the production environment. So when the management does not understand fundamental testing principles we are going to be in trouble.

Recently we faced one of the many problems that are stopping us from delivering our software. The scheduler that we use is from Flux. Due to a browser upgrade issue the web console of this tool would not open in our production servers. Monitoring became harder.

Red tape ensured that this situation could not be overcome easily. Usually tools have facilities to manipulate the runtime through backend code.

WebSphere Portal has XMLAccess. The Flux engines forming a cluster can be accessed using
the following Java code. This code was able to recover a failed Flow chart and saved the day.

I feel that if the people who we are working with are not good, the best and the most expensive tools in the world cannot help us deliver good software.

So in the end it was working code that fixed the problem.

package com.test;

import java.rmi.NotBoundException;
import java.rmi.RemoteException;

import flux.Cluster;
import flux.Engine;
import flux.EngineException;
import flux.EngineInfo;
import flux.Factory;
import flux.FlowChart;
import flux.FlowChartIterator;

public class LookupEngine {

	public static void main( String... argv ) throws IllegalArgumentException, RemoteException, NotBoundException, EngineException{
		assert( null != argv[ 0 ] );
		assert( null != argv[ 1 ] );
		assert( null != argv[ 2 ] );
	    Factory factory = Factory.makeInstance();
	    LookupEngine le = new LookupEngine();
	    le.
	    lookupCluster( factory,
	    		       argv[ 0 ],
	    		       argv[ 1 ],
	    		       argv[ 2 ]);
	}
	
	private void lookupCluster( Factory factory,
			                    String host,
			                    String namespace,
			                    String bindName ) throws IllegalArgumentException,
			                       		       NotBoundException,
                                                               EngineException,
                                                               RemoteException{
		Cluster cluster =
		    factory.lookupCluster( host,
		                           1099,
		                           bindName,
		                           "user",
		                           "password" );
        Engine engines = null;
	for( Object engine : cluster.getEngines() ){
		System.out.println( engine );
                engines = ((EngineInfo)engine).createEngineReference();
	}
        FlowChartIterator iterator = null;
	for( iterator = engines.get() ; iterator.hasNext() ; ){
                          //Get FlowChart name
		System.out.println( (( FlowChart )iterator.next() ).getName());
        }
        
        
        iterator.close();
        //So once we find the name of the FlowChart we can recover it.
        //engines.recover( "/NameSpace/FlowChart" );
                
                 
	}
	
}

What would you rather be doing?

It takes a certain inconsiderateness to kill passion and the offshore industry is fully capable of it.
The lean methodology proponents advise us to develop people and then develop products.
I am still trying to read and practise agile principles, TDD etc. but it seems to be more and more difficult. Lack of experienced managers and adaptive processes seem to be killing every project I have come across.
It is disheartening that many people do not realize that there are some new modern software development practices.
While I write this I am watching the Oredev 2009 panel debate recorded by Scott Hanselman. What a contrast ? These recordings have become the only solace in my career.

PCI requires good security

We generally add security after building the product. It is a serious issue because security is not built into the product and we do not know cryptography well enough to be able to write good code.

This Java class was coded quickly just to encrypt and decrypt using AES 256 without understanding the foundations or key security principles. Even this took a long time and still there are holes in the way I have understood it but it is a good starting point.
It is also a good idea to subscribe to forums like dev-crypto@bouncycastle.org and read. I am planning to write Java code using the GPG API in the Bouncy Castle library.

This code uses the Bouncy Castle Provider but I do not think it is needed. I also used the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction libraries.

package com.encryption;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.security.Security;

import javax.crypto.Cipher;
import javax.crypto.CipherInputStream;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;

import org.bouncycastle.jce.provider.BouncyCastleProvider;

/**
 * A AES 256 encryption routine. 
 */

public class AESKeyEncryption {
	
	 

	     public static void encrypt( File in,
	    		                     File out,
	    		                     Cipher cipher ) throws Exception { 
			byte[] buf = new byte[1024];

			FileInputStream fin = new FileInputStream( in );
	        
			final CipherOutputStream cout = 
	        	new CipherOutputStream( new FileOutputStream( out ), cipher );
	        
			try {

	            int bytes = 0;
	        
	            /* Assumption is that there is no guarantee that
	             * there will be a full buffer every time. Isn't that
	             * what is called a subtle bug ;-)
	             */
	            while ((bytes = fin.read( buf )) >= 0) {
	            	cout.write( buf, 0 , bytes );
	            }
	            
	            cout.close();
	            fin.close();
	        } catch ( IOException e ) {
				
	        	System.out.println( "IOException");
	       }finally{
	    	   if( null != cout ){
	    		   cout.close();
	    	   }
	    	   if( null != fin ){
	    		   fin.close();
	    	   }
	    }
	     }
	     

	     public static void main(String[] args) throws Exception { 

                Security.addProvider(new BouncyCastleProvider());

                /*
                 * Supposed to be picked up from the database
                 * and salted if possible.  It is a 32 byte(256-bit) Key
                 */
                String key ="11112444123123222444200012311111";
	            
                byte[] raw = key.getBytes("UTF8");
	            
                SecretKeySpec skeySpec = new SecretKeySpec(raw, "AES"); 
     	        
                System.out.println("SecretKeySpec = " + skeySpec.toString());

		       // Instantiate the cipher 
	
		       Cipher cipher = Cipher.getInstance ("AES/CBC/PKCS5Padding", "BC");
	
		       
	
		       byte iv[] = new byte[16]; //cipher.getIV(); 
	
		             
	
		       IvParameterSpec dps = new IvParameterSpec(iv); 
	
		       cipher.init(Cipher.ENCRYPT_MODE , skeySpec, dps);
	
		       /*
		        * File Encryption. Use Platform independent File separator API.
		        */
		       File in = new File( "D:\\tools\\Project Research\\PCI\\Clear Text.txt");
		       File out = new File( "D:\\tools\\Project Research\\PCI\\Cipher Text.txt");
		       
		       encrypt( in,
	                    out,
	                    cipher );	 
	
	     }
}

Payment card industry

I have not posted anything for a long time because I moved to chennai and started working for the payment card industry. Now I am working on a merchant acquiring system. This type of industry has strict SLA’s and a high level of security governed by PCIDSS and many other specifications.

These are the main architectural constraints of this project

1. Job scheduling and lodging of settlement and other files according to various timezones.

2. A fully multi-lingual website.

3. Auditing of logs, security & code reviews. Automatic notification of potential security breaches.

4. 24×7 operation with local and Disaster recovery.

5. Straight through processing of merchant card transactions.

6. PKI infrastructure

and many other aspects.

The material available on the software technical approaches to handle all these aspects is either vast or scarce. Sometimes expensive software is required like SiteMinder SSO solution or Data lifecycle management or File archival for longer periods.

I am turning to what people like Michael Nygard write.

I hope to blog more about

1. memcached

2. Multi-lingual web sites

3. Time zone handling and job scheduling

4. File transfer using various protocols like SFTP, FTPS etc.

Budget constraints mean that some tools that seem to be very useful like Splunk for log management or a full Identity LifeCycle Management tool like SiteMinder are not within our reach. Hope we can find alternate Open-source solutions.

OpenSolaris on a virtual machine

Solaris 10Creating a virtual machine and running OpenSolaris 10 on Windows XP is really cool even though I have seen a VMWare installation in 1999. Two things that piqued my interest are Intel® Threading Building Block and Solaris zones which we used in our last project. I am trying to play with both using VMWare player and OpenSolaris and a virtual machine created using easyvmx.