Rising stars or shooting stars

I have lamented in the past about organization structures in India that promote people to dizzying heights in a short span of time. These people are the ‘rising stars’ and they never gain expertise in any field. They pick a plum technical project to work on so that the next appraisal process favors them not because they are experts but because they are part of a highly visible team that boosted the financial success of the firm.

This is from ‘The Five stages of Capacity Planning’ by H Pat Artis

“Because capacity planning in this stage is a high visibility special project, it is often assigned to rising stars who will have long since been promoted by the time of the next planning cycle occurs.”

These rising stars are shooting stars who do not continue as technical leads and a newer crop of junior people take their place leading to standards that fall precipitously.

Debilitating NTLM proxy

It is interesting to note how a corporate NTLM proxy can cause untold misery.

The saviour in this case is cntlm but I had to set it up in the following ways.

Environment

admin@LINUX:~/graphite/graphite-web-0.9.9/webapp/graphite$ env | grep "proxy"
http_proxy=http://127.0.0.1:3129
ftp_proxy=http://127.0.0.1:3129
https_proxy=http://127.0.0.1:3129

~/.bashrc

export http_proxy=http://127.0.0.1:3129
export https_proxy=http://127.0.0.1:3129
export ftp_proxy=http://127.0.0.1:3129

After all this I had to set it up again for a ruby gem installation.

sudo gem install fpm --http-proxy=http://127.0.0.1:3129

Sometimes I had to watch the verbose output from cntlm to be sure that I am not
being fooled by a wrong setup.

“member of the technical staff”

I was rather surprised to find these passages – copied below – in Fred Brook’s book “The mythical man-month” that was written a long time ago. I was browsing it again after a decade. So I thought I should collect my thoughts on this.

This particular organizational structure has caused me untold grief because as a technical lead I found the gap between these two groups too yawning to bridge. The two groups want to tear each other’s guts out at every meeting. Pure hatred !!

I have been specifically told by an MNC that a manager who has a large team will always get a better appraisal than a tech. arch. who builds systems with the assistance of a few. The top management is scared of equating the two groups for fear of antagonizing the project managers.

The general position “member of the technical staff” is a good way of merging the two groups. But the only way to completely abolish the difference is by making both groups work on project management as well as technical tasks, which is a tall order.

The other way is to encourage people to gain years of experience in their chosen field before moving up the ladder. This creates specialists. People in our companies don’t want to specialize in any field. So a business analyst is not a requirements expert. She should be. Actually no one is an expert requirements architect. So one of the most important fields is left without an expert. That hurts project success.

A project manager is not an expert in Earned Value Management or Statistical Process Control. Once we have specialists we will have humble people. Humility is a great leveler. Expertise gives confidence.


Plan the Organization for Change

Structuring an organization for change is much harder than designing a system for change. Each man must be assigned to jobs that broaden him, so that the whole force is technically flexible. On a large project the manager needs to keep two or three top programmers as a technical cavalry that can gallop to the rescue wherever the battle is thickest.

Management structures also need to be changed as the system changes. This means that the boss must give a great deal of attention to keeping his managers and his technical people as interchangeable as their talents allow.

The barriers are sociological, and they must be fought with constant vigilance. First, managers themselves often think of senior people as “too valuable” to use for actual programming. Next, management jobs carry higher prestige. To overcome this problem some laboratories, such as Bell Labs, abolish all job titles. Each professional employee is a “member of the technical staff.” Others, like IBM, maintain a dual ladder of advancement, as Fig. 11.1 shows. The corresponding rungs are in theory equivalent.

Figure 11.1. IBM dual ladder of advancement

It is easy to establish corresponding salary scales for rungs. It is much harder to give them corresponding prestige. Offices have to be of equal size and appointment. Secretarial and other support services must correspond. A reassignment from the technical ladder to a corresponding level on the managerial one should never be accompanied by a raise, and it should be announced always as a “reassignment,” never as a “promotion.” The reverse reassignment should always carry a raise; overcompensating for the cultural forces is necessary.

Managers need to be sent to technical refresher courses, senior technical people to management training. Project objectives, progress, and management problems must be shared with the whole body of senior people.

Whenever talents permit, senior people must be kept technically and emotionally ready to manage groups or to delight in building programs with their own hands. Doing this surely is a lot of work; but it surely is worth it!

The whole notion of organizing surgical-type programming teams is a radical attack on this problem. It has the effect of making a senior man feel that he does not demean himself when he builds programs, and it attempts to remove the social obstacles that deprive him of that creative joy.

Furthermore, that structure is designed to minimize the number of interfaces. As such, it makes the system maximally easy to change, and it becomes relatively easy to reassign a whole surgical team to a different programming task when organizational

Ignite Chennai 2012

This is the first edition of these focused, short and interesting presentations that I have attended. I missed the previous two. It was refreshing but mostly it was dominated by youngsters in the audience. That is reflective of the milieu prevailing in Chennai where Computer games, hollywood movies and Mc Donald’s are considered fit only for the young. It would be more fun if such gatherings have a mixed audience from all age groups.

The other style of presentation that resembles this is pechakucha.

K Mohan’s innovations

This SRM university student’s witty style and penchant for innovation counts as the most inspiring. As founder and CEO of a company he proves that it is quite possible to do what university students in many advanced countries like these two Stanford graduates do, without comparable support.

To be updated

Ethical Hacking and Information Security Workshop

Hosted by the IEEE Student Branch of MIT(Feb. 29 2012 and Mar. 1 2012

Virscent Technologies Pvt. Ltd. in association with IIT-Kharagpur E-Cell is proud to present D_Code – the Ethical Hacking Workshop. The course focuses on giving hands on training to all the participants in the field of Ethical Hacking.
The modules of the workshop are:

  1. Introduction to Ethical Hacking
  2. Cyber Forensics
  3. Web Based Email Attacks & Security
  4. Windows OS Hacks
  5. Understanding Malware Working & Detection
  6. Networking Attacks & Security
  7. Wi-Fi Attacks & Security
  8. Web Server Attacks & Security
  9. Hacking Using Google
  10. Software Reverse Engineering
  11. VOIP & Mobile Hacking

IEEE Software issue on ‘Algorithms for Today’s Practitioner’

I am reading the Jan/Feb 2012 issue of IEEE Software that should strike a chord with Software engineers in the offshore industry. Even financial application developers are oblivious to the need for algorithms that are not always part of the J2SE kit or an astronomically priced tool that the management here favors.

The awareness is lacking and the motivation to read about and experiment with code and algorithms is missing.

I also read the interesting interview with David Chaiken, Chief Architect of Yahoo who visited the Chennai IIT and spoke about his work in Yahoo. I was in the audience and remember that he pointed out a particular bug ID that caused some Yahoo servers in the production data centre to mishebave after he deployed a release.

My question to him was about the testing procedures that Yahoo uses for such large-scale deployments and I was looking for some tips about testing distributed applications. He just replied that Yahoo tries to use agile testing principles in some cases.

I have to rememeber to urge the local IEEE chapter to invite such speakers more frequently and also involve the developer community. There were faculty and students on that day amongst the sparse audience.

Apache ZooKeeper

I don’t know why management of firms that deal with financial data tend to be very conservative about new technology in my part of the world. There are varied reasons for this but the lack of experience of executives is high on the list. The vision to support large-scale deployments of mission critical software is just not there.

Company honchos should read Joel Spolsky’s opinion about the role of the executives in shaping the technology roadmap of the company.

I recently explored options to monitor and operate a simple cluster of Java socket programs and came across ZooKeeper. It is not strictly a requirement for a distributed application but these individual socket programs need to be managed and some may need to be restarted.

Initially I thought of using Google protobuffer and send simple information about the number of messages and load to the centralized ZooKeeper instance so that some decision can be taken based on it. This idea is not fully explored yet but the sample program that uses ZooKeeper nodes and proto buffer messages is shown below.

The messages sent and received from ZooKeeper nodes can be serialized in any way and here proto buffer is used for that.

Producer stores messages in the node


import com.google.protobuf.InvalidProtocolBufferException;
import message.Message;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;

import java.io.IOException;

public class DistributedProducer {

    private ZooKeeperCoordinator zkc;

    private String root = "/Protocol Buffer";

    {
        try {
            zkc =
            new ZooKeeperCoordinator( "localhost:2181",
                                      root );
            zkc.connect();
            zkc.setUpRoot();
        } catch (InterruptedException e) {

            System.out.println( "InterruptedException" );

        } catch (IOException e) {

            System.out.println( "IOException" );
        }

    }

    public void send() throws IOException, InterruptedException, KeeperException {

        if( null != zkc ){
            ZooKeeper zk = zkc.getZookeeper();
            Stat stat =
            zk.setData( root + "/protocolbuffer",
                       getData(),
                       -1
                      );
            System.out.println( "Stat is  [" + stat + "]");
        }
    }

    private byte[] getData() throws InvalidProtocolBufferException {
        Message.Load message = Message.Load.newBuilder().setType(
                                 ( Message.Load.LoadType.HIGH)).build();
        Message.Load message1 = Message.Load.parseFrom( message.toByteArray() );
        System.out.println( "Distributed value is [" + message1.getType().toString() +"]");
        return message.toByteArray();
    }
}

Consumer consumes it.


import com.google.protobuf.InvalidProtocolBufferException;
import message.Message;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.CharacterCodingException;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;

public class DistributedConsumer {

    private ZooKeeperCoordinator zkc;

    private String root = "/Protocol Buffer";

    {
        try {
            zkc =
            new ZooKeeperCoordinator( "localhost:2181",
                                      root );
            zkc.connect();
        } catch (InterruptedException e) {

            System.out.println( "InterruptedException" );

        } catch (IOException e) {

            System.out.println( "IOException" );
        }

    }

    public void receive() throws IOException, InterruptedException, KeeperException {

        Stat stat = null;

            if( null != zkc ){
                ZooKeeper zk = zkc.getZookeeper();
                if( null != zk ){
                    byte[] value =
                    zk.getData( root + "/protocolbuffer",
                               false,
                               stat
                              );
                    getData( value );
                }
            }
    }

    private void getData( byte[] value ){

        try {
            Message.Load message = Message.Load.parseFrom( value );
            System.out.println( "Distributed value is  [" + message.getType().toString() + "]");

        } catch (InvalidProtocolBufferException e) {

            System.out.println( "InvalidProtocolBufferException" + e.getMessage() );
            e.printStackTrace();

        }
    }
}

A client that connects to and sets up ZooKeeper


import org.apache.zookeeper.*;
import org.apache.zookeeper.data.Stat;

import java.io.IOException;
import java.util.concurrent.CountDownLatch;

public class ZooKeeperCoordinator {
    
    private String host;
    
    private String root;

    private Stat stat;

    public ZooKeeper getZookeeper() {
        return zookeeper;
    }

    private ZooKeeper zookeeper;

    public ZooKeeperCoordinator( String host,
                                 String root ){

        this.host = host;
        this.root = root;

    }

    public void setUpRoot(){

        System.out.println( "Set up root" );
        try {
            stat = zookeeper.exists( root,
                                     false );
            if( null == stat ){
                 zookeeper.create( root,
                                   new byte[ 0 ],
                                   ZooDefs.Ids.OPEN_ACL_UNSAFE,
                                   CreateMode.PERSISTENT);
            }
        } catch (KeeperException e) {
            System.out.println( "KeeperException" );

        } catch (InterruptedException e) {
            System.out.println( "InterruptedException" );
        }
    }

    public void connect() throws InterruptedException, IOException {

        System.out.println( "connect" );
        final CountDownLatch latch = new CountDownLatch( 1 );

        Watcher watcher = new Watcher(){

            public void process(WatchedEvent watchedEvent) {

                if( watchedEvent.getState() == Event.KeeperState.SyncConnected ){
                    latch.countDown();
                }
            }
        };

        zookeeper = new ZooKeeper( host, 1000, watcher );
        latch.await();
        System.out.println( "Awaiting connection" );
    }

    public void close() throws InterruptedException {
        zookeeper.close();
    }
}

The Protocol Buffers documentation has instructions to compile this simple .proto data structure

package message;

message Load {

  enum LoadType {
    HIGH = 0;
    MEDIUM = 1;
    LOW = 2;
  }


  optional LoadType type = 1 [default = MEDIUM];

}

Mac OS Lion freezes too

I realized that the Mac OS freezes too and not for a serious reason. I just tried to install firefox and Finder caused my Laptop to become unusable.

So following the advice in this site I cleaned its preferences but there was no respite.

As a last resort I booted in single user mode and issued rm /var/db/.applesetupdone. I created a new admin. user and deleted my old administrators.

That wasted my Sunday. This has never happened to my Ubuntu desktops.

Sebastian Thrun’s democratic university

I am stumped. Stanford research professor Sebastian Thrun and other professors are planning to organize and teach an entire university course in Computer Science !! I planned to but didn’t take their initial AI course but this is too good to miss. They are following it up with an entire degree course in CS. What attracts me is the open courseware concept and the self-driving car.

As a software developer I have firsthand knowledge of the lack of taste for algorithms/CS and how the entire offshore industry is stuck in a medieval mindset.
I am much obliged.

However you look at this endeavor this proves that if one is set on thinking audaciously action will follow. Shouldn’t this stir up our interest in CS and its application to seemingly mundane software engineering?

Description: This class, taught by one of the foremost experts in AI, will teach you basic methods in Artificial Intelligence, including: probabilistic inference, computer vision, machine learning, and planning, all with a focus on robotics. Extensive programming examples and assignments will apply these methods in the context of building self-driving cars. You will get a chance to visit, via video, the leading research labs in the field, and meet the scientists and engineers who are building self-driving cars at Stanford and Google.
Prerequisites: The instructor will assume solid knowledge of programming, all programming will be in Python. Knowledge of probability and linear algebra will be helpful.

Java Path Finder

Java Path Finder(JPF) is a model checker for Java programs and can check concurrent programs for deadlocks and data races which are more serious in muti-core environments. Java threads used by concurrent programs are context switched by the OS scheduler at run-time when we ‘yield’ a thread or due to some other reason. Even though it might be possible to test these scenarios extensively by profiling a few possible combinations of thread interleavings, it is quite tedious. So JPF tests this type of code by exploring all the code using algorithms to find errors.

JPF has quite a number of plug points and gov.nasa.jpf.jvm.VMListener is one such interface. I have implemented it to get data about the number of BLOCKED threads that the System Under Test(SUT) has and the monitor that they are blocked at. The data is still not clear but it is reasonable. I am not able to get clear data showing how many threads are blocked for a particular monitor to be released at a point in time. I think the code has to be refined.

JPF can be downloaded from http://babelfish.arc.nasa.gov/trac/jpf/ by using a mercurial eclipse plugin and it is quite easy to build and use.

I implemented gov.nasa.jpf.jvm.VMListener and added the classpath entry pointing to the class file to jpf-core.native_classpath

gov.nasa.jpf.jvm.VMListener based on JPF source code

public class LockContentionListener implements VMListener{


	@Override
	public void threadStarted(JVM vm) {
		printLocks( vm );
	}


	/**
	 * Thread and lock status.
	 * @param vm
	 */

	    private void printLocks( JVM jvm ){

		StringBuffer locksAndThreads = new StringBuffer();

		for ( ThreadInfo tf : jvm.getThreadList().getThreads()) {

		    ElementInfo ei = tf.getLockObject();
		    if (ei != null) {
		      if (tf.getState() == ThreadInfo.State.WAITING ||
				  tf.getState() == ThreadInfo.State.BLOCKED ) {
			  locksAndThreads.append( "\n");
			  locksAndThreads.append(tf.getStateDescription());
			  locksAndThreads.append(ei);
			  locksAndThreads.append(" \n Lock Count (" +                 ei.getMonitor().getNumberOfBlockedThreads()+ ")");

			    LinkedList locks = tf.getLockedObjects();
			    if (locks.isEmpty()) {
				locksAndThreads.append("  call stack:");
				    for (StackFrame frame : tf){
				      if (!frame.isDirectCallFrame()) {
					  locksAndThreads.append("\tat \n ");
					  locksAndThreads.append(frame.getStackTraceInfo());
				      }
				    }
			    }
				System.out.println( "Locks owned by Threads[ " + locksAndThreads + " \n ]");

		  }
		 }
		}
	    }

SUT

public class LockContention{
	
	private static final int NUM_THREADS = 2;
	
	
	public  static void main( String... argv ) throws InterruptedException {
		
		Thread[] readThreads = new Thread[ NUM_THREADS ];

		Locker locker = new Locker();
		
		for(  int i = 0 ; i < readThreads.length; i ++ ){
			
			readThreads[ i ] = new Thread( new ReadLockContention( locker ) ){
				
				public String getName(){
					return "LockContention (Reader)[ " + getId() + " ]";
				}
				
				
			};
		}

		Thread[] writeThreads = new Thread[ NUM_THREADS ];

		for( int i = 0 ; i < writeThreads.length; i ++ ){
			writeThreads[ i ] = new Thread( new WriteLockContention( locker ) ){

				public String getName(){
					return "LockContention (Writer)[" + getId() + " ]";
				}
				
			};
		}
		
		for( Thread t : readThreads ){
			t.start();
		}
		for( Thread t : writeThreads ){
			t.start();
		}
		for( Thread t : readThreads ){
			t.join();
		}

		for( Thread t : writeThreads ){
			t.join();
		}
		
	}


}

class ReadLockContention implements Runnable{
	
	private Locker locker;
	
	public ReadLockContention( Locker locker ){
		this.locker = locker;
	}

	@Override
	public void run() {
		locker.contendForReadLock();
		
	}

}

class WriteLockContention implements Runnable{
	
	private Locker locker;
	
	public WriteLockContention( Locker locker ){
		this.locker = locker;
	}

	@Override
	public void run() {
		locker.contendForWriteLock();
		
	}

}

class Locker{
	
	private final String lock = new String();

	/**
	 * Consider this as a read lock
	 */
	void contendForReadLock(){
			synchronized( lock ){
				try {
					Thread.sleep(50000);
				} catch (InterruptedException e) {
					System.out.println( "InterruptedException" );
			    }
				
			}
	}
	
	/**
	 * Consider this as a write lock
	 */
	void contendForWriteLock(){
			synchronized( lock ){
				
			}
	}
	
}

Output

The command

jpf +listener+=LockContentionListener LockContention.jpf

produces

Threads blocked on Locks[
thread id=2,name=Thread-2,status=BLOCKED,priority=5,lockCount=0,suspendCount=0java.lang.String@137
Lock Count (1) call stack: at
Locker.contendForReadLock(LockContention.java:97) at
ReadLockContention.run(LockContention.java:67)
]
Threads blocked on Locks[
thread id=1,name=Thread-1,status=BLOCKED,priority=5,lockCount=0,suspendCount=0java.lang.String@137
Lock Count (1) call stack: at
Locker.contendForReadLock(LockContention.java:97) at
ReadLockContention.run(LockContention.java:67)
]

The monitor is shown in bold. I want to know how many threads are blocked for a lock to be released.
It seems that these messages indicate that two threads(Thread-1,Thread-2) are blocked on the same monitor but I think this has to be investigated further. There is no reply to my forum question about this.