Distributed Training using TensorFlow Federated

This is a very simple example of using multiple GPUs using a Jupyter Notebook to train a model. Obviously this involves multiple machines or VMs or in this case multiple processes in a simple Compute instance. Multiple processes in a single VM make it easier to test.

GPUs seem to be a costly affair.

I connected using SSH and these are the details of the VM

======================================
Welcome to the Google Deep Learning VM
======================================

Version: tf2-gpu.2-8.m92
Based on: Debian GNU/Linux 10 (buster) (GNU/Linux 4.19.0-20-cloud-amd64 x86_64\n)

Resources:
 * Google Deep Learning Platform StackOverflow: https://stackoverflow.com/questions/tagged/google-dl-platform
 * Google Cloud Documentation: https://cloud.google.com/deep-learning-vm
 * Google Group: https://groups.google.com/forum/#!forum/google-dl-platform

To reinstall Nvidia driver (if needed) run:
sudo /opt/deeplearning/install-driver.sh
TensorFlow comes pre-installed with this image. To install TensorFlow binaries in a virtualenv (or conda env),
please use the binaries that are pre-built for this image. You can find the binaries at
/opt/deeplearning/binaries/tensorflow/
If you need to install a different version of Tensorflow manually, use the common Deep Learning image with the
right version of CUDA

Linux distributedtraining 4.19.0-20-cloud-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.


(base) root@distributedtraining:~# nvidia-smi
Wed May 25 04:59:48 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   64C    P0    29W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:05.0 Off |                    0 |
| N/A   59C    P0    30W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Build Jupyter

Since my automatic build failed I built myself. Not sure why it should fail.

radhakrishnan_mohan@distributedtraining:~$ sudo -i
(base) root@distributedtraining:~# jupyter lab build --dev-build=False --minimize=False
[LabBuildApp] JupyterLab 3.2.9
[LabBuildApp] Building in /opt/conda/share/jupyter/lab
[LabBuildApp] Building jupyterlab assets (production, not minimized)
(base) root@distributedtraining:~# jupyter labextension list
JupyterLab v3.2.9
/opt/conda/share/jupyter/labextensions
        nbdime-jupyterlab v2.1.1 enabled OK
        jupyterlab-jupytext v1.3.8+dev enabled OK (python, jupytext)
        jupyterlab_pygments v0.2.2 enabled OK (python, jupyterlab_pygments)
        @jupyterlab/server-proxy v3.2.1 enabled OK
        @jupyterlab/git v0.37.1 enabled OK (python, jupyterlab-git)
        @jupyter-widgets/jupyterlab-manager v3.1.0 enabled OK (python, jupyterlab_widgets)

Other labextensions (built into JupyterLab)
   app dir: /opt/conda/share/jupyter/lab
        beatrix_jupyterlab v3.1.7 disabled OK
        jupyterlab-plotly v5.8.0 enabled OK
        plotlywidget v4.14.3 enabled OK
        tensorflow_model_analysis v0.34.1 enabled OK
        wit-widget v1.8.1 enabled OK
        xai_tabular_widget v0.1.0 enabled OK

Assign GPU device to worker

It seems that it is imperative to assign a particular GPU to a worker as we have 2 Tesla P4 GPUs and 2 workers. If we don’t then there is failure to allocate GPU memory adequately. This line of code does that.

os.environ['CUDA_VISIBLE_DEVICES']=str(index)

Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

Java 8 Optional

I think there are more elegant ways to check if Optional is empty or not but here I have to collect everything in a ArrayList. So I wasn’t able to include isPresent() in the lambda pipeline.

package com.test;

import java.util.*;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class OptionalTest {

    private static List<String> newImports = new ArrayList<>();

    public static Optional<List<String>> getOptionalNewImports() {

        return Optional.ofNullable(newImports);
    }

    public static void main( String... argv ){

        newImports.add("org.apache.struts");

        if( getOptionalNewImports().isPresent() ) {
            List<String> imports = new ArrayList<>();
                    getOptionalNewImports().get().stream()
                    .map(p -> "import " + p + ";")
                    .collect(Collectors.toCollection(() -> imports));
            imports.forEach( System.out::println);
        }

    }
}

This is the relevant method.

        public Optional<List<String>> getOptionalNewImports() {

            return Optional.ofNullable(newImports);
        }

This is a proper usage of ifPresent. I assign a value to a variable if the value is present.

                        rules.getOptionalClassIdentifier().ifPresent( a -> {this.classIdentifier = a;});


JPA and Spring @Transactional and JBoss Arquillian

JBoss Arquillian is a test framework that one can use to execute tests in the IDE as part of the development process. The key parts are the deployment API and container adapters that enable us to deploy, tests that execute inside a container, automatically and repeatedly.

I have written about Arquillian here.
In this post I will show how a simple Arquillian test for a JPA transaction avoids countless wasted hours. Actually I spent a few hours trying to find out why enabling the wrong Transaction Manager produces log lines almost similar to the section below and misleads one into thinking that transactions are indeed in effect. It is the wrong transaction manager and no rows are actually committed to the Database. But the logs do show some messages that indicate data is committed.

This is the correct set of log messages that show that JpaTransactionManager takes effect.

DEBUG: org.springframework.transaction.annotation.AnnotationTransactionAttribute
Source – Adding transactional method ‘TestImpl.test’ with attribute: PROPAGATION
_REQUIRED,ISOLATION_DEFAULT; ”
DEBUG: org.springframework.orm.jpa.JpaTransactionManager – Creating new transact
ion with name [com.jpa.test.TestImpl.test]: PROPAGATION_REQUIRED,ISOLATION_DEFAU
LT; ”
DEBUG: org.hibernate.internal.SessionImpl – Opened session at timestamp: 1442144
9169
TRACE: org.hibernate.internal.SessionImpl – Setting flush mode to: AUTO
TRACE: org.hibernate.internal.SessionImpl – Setting cache mode to: NORMAL
DEBUG: org.springframework.orm.jpa.JpaTransactionManager – Opened new EntityMana
ger [org.hibernate.ejb.EntityManagerImpl@8f64d] for JPA transaction
DEBUG: org.hibernate.engine.transaction.spi.AbstractTransactionImpl – begin
DEBUG: org.hibernate.engine.jdbc.internal.LogicalConnectionImpl – Obtaining JDBC
connection
DEBUG: org.springframework.jdbc.datasource.SimpleDriverDataSource – Creating new
JDBC Driver Connection to [jdbc:hsqldb:mem:dataSource]
DEBUG: org.hibernate.engine.jdbc.internal.LogicalConnectionImpl – Obtained JDBC
connection
DEBUG: org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction – initial
autocommit status: true
DEBUG: org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction – disablin
g autocommit
DEBUG: org.springframework.orm.jpa.JpaTransactionManager – Exposing JPA transact
ion as JDBC transaction [org.springframework.orm.jpa.vendor.HibernateJpaDialect$
HibernateConnectionHandle@423d24]
TRACE: org.springframework.transaction.support.TransactionSynchronizationManager
– Bound value [org.springframework.jdbc.datasource.ConnectionHolder@805780] for
key [org.springframework.jdbc.datasource.SimpleDriverDataSource@faa27c] to thre
ad [http-nio-8080-exec-5]
TRACE: org.springframework.transaction.support.TransactionSynchronizationManager
– Bound value [org.springframework.orm.jpa.EntityManagerHolder@7a72fc] for key
[org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean@1e0cc0a] to
thread [http-nio-8080-exec-5]
TRACE: org.springframework.transaction.support.TransactionSynchronizationManager
– Initializing transaction synchronization
TRACE: org.springframework.transaction.interceptor.TransactionInterceptor – Gett
ing transaction for [com.jpa.test.TestImpl.test]
INFO : jpa – TransactionSynchronizationManager.isActualTransactionActive()true
INFO : jpa – INMEMORY_DB [id=id, street=Street, area=Area, state=State, country
=LO, pin=1]
TRACE: org.springframework.transaction.support.TransactionSynchronizationManager
– Retrieved value [org.springframework.orm.jpa.EntityManagerHolder@7a72fc] for
key [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean@1e0cc0a]
bound to thread [http-nio-8080-exec-5]
TRACE: org.hibernate.engine.spi.IdentifierValue – ID unsaved-value strategy UNDE
FINED
TRACE: org.hibernate.event.internal.AbstractSaveEventListener – Transient instan
ce of: com.jpa.test.INMEMORY_DB
TRACE: org.hibernate.event.internal.DefaultPersistEventListener – Saving transie
nt instance
DEBUG: org.hibernate.event.internal.AbstractSaveEventListener – Generated identi
fier: id, using strategy: org.hibernate.id.Assigned
TRACE: org.hibernate.event.internal.AbstractSaveEventListener – Saving [com.jpa.
test.INMEMORY_DB#id]
TRACE: org.hibernate.engine.spi.ActionQueue – Adding an EntityInsertAction for [
com.jpa.test.INMEMORY_DB] object
TRACE: org.hibernate.engine.spi.ActionQueue – Adding insert with no non-nullable
, transient entities: [EntityInsertAction[com.jpa.test.INMEMORY_DB#id]]
TRACE: org.hibernate.engine.spi.ActionQueue – Adding resolved non-early insert a
ction.
TRACE: org.hibernate.action.internal.UnresolvedEntityInsertActions – No unresolv
ed entity inserts that depended on [[com.jpa.test.INMEMORY_DB#id]]
TRACE: org.hibernate.action.internal.UnresolvedEntityInsertActions – No entity i
nsert actions have non-nullable, transient entity dependencies.
TRACE: org.springframework.transaction.interceptor.TransactionInterceptor – Comp
leting transaction for [com.jpa.test.TestImpl.test]
DEBUG: org.springframework.orm.jpa.JpaTransactionManager – Initiating transactio
n commit
DEBUG: org.springframework.orm.jpa.JpaTransactionManager – Committing JPA transa
ction on EntityManager [org.hibernate.ejb.EntityManagerImpl@8f64d]
DEBUG: org.hibernate.engine.transaction.spi.AbstractTransactionImpl – committing

The source code has a copy of log4j.xml that enables the appropriate log. But this method is not repeatable in the sense that it is hard to manually check the log messages everytime we change the configuration or add new code. That is what unit tests are for and Arquillian container tests deploy our code into a container and execute tests in the IDE. The developer does not have to deploy manually and test the code. All that is required is a good regression test suite.

Arquillian uses the dependency arquillian-transaction-spring to make the test method transactional.

There are some dependencies in the pom.xml as well as in this Arquillian test that are not needed or redundant but the required ones are there.

package com.jpa.test;

import static org.junit.Assert.assertEquals;

import java.io.File;
import java.util.List;
import java.util.logging.Logger;

import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import javax.persistence.TypedQuery;
import javax.transaction.SystemException;

import org.jboss.arquillian.container.test.api.Deployment;
import org.jboss.arquillian.junit.Arquillian;
import org.jboss.arquillian.spring.integration.test.annotation.SpringConfiguration;
import org.jboss.arquillian.transaction.api.annotation.Transactional;
import org.jboss.shrinkwrap.api.ShrinkWrap;
import org.jboss.shrinkwrap.api.formatter.Formatters;
import org.jboss.shrinkwrap.api.spec.JavaArchive;
import org.jboss.shrinkwrap.api.spec.WebArchive;
import org.jboss.shrinkwrap.resolver.api.maven.Maven;
import org.junit.Test;
import org.junit.runner.RunWith;


@RunWith(Arquillian.class)
@SpringConfiguration("applicationContext.xml")
public class ShrinkWrappedJPATest {

	private static Logger l = Logger.getLogger("jpa");
		
	@PersistenceContext(unitName="testingSetup")
	private EntityManager entityManager;

    @Deployment
    public static WebArchive createWebArchive() {
  
    	final WebArchive war=ShrinkWrap.create(WebArchive.class,"ShrinkWrapJPA.war");
    	  
        JavaArchive jar = ShrinkWrap.create(JavaArchive.class)
                				.addPackage("com.jpa.test");
 

        war.addAsLibrary(jar);
    	war.addAsResource("applicationContext.xml");
    	war.addAsResource("arquillian.xml");
    	war.addAsResource("log4j.xml");
    	war.addAsResource("schema.sql");
    	war.addAsResource("test-data.sql");
    	war.addAsResource("log4j.xml");
    	war.addAsResource("persistence.xml", "META-INF/persistence.xml");
    	loadDependencies( war );
 
    	l.info(war.toString(Formatters.VERBOSE));
    	return war;
    }

        
    
    
    private static void loadDependencies( final WebArchive war ){
    	
        File springorm = Maven.
				resolver().
					resolve("org.springframework:spring-orm:4.1.6.RELEASE")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(springorm);

        File hibernate = Maven.
 				resolver().
 					resolve("org.hibernate:hibernate-core:4.1.7.FINAL")
 						.withoutTransitivity().asSingle(File.class);

         war.addAsLibraries(hibernate);

 
	    File hibernate1 = Maven.
					resolver().
						resolve("org.hibernate:hibernate-entitymanager:4.1.7.FINAL")
							.withoutTransitivity().asSingle(File.class);
	
	     war.addAsLibraries(hibernate1);

        File springexpression = Maven.
				resolver().
					resolve("org.springframework:spring-expression:4.2.0.RELEASE")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(springexpression);

        File springweb = Maven.
				resolver().
					resolve("org.springframework:spring-web:4.2.0.RELEASE")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(springweb);

        File springcore = Maven.
				resolver().
					resolve("org.springframework:spring-core:4.1.6.RELEASE")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(springcore);
        
        File springcontext = Maven.
				resolver().
					resolve("org.springframework:spring-context:4.1.6.RELEASE")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(springcontext);

        File springjdbc = Maven.
				resolver().
					resolve("org.springframework:spring-jdbc:4.1.6.RELEASE")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(springjdbc);
        File springtx = Maven.
				resolver().
					resolve("org.springframework:spring-tx:4.1.6.RELEASE")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(springtx);
        File hsqldb = Maven.
 				resolver().
 					resolve("org.hsqldb:hsqldb:2.3.1")
 						.withoutTransitivity().asSingle(File.class);

         war.addAsLibraries(hsqldb);
         File dbcp = Maven.
 				resolver().
 					resolve("commons-dbcp:commons-dbcp:1.4")
 						.withoutTransitivity().asSingle(File.class);

         war.addAsLibraries(dbcp);
       File aopalliance = Maven.
				resolver().
					resolve("aopalliance:aopalliance:1.0")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(aopalliance);
        File extensionspring = Maven.
				resolver().
					resolve("org.jboss.arquillian.extension:arquillian-service-deployer-spring-3:1.0.0.Beta3")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(extensionspring);


         File springbeans = Maven.
				resolver().
					resolve("org.springframework:spring-beans:4.1.6.RELEASE")
						.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(springbeans);

        File springaop = Maven.
				resolver().
					resolve("org.springframework:spring-aop:4.2.0.RELEASE")
					.withoutTransitivity().asSingle(File.class);

        war.addAsLibraries(springaop);


         File transactionapi = Maven.
 				resolver().
 					resolve("org.jboss.arquillian.extension:arquillian-transaction-api:1.0.1.Final")
 					.withoutTransitivity().asSingle(File.class);

         war.addAsLibraries(transactionapi);
         File transactionimplbase = Maven.
 				resolver().
 					resolve("org.jboss.arquillian.extension:arquillian-transaction-impl-base:1.0.1.Final")
 					.withoutTransitivity().asSingle(File.class);

         war.addAsLibraries(transactionimplbase);

    }
    
    @Test
    @Transactional(manager="transactionManager")
    
	public void save() throws Exception, SystemException {
 

        INMEMORY_DB a = new INMEMORY_DB();
		a.setId("id");
		a.setStreet("Street");
		a.setArea("Area");
		a.setState("State");
		a.setCountry("LO");
		a.setPin(1);
		entityManager.persist( a );
  		assertEquals(getAddressCount(), 2);
	}

 
	public int getAddressCount(){
		TypedQuery<INMEMORY_DB> query =
				entityManager.createQuery("SELECT c FROM INMEMORY_DB c", INMEMORY_DB.class);
		List<INMEMORY_DB> results = query.getResultList();	
		return results.size();
	}

}

Arquillian Unit Tests

arquillian_logoI have contributed an article to DZone about Arquillian using this source code.

Parsing Java Micro-benchmarking Harness data using dplyr – Part 2

I have to add explanations later because I have to determine if the statistical measures calculated are correct or wrong. But this is based on the previous blog post.

Update : I think the measures are correctly plotted.

Types of Error bars used to plot the diagram

Error bars Type Description
Standard error (SEM) Inferential A measure of how variable the mean will be, if you repeat the whole study many times.
Confidence interval (CI), usually 95% CI Inferential A range of values you can be 95% confident contains the true mean.

The parsing will not work if JMH changes the default format of the output file.

library(stringr)
library(dplyr)
library(ggplot2)

data <- read.table("D:\\jmh\\jmh.txt",sep="\t")

final <-data %>%
	    select(V1) %>%	
		filter(grepl("^Iteration", V1)) %>%  
        mutate(V1 = str_extract(V1, "\\d+\\.\\d*"))

final <- mutate(final,IDX = 1:n())

jc <- final %>%
		filter(IDX < 21)


gc <- final %>%
		filter(IDX > 20)

gc <- mutate(gc,IDX = 1:n())

jc <- data.frame(sapply(jc, function(x) as.numeric(as.character(x))))
gc <- data.frame(sapply(gc, function(x) as.numeric(as.character(x))))


print(summary(jc$V1))
error <- qt(0.995,df=length(jc$V1)-1)*sd(jc$V1)/sqrt(length(jc$V1))
error1 <- mean(jc$V1)-error
error2 <- mean(jc$V1)+error

q <- qplot(geom = "line",jc$IDX,jc$V1, colour='red')+geom_errorbar(aes(x=jc$IDX, ymin=jc$V1-sd(jc$V1), ymax=jc$V1+sd(jc$V1)), width=0.25)+ 
		geom_ribbon(aes(x=jc$IDX, y=jc$V1, ymin=error1, ymax=error2),fill="ivory2",alpha = 0.4)+ 
		xlab('Iterations') + ylab("Java Collections")+theme_bw() 

ggsave("D:\\jmh\\jc.png", width=6, height=6, dpi=100)

#Using error <- qt(0.995,df=length(jc$V1)-1)*sd(jc$V1)/sqrt(length(jc$V1)) 
g <- ggplot(jc, aes(x = IDX, y = V1)) +
		theme_bw() +
		geom_ribbon(aes(ymin = V1 - error, ymax = V1 + error), fill = "gray60",
				alpha = 0.3) +
		geom_line(color = "blue", size = 1) +
		geom_errorbar(aes(ymin = V1 - error, ymax = V1 + error), width = 0.25,
				color = "red") +
		labs(x = "Iterations", y = "Java collections")

ggsave("D:\\jmh\\ggplotjc.png", width=6, height=6, dpi=100)


print(summary(gc$V1))
error <- qt(0.995,df=length(gc$V1)-1)*sd(gc$V1)/sqrt(length(gc$V1))
error1 <- mean(gc$V1)-error
error2 <- mean(gc$V1)+error

q1 <- qplot(geom = "line",gc$IDX,gc$V1, colour='red')+geom_errorbar(aes(x=gc$IDX, ymin=gc$V1-sd(gc$V1), ymax=gc$V1+sd(gc$V1)), width=0.25)+ 
		geom_ribbon(aes(x=gc$IDX, y=gc$V1, ymin=error1, ymax=error2),fill="ivory2",alpha = 0.4)+ 
		xlab('Iterations') + ylab("Goldmansachs Collections")+theme_bw() 


ggsave("D:\\jmh\\gc.png", width=6, height=6, dpi=100)

#Using error <- qt(0.995,df=length(gc$V1)-1)*sd(gc$V1)/sqrt(length(gc$V1)) 
g1 <- ggplot(gc, aes(x = IDX, y = V1)) +
		theme_bw() +
		geom_ribbon(aes(ymin = V1 - error, ymax = V1 + error), fill = "gray60",
				alpha = 0.3) +
		geom_line(color = "blue", size = 1) +
		geom_errorbar(aes(ymin = V1 - error, ymax = V1 + error), width = 0.25,
				color = "red") +
		labs(x = "Iterations", y = "Goldmansachs collections")

ggsave("D:\\jmh\\ggplotgc.png", width=6, height=6, dpi=100)

jc

gc

Suggested by the R user forum to improve the aesthetics of the plot. The Confidence Interval of 99% shown in the plots above is not correct. But the curves and error bars are correct.

g1 <- ggplot(gc, aes(x = IDX, y = V1)) +
		theme_bw() +
		geom_ribbon(aes(ymin = V1 - error, ymax = V1 + error), fill = "gray60",
				alpha = 0.3) +
		geom_line(color = "blue", size = 1) +
		geom_errorbar(aes(ymin = V1 - error, ymax = V1 + error), width = 0.25,
				color = "red") +
		labs(x = "Iterations", y = "Goldmansachs collections")

ggplot creates these two graphs. So instead of qplot code we should use ggplot.

ggplotjc

ggplotgc

Update : See this

Parsing Java Micro-benchmarking Harness data using dplyr – Part 1

This is about the venerable JMH and Hadley Wickham’s dplyr and pipes package. dplyr enables you to have too much fun with data. Its pipes are so powerful and makes short shrift of even messy data.

# VM invoker: D:\Java\bin\java.exe
# VM options: -XX:-TieredCompilation -Dbenchmark.n=10000
# Warmup: 5 iterations, 50 ms each
# Measurement: 20 iterations, 50 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: oracle.stream.javaone.CollectionComparison.goldmansachscollections

# Run progress: 0.00% complete, ETA 00:00:02
# Fork: 1 of 1
# Warmup Iteration 1: 0.443 us/op
# Warmup Iteration 2: 0.290 us/op
# Warmup Iteration 3: 0.343 us/op
# Warmup Iteration 4: 0.350 us/op
# Warmup Iteration 5: 0.388 us/op
Iteration 1: 0.796 us/op
Iteration 2: 0.542 us/op
Iteration 3: 0.510 us/op
Iteration 4: 0.617 us/op
Iteration 5: 0.482 us/op
Iteration 6: 0.387 us/op
Iteration 7: 0.272 us/op
Iteration 8: 0.536 us/op
Iteration 9: 0.498 us/op
Iteration 10: 0.402 us/op
Iteration 11: 0.328 us/op
Iteration 12: 0.542 us/op
Iteration 13: 0.299 us/op
Iteration 14: 0.647 us/op
Iteration 15: 0.291 us/op
Iteration 16: 0.815 us/op
Iteration 17: 0.680 us/op
Iteration 18: 0.363 us/op
Iteration 19: 0.560 us/op
Iteration 20: 0.334 us/op

Result: 0.495 ¦(99.9%) 0.140 us/op [Average]
Statistics: (min, avg, max) = (0.272, 0.495, 0.815), stdev = 0.162
Confidence interval (99.9%): [0.355, 0.636]

# VM invoker: D:\Java\bin\java.exe
# VM options: -XX:-TieredCompilation -Dbenchmark.n=10000
# Warmup: 5 iterations, 50 ms each
# Measurement: 20 iterations, 50 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: oracle.stream.javaone.CollectionComparison.javacollections

# Run progress: 50.00% complete, ETA 00:00:05
# Fork: 1 of 1
# Warmup Iteration 1: 0.475 us/op
# Warmup Iteration 2: 0.696 us/op
# Warmup Iteration 3: 0.816 us/op
# Warmup Iteration 4: 0.622 us/op
# Warmup Iteration 5: 0.574 us/op
Iteration 1: 0.987 us/op
Iteration 2: 0.585 us/op
Iteration 3: 0.770 us/op
Iteration 4: 0.711 us/op
Iteration 5: 0.546 us/op
Iteration 6: 0.553 us/op
Iteration 7: 1.164 us/op
Iteration 8: 1.096 us/op
Iteration 9: 1.477 us/op
Iteration 10: 0.824 us/op
Iteration 11: 1.002 us/op
Iteration 12: 0.504 us/op
Iteration 13: 1.019 us/op
Iteration 14: 0.834 us/op
Iteration 15: 0.589 us/op
Iteration 16: 0.557 us/op
Iteration 17: 1.338 us/op
Iteration 18: 0.906 us/op
Iteration 19: 0.486 us/op
Iteration 20: 0.587 us/op

Result: 0.827 ¦(99.9%) 0.252 us/op [Average]
Statistics: (min, avg, max) = (0.486, 0.827, 1.477), stdev = 0.291
Confidence interval (99.9%): [0.574, 1.079]

# Run complete. Total time: 00:00:10

Benchmark Mode Samples Score Scor
e error Units
o.s.j.CollectionComparison.goldmansachscollections avgt 20 0.495
0.140 us/op
o.s.j.CollectionComparison.javacollections avgt 20 0.827
0.252 us/op

library(stringr)
library(dplyr)

data <- read.table("D:\\jmh\\jmh.txt",sep="\t")

final <-data %>%
	    select(V1) %>%	
		filter(grepl("^Iteration", V1)) %>%  
        mutate(V1 = str_extract(V1, "\\d+\\.\\d*"))

print(final)

V1
1 0.796
2 0.542
3 0.510
4 0.617
5 0.482
6 0.387
7 0.272
8 0.536
9 0.498
10 0.402
11 0.328
12 0.542
13 0.299
14 0.647
15 0.291
16 0.815
17 0.680
18 0.363
19 0.560
20 0.334
21 0.987
22 0.585
23 0.770
24 0.711
25 0.546
26 0.553
27 1.164
28 1.096
29 1.477
30 0.824
31 1.002
32 0.504
33 1.019
34 0.834
35 0.589
36 0.557
37 1.338
38 0.906
39 0.486
40 0.587

Java byte code in practice

I am listening to Rafael on Virtual JUG

Screen Shot 2015-05-20 at 10.27.03 am

‘mvn package’ through our debilitating NTLM proxy

I was morose, grief-stricken and close to tears when our evil corporate proxy stopped me from doing anything. Each tool needs a different type of parameters to pass through this proxy. I tried to use cntlm but that did not help. After many hours I realized that this settings.xml builds everything properly.

There are two proxy sections but I have not attempted to remove one. It is working as it is.

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <activeProfiles>
    <!--make the profile active all the time -->
    <activeProfile>securecentral</activeProfile>
  </activeProfiles>
  <profiles>
    <profile>
      <id>securecentral</id>
      <!--Override the repository (and pluginRepository) "central" from the
         Maven Super POM -->
      <repositories>
        <repository>
          <id>central</id>
          <url>http://repo1.maven.org/maven2</url>
          <releases>
            <enabled>true</enabled>
          </releases>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>central</id>
          <url>http://repo1.maven.org/maven2</url>
          <releases>
            <enabled>true</enabled>
          </releases>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
  <pluginRepositories>
    <pluginRepository>
      <id>central</id>
      <name>Maven Plugin Repository</name>
      <url>http://repo1.maven.org/maven2</url>
      <layout>default</layout>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
      <releases>
        <updatePolicy>never</updatePolicy>
      </releases>
    </pluginRepository>
  </pluginRepositories>
  <proxies>
    <proxy>
      <id>example-proxy</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>proxy</host>
      <port>port</port>
      <username>user</username>
      <password>password</password>
      <nonProxyHosts>www.google.com|*.example.com</nonProxyHosts>
    </proxy>
    <proxy>
      <id>example-proxy</id>
      <active>true</active>
      <protocol>https</protocol>
      <host>proxy</host>
      <port>port</port>
      <username>user</username>
      <password>password</password>
      <nonProxyHosts>www.google.com|*.example.com</nonProxyHosts>
    </proxy>
  </proxies>
</settings>

JDK9 REPL

I have been building and quickly exploring various JDK 9 features during this past weekend.
There is a new REPL now among other gems. I will update this post as I explore it further.

Mohans-MacBook-Pro:openjdk radhakrishnan$ java -version
java version “1.9.0-ea”
Java(TM) SE Runtime Environment (build 1.9.0-ea-b61)
Java HotSpot(TM) 64-Bit Server VM (build 1.9.0-ea-b61, mixed mode)
Mohans-MacBook-Pro:openjdk radhakrishnan$ java -jar kulla-0.508-20150510054454.jar
| Welcome to JShell — Version 0.508
| Type /help for help

->

-> String s = “__mainn__”.replaceAll(“[^a-z\\s]”, “”);
| Added variable s of type String with initial value “mainn”

-> System.out.println(s);

-> mainn

-> final Map count = s.chars().map(Character::toLowerCase).collect(TreeMap::new, (m, c) -> m.merge((char) c, 1, Integer::sum), Map::putAll);
| Warning:
| Modifier ‘final’ not permitted in top-level declarations, ignored
| final Map count = s.chars().map(Character::toLowerCase).collect(TreeMap::new, (m, c) -> m.merge((char) c, 1, Integer::sum), Map::putAll);
| ^—^
| Added variable count of type Map with initial value {a=1, i=1, m=1, n=2}

-> int x = 26;
| Added variable x of type int with initial value 26

-> count.entrySet().stream().sorted((l, r) -> r.getValue().compareTo(l.getValue())).forEach(e -> count.merge(e.getKey(), x–, Math::multiplyExact));

-> System.out.println(count.entrySet().stream());
java.util.
-> stream.ReferencePipeline$Head@548a9f61

-> System.out.println(count.entrySet().stream().mapToDouble(e -> e.getValue()).sum());
124.0

Everything can be changed in the REPL. Nothing is final and it is ignored. That is what kulla-dev@openjdk.java.net told me.

Resolution for MacOS issue related to freetype

A new pull request is submitted.

git clone http://github.com/mohanr/adoptopenjdk-getting-started-kit.git

cd adoptopenjdk-getting-started-kit

git remote add upstream https://github.com/neomatrix369/adoptopenjdk-getting-started-kit.git

git fetch upstream

From https://github.com/neomatrix369/adoptopenjdk-getting-started-kit
* [new branch] Nashorn -> upstream/Nashorn
* [new branch] master -> upstream/master
Mohans-MacBook-Pro:adoptopenjdk-getting-started-kit radhakrishnan$ ls
LANGS.md cover.jpg cover_small.jpg en pt
Mohans-MacBook-Pro:adoptopenjdk-getting-started-kit radhakrishnan$ cd pt
Mohans-MacBook-Pro:pt radhakrishnan$ ls
AdoptOpenJDKLogo-100×100.png feedback.md
README.md handy-scripts-for-OpenJDK-developers.md
SUMMARY.md intermediate-steps
adopt-openjdk-getting-started known-issues
adoptopenjdk-projects openjdk-mailing-lists.md
advanced-steps openjdk-projects
binaries preparations.md
contribute.md source-code
contributors.md thanks_and_support.md
cover.jpg virtual-machines
Mohans-MacBook-Pro:pt radhakrishnan$ cd known-issues/
Mohans-MacBook-Pro:known-issues radhakrishnan$ ls
known_issues.md known_issues_sonarqube.md
known_issues_linuxunix.md known_issues_virtual_machine.md
known_issues_macos.md known_issues_windows.md
known_issues_mercurial.md
Mohans-MacBook-Pro:known-issues radhakrishnan$ cat known_issues_macos.md
# MacOS

The configuration fails because it does not find **freetype

configure: error: Could not find freetype!
configure exiting with result code 1

Check if **freetype** is installed

$ brew install freetype
Error: freetype-2.5.3_1 already installed
To install this version, first `brew unlink freetype’

Since it is already installed we configure like this.

$ bash configure –with-freetype-include=/usr/X11/include/freetype2 –with-freetype-lib=/usr/X11/lib
A new configuration has been successfully created in
/Users/radhakrishnan/OpenJDK/jdk9/build/macosx-x86_64-normal-server-release
using configure arguments ‘–with-freetype-include=/usr/X11/include/freetype2 –with-freetype-lib=/usr/X11/lib’.
Mohans-MacBook-Pro:known-issues radhakrishnan$ git commit -m “Resolution for MacOS issue related to freetype” known_issues_macos.md
[master 9173bc0] Resolution for MacOS issue related to freetype
1 file changed, 18 insertions(+)
Mohans-MacBook-Pro:known-issues radhakrishnan$ git push origin master
Counting objects: 10, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 786 bytes | 0 bytes/s, done.
Total 5 (delta 3), reused 0 (delta 0)
To http://github.com/mohanr/adoptopenjdk-getting-started-kit.git
6f1c9e0..9173bc0 master -> master