Don't download the internet. Share Maven and Ivy repositories with Docker containers

Downloading the internet. A term commonly used when building Maven projects. It is due to downloading your applications dependencies and their transitive dependencies. With a normal application those dependencies can add up to quite a few jars and a lot of megabytes to download. Especially on a 'clean' machine without cached dependencies in your local repository.

Running a new container in Docker sometimes feels the same as every image layer it is built on top of also has to be downloaded. Eventually these are also cached in the Docker cache.

If you then have a Maven based application in Docker you have the perfect storm of bandwidth hogging. Especially as your Docker image's Maven configuration will by default not reuse any cached dependencies and instead always download everything from Maven Central. And for each new build or launch it will re-download the internet as it knows of no local cached dependencies.

Work around, not solution

My work around is to mount my local host repositories for Maven, and for its derivative Ivy, as data volumes for the Docker container. This then avoids re-downloading the internet on every further build and launch and will also reuse dependencies I most likely have already downloaded on the host.

It is not the best solution as your images should ideally not be influenced by what you have on your host machine, but it saves a lot of time, and a lot of grief.

Vagrant

When I run my docker containers inside a Vagrant instance I first mount my host repositories by adding these 2 lines to the Vagrantfile:

config.vm.synced_folder "/Users/myusername/.m2", "/home/vagrant/.m2"

config.vm.synced_folder "/Users/myusername/.ivy2", "/home/vagrant/.ivy2"

Boot2docker

Since Docker 1.3 the OSX /Users path has been by default shared with the boot2docker VM on the same path. So to share the maven repositories you need to link the /Users path to your docker user home folder.

   boot2docker ssh;

   ln -s /Users/mysername/.m2 /home/docker/.m2;

   ln -s /Users/mysername/.ivy2 /home/docker/.ivy2

Data volume container

For easy sharing these folders between many docker containers I mount them as a data volume container and naming it ‘maven’.

   docker run -d -P --name maven \

   -v ~/.m2:/root/.m2:rw \
   -v ~/.ivy2:/root/.ivy2:rw ubuntu

I am basing it on the basic Ubuntu image, and it will stop immediately as no process is running. That is fine, the mounted volumes will still work.

Alternatively instead of mounting the host folders you can have a persistent container with these folders exposed as VOLUME in the Dockerfile so it is shared in a similar manner.

Launch container 

   docker run -d --volumes-from maven \
   yourapplicationimage

Obviously you probably have other options on your docker launch, but the important bit here is the ‘--volumes-from maven’ which maps all the volumes from the data volume container called maven into this container.

Now in theory all containers with Maven and Ivy based build tools such as SBT should first look at the mounted Maven and Ivy repositories for their dependencies.

Build problem

Unfortunately if you build new images frequently you will have the same problem still as the above solution is only for running containers not building. During the build stage Docker does not allow you mount any volumes. This is so it is reproducible anywhere.

In your Dockerfile you can ADD or COPY whole repositories into your image, but that will make it very bloated with a lot of irrelevant dependencies unless you somehow construct and maintain a perfect repository. (Ps. don’t do this).

Maven repository manager

One solution that makes sense in general is to add you companies Maven repository manager’s public Maven Central mirror to the image. That way at least you will only download the intranet, not the internet.

For a Maven build add a settings.xml file to your image folder with at least these settings if using Nexus:

<mirrors>
   <mirror>
      <id>Nexus</id>
      <name>Nexus Public Mirror</name>
      <url>http://nexusmachine/nexus/content/groups/public</url>
      <mirrorOf>central</mirrorOf>
   </mirror>
</mirrors>

                

Then add this to your Dockerfile:

   ADD settings.xml /root/.m2/settings.xml

Note this will be overwritten if you later during the run stage mount .m2 folder on top of it, but for most that is fine as the .m2 folder usually contain a relevant settings.xml file anyway.

For a SBT build add this repositories file to your image folder with this content:

[repositories]

local 

maven-local

company-repo: http://nexusmachine/nexus/content/groups/public

scala-tools-releases

maven-central

Then add this to your Dockerfile:

   ADD repositories /root/.sbt/repositories

Repository manager container

A further solution is to run a repository manager as another container and have all Maven/SBT builds refer to it instead. This avoids the involvement of the host computer yet saves any network traffic as everything is localhost.

Pull down something like https://registry.hub.docker.com/u/mattgruter/artifactory/ .

    docker pull mattgruter/artifactory

Configure then run and name it:

    docker run -d --name artifactory \
    mattgruter/artifactory

Modify the above settings.xml and repositories to refer to you locally linked artifactory name as url instead, e.g this one liner:

   company-ivy-repo: http://artifactory/artifactory/repo/,
   [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
   company-repo: http://artifactory/artifactory/repo/

(The example above added it as an Ivy repository which Artifactory supports).

Hopefully these tips will avoid downloading the internet too often and save a few grey hairs.

Create, populate and reset a dynamic database (HSQLDB,Hibernate,SQLMaven,DbUnit)

Thought I'd jot down how I create, populate and reset databases for development and testing. I use this method in my Java based pet projects such as Snaps ^[app][code] and Wishlist ^[app][code]. I also include this setup as default in my project template.

My projects are Maven based, using Jetty as java container through its maven plugin. In addition I also use JRebel, IntelliJ IDEA and Ubuntu, a setup I described in this howto: Ubuntu + IntelliJ + Maven + Jetty + JRebel, but none of these are required.

This setup is enables a fluid, very dynamic and quick development process. In addition I then use the in memory database HSQLDB for quick and standalone database interaction.

Using JPA/Hibernate with hsqldb, my tables can be created automatically when I start Jetty. However in developement and testing I prefer to have some default data pre populated.

This is where DbUnit comes in to play. With DBunit's maven plugin I can export my current data and populate future databases with the same data. As it is all easy to read XML I can edit manually the basic stub data as well.

However DbUnit can not create the database as it is run before jetty:run action, so I use the SQL-Maven plugin for this purpose. Again SQL Maven needs to know what tables to create, so I use the Maven Hibernate3 Plugin to export a schema from the JPA annotations.


This combination allows me to:

1. Check out my project anywhere


2. Create and populate the database with stub data with one command.


3. Run the application via jetty


4. Test the application straight away


5. or develop dynamically with instant feedback

How to set up Hibernate, SQLMaven and DbUnit plugins

Prerequisites/assumptions

• Java


• Maven


• Jetty (can be tomcat maven plugin)


• JPA annotations


• HSQLDB (using file based hsqldb persistance to survive restarts)

JPA annotations -> Schema: Hibernate

To export the JPA annotations into a database schema we include the Maven Hibernate3 Plugin in your pom.xml. This is quite a long plugin section as it defines a few dependencies which otherwise may be overriden by the plugin and lead to problems like this mapping exception.


<profile>
  <id>hbm-export</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
   <artifactId>hibernate3-maven-plugin</artifactId>
        <version>2.2</version>
        <executions>
          <execution>
            <phase>process-classes</phase>
            <goals>
              <goal>hbm2ddl</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <components>
            <component>
              <name>hbm2ddl</name>
      <implementation>jpaconfiguration</implementation>
            </component>
          </components>
          <componentProperties>
    <persistenceunit>${project.artifactId}</persistenceunit>
            <outputfilename>schema.ddl</outputfilename>
            <drop>false</drop>
            <create>true</create>
            <export>false</export>
            <format>true</format>
          </componentProperties>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>hsqldb</groupId>
       <artifactId>hsqldb</artifactId>
            <version>${hsqldb.version}</version>
          </dependency>
          <dependency>
            <groupId>org.hibernate</groupId>
       <artifactId>hibernate-entitymanager</artifactId>
            <version>${hibernate.version}</version>
            <exclusions>
              <exclusion>
                <groupId>cglib</groupId>
         <artifactId>cglib</artifactId>
              </exclusion>
              <exclusion>
                <groupId>commons-logging</groupId>
    <artifactId>commons-logging</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
          <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-core</artifactId>
            <version>${hibernate.version}</version>
            <exclusions>
              <exclusion>
                <groupId>cglib</groupId>
          <artifactId>cglib</artifactId>
              </exclusion>
              <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
          <dependency>
            <groupId>org.hibernate</groupId>
      <artifactId>hibernate-annotations</artifactId>
            <version>${hibernate.version}</version>
            <exclusions>
              <exclusion>
                <groupId>cglib</groupId>
                <artifactId>cglib</artifactId>
              </exclusion>
              <exclusion>
                <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
        </dependencies>
      </plugin>
    </plugins>
  </build>
</profile>


To run the schema creation action on its own:

mvn -DskipTests \

-Dhibernate.dialect=org.hibernate.dialect.HSQLDialect \

-P hbm-export compile hibernate3:hbm2ddl;

Note: We need to compile the JPA classes so that the plugin can extract annotation information from them.

Note2: that Hibernate/Red Hat have a bad track record of including their plugins and releases properly in maven central repository, so you might either want to adjust your dependeinces and repositories accordingly. Or you can use my proxy repository:


<repositories>
  <repository>
    <id>code-flurdy-repo</id>
    <name>code@flurdy repository</name>
    <url>http://code.flurdy.com/nexus/content/groups/noncentral</url>
    <releases><enabled>true</enabled></releases>
    <snapshots><enabled>true</enabled></snapshots>
  </repository>
</repositories>
<pluginRepositories>
  <pluginRepository>
    <id>code-flurdy-repo</id>
    <name>code@flurdy repository</name>
    <url>http://code.flurdy.com/nexus/content/groups/noncentral</url>
    <releases><enabled>true</enabled></releases>
    <snapshots><enabled>false</enabled></snapshots>
  </pluginRepository>
</pluginRepositories>


Please if your project is popular do not use my repo as it will explode my ec2/S3 data bandwidth usage!

Schema -> Database: SQLMaven

To create your database from this schema with SQL-Maven, add this to you pom.xml


<profile>
  <id>sqlmaven</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>sql-maven-plugin</artifactId>
        <version>1.3</version>
        <dependencies>
          <dependency>
            <groupId>hsqldb</groupId>
            <artifactId>hsqldb</artifactId>
            <version>${hsqldb.version}</version>
          </dependency>
        </dependencies>
        <configuration>
          <driver>${db.driverClassName}</driver>
          <url>${db.url}</url>
          <username>${db.username}</username>
          <password>${db.password}</password>
          <autocommit>true</autocommit>
     <srcFiles>
     <srcFile>target/hibernate3/sql/schema.ddl</srcFile>
          </srcFiles>
        </configuration>
      </plugin>
    </plugins>
  </build>
</profile>


To run the database creation action on its own:

mvn -DskipTests -P sqlmaven sql:execute;

Database population: DbUnit

To populate your database via DbUnit, add this to you pom.xml


<profile>
  <id>dbunit</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>dbunit-maven-plugin</artifactId>
        <version>1.0-beta-3</version>
        <dependencies>
          <dependency>
            <groupId>hsqldb</groupId>
            <artifactId>hsqldb</artifactId>
            <version>${hsqldb.version}</version>
          </dependency>
        </dependencies>
        <configuration>
          <dataTypeFactoryName>org.dbunit.ext.hsqldb.HsqldbDataTypeFactory</dataTypeFactoryName>
          <url>jdbc:hsqldb:file:${project.basedir}/target/db/build;shutdown=true</url>
          <driver>org.hsqldb.jdbcDriver</driver>
          <username>sa</username>
          <password></password>
          <type>CLEAN_INSERT</type>
          <src>src/test/data/stub.xml</src>
        </configuration>
      </plugin>
    </plugins>
  </build>
</profile>


First you need data to export. I suggest you run jetty:run action and add/use the application to create data in the database. Eg.: Register a user, create default domain objects, or whatever your application uses.

You then stop the application and export the data created:

mvn -DskipTests -P dbunit dbunit:export;

These are by default exported to target/dbunit/export.xml.

These are a good base data. You may try and adjust and extend this file if you like, but be aware that you might brake it, so prehaps try unadjusted initially.

Copy the export file to test/data so that it can be part of your source control etc. You may have noticed I already included the test/data as a src element in the plugin.

cp target/dbunit/export.xml test/data/stub.xml;

Run the database population action (remember to have a new clean database):

mvn -DskipTests -P dbunit dbunit:operation;


Example configuration of all these plugins can be found in this project's pom.xml.

Combined

Above are the basic steps. These I then aggregate into a couple of common one liners:

Create database:

mvn -o -DskipTests \

-Dhibernate.dialect=org.hibernate.dialect.HSQLDialect \

-P hbm-export,sqlmaven \

compile hibernate3:hbm2ddl sql:execute;



Create and populate database:

mvn -o -DskipTests \

-Dhibernate.dialect=org.hibernate.dialect.HSQLDialect \

-P hbm-export,sqlmaven,dbunit \

compile hibernate3:hbm2ddl \

sql:execute dbunit:operation;



Reset database:

rm -rf target/db;

mvn -o -DskipTests \

-Dhibernate.dialect=org.hibernate.dialect.HSQLDialect \

-P hbm-export,sqlmaven,dbunit \

compile hibernate3:hbm2ddl \

sql:execute dbunit:operation;



Clean, rebuild database and run jetty:

mvn -o -DskipTests \

-Dhibernate.dialect=org.hibernate.dialect.HSQLDialect \

-P hbm-export,sqlmaven,dbunit \

clean compile hibernate3:hbm2ddl \

sql:execute dbunit:operation jetty:run;

As you can see these one liners can be quite long so I again wrap these into bash scripts that I put in a bin folder.

You can extend this further to include these plugins as enabled by default and the maven goals as part of your other goals: E.g. include hbm:export and sqlmaven as part of install or test command or dbunit:operation with the jetty:run command, so that they are totally automatic. However I prefer a bit more control of when my data is reset etc.

Summary

With these plugins I can in one command create and populate my database, so that my app is up and running with data very quickly. Hope this is of use to others.

org.hibernate.MappingException: Could not determine type for Set / List

I was simplifying the domain model of a pet project by removing an unneccesarry entity by using JPA 2.0's ElementCollection annotation:

public class SecurityDetail {

public enum AuthorityRole{
  ROLE_ADMIN,
  ROLE_SUPER,
  ROLE_USER
}
....
@ElementCollection
@CollectionTable(
  name="Authority",
  joinColumns=@JoinColumn(name="username")
)
private Set authorities;
....

However I was getting this error message:
org.hibernate.MappingException: Could not determine type for: java.util.Set, at table: SecurityDetail, for columns: [org.hibernate.mapping.Column(authorities)]


Digging around (googling) I ended up on this blog entry: blog.m1key.me/2010/06/orghibernatemappingexception-could-not.html, which main solution is to make sure you have a newer hibernate, version 3.5.3 or later.


I use maven, with the hibernate3-maven-plugin, sql-maven-plugin and dbunit-maven-plugin to create the schema, database and to populate data. So I bumped the org.hibernate:hibernate-entitymanager dependency from 3.5.0 to 3.5.3-Final. (and fixed my own nexus repository to pull JBoss's latest jars.)

However it did not fix the problem. But by reading the comments of the blog entry above I also realised that my hibernate3-maven-plugin may be using older dependencies. So I explicitly added the recent verions of org.hibernate:hibernate-core dependency to both the plugin and the app, as well as org.hibernate:hibernate-entitymanager to the plugin as well:

<properties>
  <hibernate.version>3.5.3-Final</hibernate.version>
</properties>
....
<dependencies>
  <dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-entitymanager</artifactId>
    <version>${hibernate.version}</version>
    <exclusions>
      <exclusion>
        <groupId>cglib</groupId>
        <artifactId>cglib</artifactId>
      </exclusion>
      <exclusion>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
      </exclusion>
    </exclusions>
  </dependency>
  <dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-core</artifactId>
    <version>${hibernate.version}</version>
    <exclusions>
      <exclusion>
        <groupId>cglib</groupId>
        <artifactId>cglib</artifactId>
      </exclusion>
      <exclusion>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
      </exclusion>
    </exclusions>
  </dependency>
</dependencies>
....
<plugins>
  <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>hibernate3-maven-plugin</artifactId>
    <version>2.2</version>
    <executions>
      <execution>
        <phase>process-classes</phase>
        <goals>
          <goal>hbm2ddl</goal>
        </goals>
      </execution>
    </executions>
    <configuration>
      <components>
        <component>
          <name>hbm2ddl</name>
          <implementation>jpaconfiguration</implementation>
        </component>
      </components>
      <componentProperties>
        <persistenceunit>${project.artifactId}</persistenceunit>
        <outputfilename>schema.ddl</outputfilename>
        <drop>false</drop>
        <create>true</create>
        <export>false</export>
        <format>true</format>
      </componentProperties>
    </configuration>
    <dependencies>
      <dependency>
        <groupId>hsqldb</groupId>
        <artifactId>hsqldb</artifactId>
        <version>${hsqldb.version}</version>
      </dependency>
      <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-entitymanager</artifactId>
        <version>${hibernate.version}</version>
        <exclusions>
          <exclusion>
            <groupId>cglib</groupId>
            <artifactId>cglib</artifactId>
          </exclusion>
          <exclusion>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
          </exclusion>
        </exclusions>
      </dependency>
      <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-core</artifactId>
        <version>${hibernate.version}</version>
        <exclusions>
          <exclusion>
            <groupId>cglib</groupId>
            <artifactId>cglib</artifactId>
          </exclusion>
          <exclusion>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
          </exclusion>
        </exclusions>
      </dependency>
    </dependencies>
  </plugin>
</plugins>

Skinny WAR Assembly in maven

If you want to create a "Skinny WAR" so that you can package it as part of an EAR or as in my case as part of an assembly later, then you can do this:


<plugin>
   <artifactId>maven-war-plugin</artifactId>
   <version>2.1-beta-1</version>
   <configuration>
     <packagingExcludes>WEB-INF/lib/*.jar</packagingExcludes>
     <archive>
       <manifest>
         <addClasspath>true</addClasspath>
         <classpathPrefix>lib/</classpathPrefix>
       </manifest>
     </archive>
   </configuration>
</plugin>


If you then later in another pom.xml want to include this war and the excluded libraries:


<dependency>
   <groupId>${project.groupId}</groupId>
   <artifactId>you war module name</artifactId>
   <version>${project.version}</version>
   <type>war</type>
</dependency>
<dependency>
   <groupId>${project.groupId}</groupId>
   <artifactId>you war module name</artifactId>
   <version>${project.version}</version>
   <type>pom</type>
</dependency>


IntelliJ IDEA on Ubuntu (with Maven , Jetty and JRebel)

I have written an howto on Ubuntu+IntelliJ+Maven+Jetty+JRebel, basically a quick quide to get IntelliJ IDEA working on Ubuntu, with Maven, the Jetty plugin for Maven and with JRebel (JavaRebel).

It assumes a clean basic Ubuntu install, and is quite terse for now. I may extend it with more example work flows etc in the future.

Comments are welcome.

groovy maven netbeans jetty facebook spring

Time to rewrite my old wishlist facebook application.

The old one (v3.x) dont work anymore as the web services are gone, I took them offline many months ago. The new application will use the wishlist 4.x web services running on wish.flurdy.com.

Tech stack for this project will be:

Language: groovy.
Build tool: maven.
Web server: jetty (maven plugin).
IDE: netbeans (6.5, with groovy plugin and maven plugin).
Framework: spring (+mvc and WebServices).

So really, only groovy is new compared to the other wishlist projects, but it will be nice to see groovy interact with them.

Already hitting some snags with groovy, maven and netbeans. Seems to be a few posts on how to use netbeans and groovy, or maven and groovy, but not all 3 together.

May at some stage try grails.

Netbeaning again

Return of the lost child...

Been an Eclipse devotee for several years now. But have recently returned to NetBeans. I was using netbeans before eclipse, but changed as eclipse offered more and faster development. (And JBuilder, Kawa, jedit in ancient time before then...)

Been really impressed with NetBeans 6.1, it has really moved on from 4.x range I was using, and the 5.0 version I last checked out.

First impression was how well maven works within netbeans. In eclipse the m2eclipse plugin does somehow works, but is cumbersome, and always something is a pain. In netbeans, the mevenide just works. Plain and simple. It just works as it should do, with full module structure, easy usage, seem fully integrated with the rest of the ide. nice.

In general netbeans is a lot clearer and cleaner, eclipse interface is very messy and cumbersome.

So Ill be a Sun slave for awhile now, till the next new better thing arrive...

( looking through my blog, it is becomming more and more geeky. oh well. :) )