Don't download the internet. Share Maven and Ivy repositories with Docker containers

Downloading the internet. A term commonly used when building Maven projects. It is due to downloading your applications dependencies and their transitive dependencies. With a normal application those dependencies can add up to quite a few jars and a lot of megabytes to download. Especially on a 'clean' machine without cached dependencies in your local repository.

Running a new container in Docker sometimes feels the same as every image layer it is built on top of also has to be downloaded. Eventually these are also cached in the Docker cache.

If you then have a Maven based application in Docker you have the perfect storm of bandwidth hogging. Especially as your Docker image's Maven configuration will by default not reuse any cached dependencies and instead always download everything from Maven Central. And for each new build or launch it will re-download the internet as it knows of no local cached dependencies.

Work around, not solution

My work around is to mount my local host repositories for Maven, and for its derivative Ivy, as data volumes for the Docker container. This then avoids re-downloading the internet on every further build and launch and will also reuse dependencies I most likely have already downloaded on the host.

It is not the best solution as your images should ideally not be influenced by what you have on your host machine, but it saves a lot of time, and a lot of grief.

Vagrant

When I run my docker containers inside a Vagrant instance I first mount my host repositories by adding these 2 lines to the Vagrantfile:

config.vm.synced_folder "/Users/myusername/.m2", "/home/vagrant/.m2"

config.vm.synced_folder "/Users/myusername/.ivy2", "/home/vagrant/.ivy2"

Boot2docker

Since Docker 1.3 the OSX /Users path has been by default shared with the boot2docker VM on the same path. So to share the maven repositories you need to link the /Users path to your docker user home folder.

   boot2docker ssh;

   ln -s /Users/mysername/.m2 /home/docker/.m2;

   ln -s /Users/mysername/.ivy2 /home/docker/.ivy2

Data volume container

For easy sharing these folders between many docker containers I mount them as a data volume container and naming it ‘maven’.

   docker run -d -P --name maven \

   -v ~/.m2:/root/.m2:rw \
   -v ~/.ivy2:/root/.ivy2:rw ubuntu

I am basing it on the basic Ubuntu image, and it will stop immediately as no process is running. That is fine, the mounted volumes will still work.

Alternatively instead of mounting the host folders you can have a persistent container with these folders exposed as VOLUME in the Dockerfile so it is shared in a similar manner.

Launch container 

   docker run -d --volumes-from maven \
   yourapplicationimage

Obviously you probably have other options on your docker launch, but the important bit here is the ‘--volumes-from maven’ which maps all the volumes from the data volume container called maven into this container.

Now in theory all containers with Maven and Ivy based build tools such as SBT should first look at the mounted Maven and Ivy repositories for their dependencies.

Build problem

Unfortunately if you build new images frequently you will have the same problem still as the above solution is only for running containers not building. During the build stage Docker does not allow you mount any volumes. This is so it is reproducible anywhere.

In your Dockerfile you can ADD or COPY whole repositories into your image, but that will make it very bloated with a lot of irrelevant dependencies unless you somehow construct and maintain a perfect repository. (Ps. don’t do this).

Maven repository manager

One solution that makes sense in general is to add you companies Maven repository manager’s public Maven Central mirror to the image. That way at least you will only download the intranet, not the internet.

For a Maven build add a settings.xml file to your image folder with at least these settings if using Nexus:

<mirrors>
   <mirror>
      <id>Nexus</id>
      <name>Nexus Public Mirror</name>
      <url>http://nexusmachine/nexus/content/groups/public</url>
      <mirrorOf>central</mirrorOf>
   </mirror>
</mirrors>

                

Then add this to your Dockerfile:

   ADD settings.xml /root/.m2/settings.xml

Note this will be overwritten if you later during the run stage mount .m2 folder on top of it, but for most that is fine as the .m2 folder usually contain a relevant settings.xml file anyway.

For a SBT build add this repositories file to your image folder with this content:

[repositories]

local 

maven-local

company-repo: http://nexusmachine/nexus/content/groups/public

scala-tools-releases

maven-central

Then add this to your Dockerfile:

   ADD repositories /root/.sbt/repositories

Repository manager container

A further solution is to run a repository manager as another container and have all Maven/SBT builds refer to it instead. This avoids the involvement of the host computer yet saves any network traffic as everything is localhost.

Pull down something like https://registry.hub.docker.com/u/mattgruter/artifactory/ .

    docker pull mattgruter/artifactory

Configure then run and name it:

    docker run -d --name artifactory \
    mattgruter/artifactory

Modify the above settings.xml and repositories to refer to you locally linked artifactory name as url instead, e.g this one liner:

   company-ivy-repo: http://artifactory/artifactory/repo/,
   [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
   company-repo: http://artifactory/artifactory/repo/

(The example above added it as an Ivy repository which Artifactory supports).

Hopefully these tips will avoid downloading the internet too often and save a few grey hairs.

Dockerise it all - containerised addiction

TL;DR: CDD - Container Driven Development 

(CDL - Container Driven Life)

Occasionally you come across new interesting technology that is easy to use, and suddenly a veil is unveiled and you become aware of so many opportunities and potential. Maybe it was the same the first time you discovered programming, or OO or FP programming, message queues or NoSQL databases, or distributed source control, or moving from IDE only to proper reproducible build tools (Ant, Maven, Gradle, SBT etc). With certain tools it becomes your favourite hammer that you carry everywhere searching for nails to hit it with.

Docker is such a revelation, at least for me. And I am hammering a wide variety of nails with it. And with the hammer-nail pattern/anti-pattern some of those nails really didn’t need hammering but it was fun.

Docker history

Docker was only revealed in March 2013 by Solomon Hykes and other people at dotCloud . It got a lot of publicity in the hacker news section of the world as people could see the possibly potential in this new tangent of virtualisation. It baked and matured through 2013 as people occasionally showed examples of applying it to their workflows, but still it was too rough for most to use.

But during 2014 especially since the summer the  publicity and widespread use of Docker has exploded. Downloads from Docker’s registry hub of ready made images has exponentially rocketed sine the summer of 2014 by 1387%! Now the majority of people I talk to or follow on twitter mention Docker and show examples of how they use it. Conferences and tech news are flooded with new ways to use Docker. And projects that extend or are based on Docker are multiplying all the time.

As someone who has always been interested in virtualisation, automation, reproducibility, build tools and provisioning, introducing containers like Docker has been a welcomed evolution and a revelation. Being already heavily invested in the use of Vagrant for development and systems the migration to/ combination with Docker has been smooth.

Docker addiction step 1. Exploration

First steps are usually to try the easy hello-world-ish examples. Downloading an Ubuntu image and launching bash inside a Docker container. Basically exploring Docker and its commands, which is what I cover in my how to install, basic use patterns and many handy tips with Docker howto.

Docker addiction step 2. Imagination

My howto document also takes you into step 2 which is more about creating your own basic images. Creating a simple Apache or Nginx container with a simple volumised website. Creating images ready with a Java SDK or node.js etc. I have made some simple Dockerfiles of these type of images available at github.com/flurdy/Dockerfiles.

Then you usually also wrap a simple application of your own in an image and running it on other computers. You start to understand how and where Docker can be used.

Docker addiction step 3. Diversity

Once you got one application inside a container you start to experiment with other applications. Might take a look at other types of applications and more likely support applications such as databases inside a container.

You might start to link containers to each other such as an application to a database container and some of the other handy Docker tips in my howto.

Docker addiction step 4. Proliferation

Now you have tried a mixture of applications with Docker you might spread the use to even more core applications and more. You start to link multiple containerised applications to each other.

Replacing third party integration and old legacy applications with a container image, wiring up and switching between different versions of an application via container linking, etc.

As you start to basically recreate environments using Docker containers you might start to facilitate promotion between staging environments only using container images. And eventually production is containerised.

You are probably already sharing and discovering images at work and on the internet. Using the public Docker registry hub and/or internally with tools such as Quay.io

Docker addiction step 5. Eccentricity

When comfortable with writing Dockerfiles and fully aware of the Docker way of layers and process then you start to experiment with more unconventional Docker images.

You might create Android SDK in an image, ready made SSH tunnels to apps or databases in another. You add tools such as a local maven repository manager, local DNS servers, mail servers, etc. into containers.  You realise desktop applications can actually be containerised.

Eventually tools to facilitate easier and more manageable Docker life such as Fig, Panamax, Shipyard, Dokku, etc. are maybe among your toolset.

Docker addiction step 6. Ubiquitous

By now all your core applications are run/runnable inside a container.

They link to databases, message queues, 3rd party adapters, and other smaller applications that are all also inside containers.

Your testing environments, staging environments, production environment, development environment and REPL are all running inside containers.

When you create a new application or touch an old legacy application your first step is to write a Dockerfile and containerise the application. You design applications and processes from the ground up to be containerised.

This is what I call CDD - Container Driven Development.

Possibly by now you are convinced everything belongs inside a container, so a better name is perhaps CDL- Container Driven Life.