Rusting, a quick dabble with the Rust language

I have spent a few spare moments in the last week looking at Rust, a relative new language.

With the kids on Easter break and naturally requiring frequent attention, and a few brief actual paid client work, I have not had as long uninterrupted time focused on Rust as I would have liked. But I have managed to knock together some basic code.

Rustic Pizza

I created github.com/flurdy/rustic-pizza as my playground with rust and various web frameworks. It is very basic code and I expect I will be quite embarrassed by it in just a few months but it is a start. Rustic pizza will eventually contain several very basic web applications each in a different Rust web frameworks, all for ordering pizza. The pizzeria is a concept that I have created many times over in Java and Scala as an example for piece of framework or similar, some of which I have documented on my flurdy.com website.

Experience so far

So far I have found Rust interesting. 

It perhaps use a bit too much of lazy abbreviated names for its keywords, primitives and core methods than I feel is reasonable, but that is me being pedantic. The ownership transfers, lifetimes and everything by reference is a bit of headfuck, but I think I got the hang of it. Rust does also have some of the monadic traits I am comfortable with from the Scala world, though its Futures seems to be somewhat in its infancy at the moment.

Install Rust

Rust can be installed in numerous ways, including via Brew for both Rust and Cargo, its build tool. But I recommend using Rustup, rustup.rs. Rustup lets you toggle between release and nightly version of Rust, and some frameworks require nightly builds.

Learning Rust

The main Rust website, rust-lang.org, is a good place to start to learn about Rust. Especially the "Rust book", https://doc.rust-lang.org/book/README.html, is a great source to learn about basic Rust concepts. Further reading at rustbyexample.com/index.html and aturon.github.io was very useful. As well as various blogs on Rust such as hoverbear.org and hermanradtke.com.

Once you start coding I also recommend the standard library documentation, doc.rust-lang.org/std/, to find out what methods are available, e.g. does option have map, and what is getOrElse called in rust (unwrap) etc.

Web frameworks

As I most of the time create web services or full webapps an important part of my interest is which web frameworks are available. Fortunately the Rust community also recognise the importance of this, so www.arewewebyet.org and github.com/flosse/rust-web-framework-comparison are great information on which frameworks and libraries are available with Rust.

This lead me to take a look at the newest kid on the block Rocket, rocket.rs. So far Rocket have worked very well for me.

For my Rustic Pizza I plan to also take a look at Iron, http://ironframework.io, as it is the most popular Rust web framework on github. Though it has not been updated as much lately compared to Rocket.

Hopefully if time allows I may get to look more in depth into Nickel, Conduit and Rustful as well.

Keep rusting?

Whether I will keep using Rust time will tell. Core Rust concepts such as ownership, references, memory management etc are not aspects I care to manually control and think about too much as I hope a compiler and garbage collector optimises and handles those for me. But I will keep at it so maybe Rust will accompany if not replace Scala as the main hammer in my toolbox some day.

ROC theorem: Readable, Optimised and Correct code. Pick two.

ROC theorem

With database you have the well known CAP theorem. Consistency, Availability and Partition Tolerance. You can only have two. Databases have to make compromises between these pillars. You can not fully have all three.

With code you also have to decide on compromises between readable code, optimised code and correct code (ROC). And you can not have all three.

This often creates arguments between people on soap boxes from the various camps.

Correct code

Correct code, clever, terse, generic code that avoids handling a lot of edge cases. Often functional code that can be very elegant with little to no theoretic side effects. And easily composed as part of other code.

Correct code can be readable and fast, but also sometimes horrible to understand and very costly to train, write and maintain.

Optimised code

Optimised code, fast and scalable. No unnecessary cruft and takes short cuts to achieve the end results so performant from day one.

Scalable code, code you optimise to support horizontal scalable solutions, with little state and restartable.

May discourage typed system e.g. a message based Actor system, multiple layers of caching, or overuse of parallelised futures to avoid bottlenecks.

Optimised code can be "correct" code but often full of unclear and undocumented short cuts and frustratingly slow/buggy to develop.

Readable code

Readable code, simple to read and quick to understand, by people of different levels of skill. Easy to spot bugs and is maintainable by anyone. It is pragmatic in its approach and quick to develop. Can be terse if it is the most readable but often more verbose.

Readable can be "correct" as flow is easier to understand, but often not particularly performant and can be at more at risk of bugs due to more exposed code.

Not mutually exclusive

You can have all 3 pillars for some smaller sets of code. But not for whole code bases and at a cost for how much.

This is more about the priority and focus of the code you write.

• Will others work on the code base, today, next week, next year? Then readable is important.
• Is multiple people working on the same part of the code base? From different teams? With varied experience, or even just different locations? Then readable is important.
• Is the solution used by millions? Does big O make any difference? Then consider optimised code. Note: very, very few companies/products actually need this.
• Is a bottleneck been proven in production? Then optimised code is valid. But not necessarily across the code base.
• Is the product business critical? With heavy integration dependants? Then correct code may be a priority.
• Is the team highly skilled? Not that large, and low churn? Then correct code is an option.
• Do you pair program 100%, from day 1 of onboarding? Then correct code is an option.

With unlimited time to implement and continuously heavy training, and therefore a lot money, you may achieve higher levels of 2 of them or even all 3. But that is not realistic. Pick your priorities. These are not mutually exclusive but they are at a cost of each other.

All the ROCs

You may detect my preferences. I prefer very simple and readable code, that is functional, and scalable. In that order.

I like that anyone can pick up and work on a task for most parts of a code base. I subscribe to the idea of frequent pair rotation across tasks and systems to make sure multiple people is aware of and had an input into any part of an architecture. That leads towards readable code so the overhead of swapping is low.

That a new member of our team or from another team in 6 months time can easily contribute to "our" code base for a small pull request without learning "our take" on category theory is valuable.

I prefer functional programming, with proper type checking, using functors and monads for composition. I like terse code that I can trust, but it must still be readable and maintainable by others than the original author(s).

So some overuse of higher kinded types, free monads, etc adds too much cruft for me, and risky recruitment demands. (At the moment, I am prone to evolve and may have completely changed my mind by the time you read this...)

Horizontally scalable, concurrent code is in the back of my mind of most of solutions. No state, using futures, REST principles etc are core to all my code.

But I detest premature optimisation. Only occasionally in my career have I had to modify any code to support some optimised flow. I do not work for Facebook/Twitter/Google (yet) but I have worked for financial, telcos, and games companies with enormous traffic, and still this was rarely a problem at code level.

I often spot potential optimisations and consciously say no, not yet, if it is not also the most readable and correct alternative. I even avoid parallelisation of futures if there is not yet any obvious need especially if it makes the code less readable.

Readable, Optimised and Correct code. ROC. Pick two. Or just rank them.

Evolve or wither slowly

3 years ago (wow that went fast!) I wrote a blog post called Do not rewrite. The main points in it was to never rewrite whole applications. Never. Instead gradually rewrite one feature at the time whilst still delivering new business value.

I also wrote a more recent post about Paper cuts and broken windows. Which emphasises the importance of fixing problems straight away and not let them fester.

I want to extend these posts to include you should rewrite all the time, even when not immediately necessary.

Recap

Extracting the main points in the previous blogs why big rewrites are bad and continual rewrites are good, and to also sum the obvious why no writes at all are very bad.

Big rewrite disadvantages

• Cost
• No business feature - no value
• Risk
• Big deployments
• Forgotten features
• Likely to never finish
• Not fully replacing old system
• Support yet another system

Continual rewrite benefits

• Reduced risk
• Smaller delta
• Quicker feedback
• Modularising
• Faster - less bottlenecks
• Leaner
• Remove features and bloat
• Actually finishing

No rewrite disadvantages

• Death by thousand paper cuts
• Even the smallest change becomes slow and painful
• Broken windows
• People do not care if they introduce bugs or break other things
• Staff exodus
• Few wants to work with painful systems

Less obvious benefits

There are other perhaps unexpected but important benefits not raised in the previous post. 

Business domain knowledge

Every time a refactor or minor rewrite takes place the knowledge of that part is refreshed and stay current in the company. If a system is not touched for a long while the knowledge of it may be forgotten especially if staff involved has moved on. 

If a critical bug or urgent feature needs to be added to that system then the turn around is exponentially different between a recently updated system and one forgotten.

Technical knowledge

This is also true for the technical part of a system, the how, not just the what and why it does something. An old COBOL or complicated legacy custom build Java application will take a long time to work out compared to a contemporary tech stack application.  You might even have to hire new staff or expensive contractors to fix it.

With up to date knowledge it will be a lot less risk that they might not do it properly.

Quick turnaround

If a new feature needs to be added then to any system then getting people ramped up and delivering it will be much quicker if it is on a contemporary and well known tech stack. If a new system needs to be integrated with existing systems then the API integration will be smoother and a lot quicker.

Technical migration

If a mass migration of a technology stack is needed, perhaps migrating from in house data warehouse to a public cloud based provider, or moving from monolithic to horizontally auto scaled instances, then having most systems on a contemporary and probably quite similar technology stack will speed any urgent migrations. This will avoid/reduce the need to for many deep cave explorations of old customised mystery legacy applications.

Staff retention

A very important reason to rewrite applications and features and in general keep technology up to date is staff retention. (Obviously a million other reasons exists for staff churn as well).

If you avoid death by a thousand paper cuts and broken windows staff will not mind working on the products. A negative culture will be less likely.

If you migrate to newer technologies and let people frequently learn new stuff they will be much more interested in the work and less likely to want to move elsewhere. (refer to my blog post Peak interest - the learning and sharing curve).

Recruitment

If word spreads that you keep an up to date and interesting technology stack then hiring new members of staff will be a lot easier, and you will hopefully attract more qualified candidates. On boarding will also be quick and people will be up to speed quicker with newer well known technology.

Lean technology - lean organisation

If the applications are continually refactored and evolved (as long as it is not done in a hacky ninja rock star development fashion) the architectures will become modularised, scalable and leaner in general. 

Not necessarily a micro service architecture but a lean architecture that are more likely to be adaptable for the future.

Hopefully a lean architecture and highly skilled retained staff will also lead to / need a lean organisation so the business is more likely to succeed.

Rewrite when not needed

I would also suggest people and organisations refactor and rewrite when they do not see the obvious reasons for it.

I do not entirely mean rewriting applications on a whim when there is no features to be added or bugs to be fixed, after all I do emphasise the need for delivering business value along with all rewrites. 

But I do think even when the obvious pain is not there that you should try to refactor and update applications. For example if the application is just one or two version behind the latest but still seems fine then still update it. It will be less painful than when it next time is 3-4 version behind with a bigger delta of change.

Rewrite encapsulation

One type of no business value rewrite I would suggest is a good idea is to encapsulate legacy systems API as soon as you come across it.

Even if you are not changing a legacy system but merely reading data from for example, then adding a contemporary facade to it straight away will be of value in the future when you do need to update it.

Experiment and rewrite experiments

Another important staff retention technique on top generally keep up to date with technology is to let them experiment with new technology. 

I would avoid core systems, especially customer facing systems, but internal tools is prime candidates for field testing technology.

This will also lead to discovering technology that you can use in other applications if proven useful that otherwise would have been missed or delayed.

Naturally these experimental applications should also not be abandonware and be rewritten as often as other systems. Though they will probably always be good candidates for next set of technology experimentations.

Rewrite exploration and euthanasia

Certain system never comes involved in new features so will not be included in a normal rewrite. These will consciously have to be found and rewritten without business value. Though probably just as a contemporary facade initially. 

Leaving these as abandonware is not a good idea. Either kill them or update them.

Summary

The main points was already covered in my previous blogs, but I hope people see the value of rewriting frequently to especially keep staff, organisational architecture and company in general up to date for whatever happens in the future.

A utopian expectation of all applications being up to date is too much to hope for. At the other end where companies that rarely and/or minimally update their applications, they are doomed to fail. The grey scale in between decide whether companies will wither and eventually die or survive. 

Don't download the internet. Share Maven and Ivy repositories with Docker containers

Downloading the internet. A term commonly used when building Maven projects. It is due to downloading your applications dependencies and their transitive dependencies. With a normal application those dependencies can add up to quite a few jars and a lot of megabytes to download. Especially on a 'clean' machine without cached dependencies in your local repository.

Running a new container in Docker sometimes feels the same as every image layer it is built on top of also has to be downloaded. Eventually these are also cached in the Docker cache.

If you then have a Maven based application in Docker you have the perfect storm of bandwidth hogging. Especially as your Docker image's Maven configuration will by default not reuse any cached dependencies and instead always download everything from Maven Central. And for each new build or launch it will re-download the internet as it knows of no local cached dependencies.

Work around, not solution

My work around is to mount my local host repositories for Maven, and for its derivative Ivy, as data volumes for the Docker container. This then avoids re-downloading the internet on every further build and launch and will also reuse dependencies I most likely have already downloaded on the host.

It is not the best solution as your images should ideally not be influenced by what you have on your host machine, but it saves a lot of time, and a lot of grief.

Vagrant

When I run my docker containers inside a Vagrant instance I first mount my host repositories by adding these 2 lines to the Vagrantfile:

config.vm.synced_folder "/Users/myusername/.m2", "/home/vagrant/.m2"

config.vm.synced_folder "/Users/myusername/.ivy2", "/home/vagrant/.ivy2"

Boot2docker

Since Docker 1.3 the OSX /Users path has been by default shared with the boot2docker VM on the same path. So to share the maven repositories you need to link the /Users path to your docker user home folder.

   boot2docker ssh;

   ln -s /Users/mysername/.m2 /home/docker/.m2;

   ln -s /Users/mysername/.ivy2 /home/docker/.ivy2

Data volume container

For easy sharing these folders between many docker containers I mount them as a data volume container and naming it ‘maven’.

   docker run -d -P --name maven \

   -v ~/.m2:/root/.m2:rw \
   -v ~/.ivy2:/root/.ivy2:rw ubuntu

I am basing it on the basic Ubuntu image, and it will stop immediately as no process is running. That is fine, the mounted volumes will still work.

Alternatively instead of mounting the host folders you can have a persistent container with these folders exposed as VOLUME in the Dockerfile so it is shared in a similar manner.

Launch container 

   docker run -d --volumes-from maven \
   yourapplicationimage

Obviously you probably have other options on your docker launch, but the important bit here is the ‘--volumes-from maven’ which maps all the volumes from the data volume container called maven into this container.

Now in theory all containers with Maven and Ivy based build tools such as SBT should first look at the mounted Maven and Ivy repositories for their dependencies.

Build problem

Unfortunately if you build new images frequently you will have the same problem still as the above solution is only for running containers not building. During the build stage Docker does not allow you mount any volumes. This is so it is reproducible anywhere.

In your Dockerfile you can ADD or COPY whole repositories into your image, but that will make it very bloated with a lot of irrelevant dependencies unless you somehow construct and maintain a perfect repository. (Ps. don’t do this).

Maven repository manager

One solution that makes sense in general is to add you companies Maven repository manager’s public Maven Central mirror to the image. That way at least you will only download the intranet, not the internet.

For a Maven build add a settings.xml file to your image folder with at least these settings if using Nexus:

<mirrors>
   <mirror>
      <id>Nexus</id>
      <name>Nexus Public Mirror</name>
      <url>http://nexusmachine/nexus/content/groups/public</url>
      <mirrorOf>central</mirrorOf>
   </mirror>
</mirrors>

                

Then add this to your Dockerfile:

   ADD settings.xml /root/.m2/settings.xml

Note this will be overwritten if you later during the run stage mount .m2 folder on top of it, but for most that is fine as the .m2 folder usually contain a relevant settings.xml file anyway.

For a SBT build add this repositories file to your image folder with this content:

[repositories]

local 

maven-local

company-repo: http://nexusmachine/nexus/content/groups/public

scala-tools-releases

maven-central

Then add this to your Dockerfile:

   ADD repositories /root/.sbt/repositories

Repository manager container

A further solution is to run a repository manager as another container and have all Maven/SBT builds refer to it instead. This avoids the involvement of the host computer yet saves any network traffic as everything is localhost.

Pull down something like https://registry.hub.docker.com/u/mattgruter/artifactory/ .

    docker pull mattgruter/artifactory

Configure then run and name it:

    docker run -d --name artifactory \
    mattgruter/artifactory

Modify the above settings.xml and repositories to refer to you locally linked artifactory name as url instead, e.g this one liner:

   company-ivy-repo: http://artifactory/artifactory/repo/,
   [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
   company-repo: http://artifactory/artifactory/repo/

(The example above added it as an Ivy repository which Artifactory supports).

Hopefully these tips will avoid downloading the internet too often and save a few grey hairs.

Dockerise it all - containerised addiction

TL;DR: CDD - Container Driven Development 

(CDL - Container Driven Life)

Occasionally you come across new interesting technology that is easy to use, and suddenly a veil is unveiled and you become aware of so many opportunities and potential. Maybe it was the same the first time you discovered programming, or OO or FP programming, message queues or NoSQL databases, or distributed source control, or moving from IDE only to proper reproducible build tools (Ant, Maven, Gradle, SBT etc). With certain tools it becomes your favourite hammer that you carry everywhere searching for nails to hit it with.

Docker is such a revelation, at least for me. And I am hammering a wide variety of nails with it. And with the hammer-nail pattern/anti-pattern some of those nails really didn’t need hammering but it was fun.

Docker history

Docker was only revealed in March 2013 by Solomon Hykes and other people at dotCloud . It got a lot of publicity in the hacker news section of the world as people could see the possibly potential in this new tangent of virtualisation. It baked and matured through 2013 as people occasionally showed examples of applying it to their workflows, but still it was too rough for most to use.

But during 2014 especially since the summer the  publicity and widespread use of Docker has exploded. Downloads from Docker’s registry hub of ready made images has exponentially rocketed sine the summer of 2014 by 1387%! Now the majority of people I talk to or follow on twitter mention Docker and show examples of how they use it. Conferences and tech news are flooded with new ways to use Docker. And projects that extend or are based on Docker are multiplying all the time.

As someone who has always been interested in virtualisation, automation, reproducibility, build tools and provisioning, introducing containers like Docker has been a welcomed evolution and a revelation. Being already heavily invested in the use of Vagrant for development and systems the migration to/ combination with Docker has been smooth.

Docker addiction step 1. Exploration

First steps are usually to try the easy hello-world-ish examples. Downloading an Ubuntu image and launching bash inside a Docker container. Basically exploring Docker and its commands, which is what I cover in my how to install, basic use patterns and many handy tips with Docker howto.

Docker addiction step 2. Imagination

My howto document also takes you into step 2 which is more about creating your own basic images. Creating a simple Apache or Nginx container with a simple volumised website. Creating images ready with a Java SDK or node.js etc. I have made some simple Dockerfiles of these type of images available at github.com/flurdy/Dockerfiles.

Then you usually also wrap a simple application of your own in an image and running it on other computers. You start to understand how and where Docker can be used.

Docker addiction step 3. Diversity

Once you got one application inside a container you start to experiment with other applications. Might take a look at other types of applications and more likely support applications such as databases inside a container.

You might start to link containers to each other such as an application to a database container and some of the other handy Docker tips in my howto.

Docker addiction step 4. Proliferation

Now you have tried a mixture of applications with Docker you might spread the use to even more core applications and more. You start to link multiple containerised applications to each other.

Replacing third party integration and old legacy applications with a container image, wiring up and switching between different versions of an application via container linking, etc.

As you start to basically recreate environments using Docker containers you might start to facilitate promotion between staging environments only using container images. And eventually production is containerised.

You are probably already sharing and discovering images at work and on the internet. Using the public Docker registry hub and/or internally with tools such as Quay.io

Docker addiction step 5. Eccentricity

When comfortable with writing Dockerfiles and fully aware of the Docker way of layers and process then you start to experiment with more unconventional Docker images.

You might create Android SDK in an image, ready made SSH tunnels to apps or databases in another. You add tools such as a local maven repository manager, local DNS servers, mail servers, etc. into containers.  You realise desktop applications can actually be containerised.

Eventually tools to facilitate easier and more manageable Docker life such as Fig, Panamax, Shipyard, Dokku, etc. are maybe among your toolset.

Docker addiction step 6. Ubiquitous

By now all your core applications are run/runnable inside a container.

They link to databases, message queues, 3rd party adapters, and other smaller applications that are all also inside containers.

Your testing environments, staging environments, production environment, development environment and REPL are all running inside containers.

When you create a new application or touch an old legacy application your first step is to write a Dockerfile and containerise the application. You design applications and processes from the ground up to be containerised.

This is what I call CDD - Container Driven Development.

Possibly by now you are convinced everything belongs inside a container, so a better name is perhaps CDL- Container Driven Life.

Pair with people you like and code reviews with people you don't like

Pair with people you like and code reviews with people you don't like (misquoted from someone way smarter than me).

That sums it up very well for how I feel about both practices.

With large teams, especially if distributed or partially outsourced, code reviews can ensure code quality and are quite essential. It will allow you to share some knowledge and instill best practices. It will be a reminder for people that their code will be viewed by others so don’t take shortcuts.

However code review can also be a total bottleneck if over-bureaucratic. It will add an overhead for all work. If there are some high and mighty gate keepers that will stop you from pushing your code frequently then you have a complete velocity block. If your code is shit then fair enough, but if it is merely nit picking or just disagreements between styles then it is very costly. If however if any compulsory reviewer is not in your office, country, time zone or just very busy then that adds a large delay in the feedback loop.

Another issue with code reviews is that quite a few reviews are of low quality due to lack of context. They do not necessarily know all the discussions, history of why a piece of code works this way, project code style, etc. Especially if they are of the tainted ivory tower architect affliction.


In smaller, agile and especially collocated teams code reviews will flag issue unnecessarily late in the process. Just pair from the start instead to ensure no short cuts or dodgy code slips through, and automatically spread the knowledge. If you do not trust two of your developers combined then you do have a serious problem.

You can though in addition have small and short swarming/tripling/quadrupling sessions in front of 1 computer to look at especially important issues when they are worked on, not afterwards.

If you do neither code reviews nor pairing then you are in trouble.

Expanded from my own Hacker News comment https://news.ycombinator.com/item?id=6693064

One button - The aim - Nearly there

An aim for a development team (and company) is to achieve one button deploys of your applications to all environment.

By achieving that it involves several solutions which in total results several direct and indirect benefits for the company. Benefits such as:

• Quicker releases of new features
• Frequent releases of new features
• Quick and painless rollback of unwanted releases
• Reliable release process
• Confidence in testing in different environments
• Reduced risk in deployments
• Predictable roadmaps with more stable velocities
• Less context switching for ops and developers
• Developer time focused on delivering features instead of on processes

I joined as a Tech Lead at my current company last year. Since then my team, or more appropriately our fantastic Ops/Devops, Tools as well as other teams, have in a year made great strides towards One button deploys. We are more or less there and the company is rewarded more and more with the benefits mentioned above. We have achieved this by doing the following since last year:

Move to Git and internally hosted repository server

Subversion was once great but no longer. Whilst we used git-svn for a while, we eventually fully moved all applications’ source code to Git.

With the addition of first Stash then GitLab we removed a lot of bottlenecks, restrictions and frustrations. Code could be easily referenced without having to check out. Collaboration via Pull requests for code bases you may not be comfortable with. Code review when needed. Forking and creating new projects became instant. Project discovery and general code share became very very easy.

External configuration and one binary

The main applications in my team used to use Maven profiles to build separate binaries for each environment. This was because the configuration was baked in via Maven filters.

This is not great for QA sign off as you basically have different binaries. It also slows down deployment as you have to rebuild the binary all the time. It definitely makes it difficult to rollback releases or investigate older releases as you have to rebuild an older version from scratch. And it makes it very fragile as each machine might build it differently (different JDK, broken local Maven repositories, etc)

Moving to external configuration meant we had one binary that gets promoted through all environment. The one binary gets upload to the repository manager (initially Nexus then later Artifactory). This binary is then downloaded as part of every deploy job.

Changing configuration is a configuration change and no change to the binary.

Configuration in source control and rolled out with Puppet

With external configuration we enhanced reliability by adding these configurations into source control. This also meant configuration change became very easy. Push changes for the environment to Git and it was then automatically sync and rollout via Puppet. This removed a very annoying and typo prone bottleneck.

Production configuration changes are not activated automatically but are rolled out via a single command.

Process automation, Teamcityfy everything...

We aimed to automate as many processes as possible. Our continuous integration server, TeamCity, have a number of automated builds and a whole range of manual jobs.

Testing and building a binary automatically on every check in. Automatic deployment to development servers on successful builds.

• one button jobs to deploy that binary to other environments
• one button to tag a release
• one button to push release binaries to environments
• one button to create bugfix branch
• one button to rebuild databases
• one button to migrate database schemas
• one button to restart servers
• one button to test 3rd party APIs and environments
• one button to run acceptance tests
• one button to smoke test environments
• one button to trigger load tests (Gatling)

We did use Fabric directly for deploy tasks but Fabric is now triggered from within TeamCity jobs. Deployinator was nice but we phased it out in preference for all deploys in TeamCity for every environment.

Test separation

There was already an extensive JUnit suite of tests in the applications my team was responsible for. However they were a mix of integration tests masquerading as unit tests which slowed down development and feedback loops.

We relabeled unit test that required frameworks (Spring, Hibernate, etc) or databases as integration tests instead. That reduced our feedback loop time.

We added separate verification builds for just integration test. We separated and migrated 3rd party test to 3rd party libraries. We added stand alone acceptance and smoke tests using Cucumber, Specs2 and Selenium.

Test data

We wrote command line and web applications with Node.js and Spray that quickly create test data and tools that review test data. This sped up development testing and proper QA testing.

3rd party mock applications

Mocking out 3rd party systems or even some internal systems entirely by creating test harness applications in integration environments that pretend to be those systems have speeded up our QA process a lot.

Removed restrictions

Restricting who can commit, run certain jobs or see certain data might sound like a sensible option, but in reality it slows down collaboration. We decided we just trust the majority more than we distrust a minority.

A great benefit was that we opened up all Git repositories to everyone. Anyone can commit, although people less involved in a project prefer to use Pull requests.

Team City jobs can be run by anyone. There is an audit trail, but there has never been an issue of anyone ran a job they should not have. It is a great help if people are in meetings or similar that anyone can push code out to the next environment for example.

Production data for some applications due to PCI and CRB restrictions, production credentials and some production jobs for our critical applications are still restricted but we try to minimise this as well.

Feature toggles

Whilst it is a good practice to keep these toggles to a minimum, adding feature toggles that can be overridden via external configuration has been a good change. We can now quickly disable broken new features. We can dark release features or we can canary release features, then via Git & Puppet enable the feature and it becomes available for all.

Content migration tools

Instead of applying content directly and manually as SQL scripts, or scp/ftp/rsync we started writing tools in Play! to help create data sets of new content, then promote those through environment, cross checking which environment the data is in.

Adding scripts that interact with 3rd party portals was also helpful.

This avoid typos, avoids forgetting to run a script in an environment. And greatly speeds up data migration.



Database migration

Many of our applications use NoSQL solutions such as Cassandra and Redis as some of our flows have to handle millions of interactions, but the core data are still mostly in PostgreSQL.

Whilst we use Flyway and DBDeploy to migrate those database schemas and stub data to some environments, we do not use it all the way to production. This is one area we need to improve.

Environment creation

One element that is important and which our Ops team is just starting to roll out is an internal PAAS/IAAS solution. One button to create a new VM environment or one button to create a database, AWS SQS queue etc. Further enhancements, such as one button to clone an existing VM or database will be nice. 

No button

Obviously there were other enhancements that is not really related to "One button", such as phasing out old legacy Java applications with newer Scala applications by applying Strangler Application pattern, replacing Quartz based batch jobs with Akka and Camel, measuring new features effect with AB testing, monitor applications metrics with Graphite, log analysis via Logstash, etc.

Company profitability

All these combined has made certain part of our development and release process so easy and quick. I am sure we have covertly increased the profitability of our company as we now can release quicker, more frequently, with less broken releases or bugs in general.

Future

We still have a lot of work to do. As mentioned we are not quite there with DB migration and Environment creation. We are experimenting with using Vagrant locally and promoting to environments, ZooKeeper for better configuration change without downtime, etc. Eventually we may achieve most of the ideas behind Continuous Delivery.