Evolve or wither slowly

3 years ago (wow that went fast!) I wrote a blog post called Do not rewrite. The main points in it was to never rewrite whole applications. Never. Instead gradually rewrite one feature at the time whilst still delivering new business value.

I also wrote a more recent post about Paper cuts and broken windows. Which emphasises the importance of fixing problems straight away and not let them fester.

I want to extend these posts to include you should rewrite all the time, even when not immediately necessary.

Recap

Extracting the main points in the previous blogs why big rewrites are bad and continual rewrites are good, and to also sum the obvious why no writes at all are very bad.

Big rewrite disadvantages

• Cost
• No business feature - no value
• Risk
• Big deployments
• Forgotten features
• Likely to never finish
• Not fully replacing old system
• Support yet another system

Continual rewrite benefits

• Reduced risk
• Smaller delta
• Quicker feedback
• Modularising
• Faster - less bottlenecks
• Leaner
• Remove features and bloat
• Actually finishing

No rewrite disadvantages

• Death by thousand paper cuts
• Even the smallest change becomes slow and painful
• Broken windows
• People do not care if they introduce bugs or break other things
• Staff exodus
• Few wants to work with painful systems

Less obvious benefits

There are other perhaps unexpected but important benefits not raised in the previous post. 

Business domain knowledge

Every time a refactor or minor rewrite takes place the knowledge of that part is refreshed and stay current in the company. If a system is not touched for a long while the knowledge of it may be forgotten especially if staff involved has moved on. 

If a critical bug or urgent feature needs to be added to that system then the turn around is exponentially different between a recently updated system and one forgotten.

Technical knowledge

This is also true for the technical part of a system, the how, not just the what and why it does something. An old COBOL or complicated legacy custom build Java application will take a long time to work out compared to a contemporary tech stack application.  You might even have to hire new staff or expensive contractors to fix it.

With up to date knowledge it will be a lot less risk that they might not do it properly.

Quick turnaround

If a new feature needs to be added then to any system then getting people ramped up and delivering it will be much quicker if it is on a contemporary and well known tech stack. If a new system needs to be integrated with existing systems then the API integration will be smoother and a lot quicker.

Technical migration

If a mass migration of a technology stack is needed, perhaps migrating from in house data warehouse to a public cloud based provider, or moving from monolithic to horizontally auto scaled instances, then having most systems on a contemporary and probably quite similar technology stack will speed any urgent migrations. This will avoid/reduce the need to for many deep cave explorations of old customised mystery legacy applications.

Staff retention

A very important reason to rewrite applications and features and in general keep technology up to date is staff retention. (Obviously a million other reasons exists for staff churn as well).

If you avoid death by a thousand paper cuts and broken windows staff will not mind working on the products. A negative culture will be less likely.

If you migrate to newer technologies and let people frequently learn new stuff they will be much more interested in the work and less likely to want to move elsewhere. (refer to my blog post Peak interest - the learning and sharing curve).

Recruitment

If word spreads that you keep an up to date and interesting technology stack then hiring new members of staff will be a lot easier, and you will hopefully attract more qualified candidates. On boarding will also be quick and people will be up to speed quicker with newer well known technology.

Lean technology - lean organisation

If the applications are continually refactored and evolved (as long as it is not done in a hacky ninja rock star development fashion) the architectures will become modularised, scalable and leaner in general. 

Not necessarily a micro service architecture but a lean architecture that are more likely to be adaptable for the future.

Hopefully a lean architecture and highly skilled retained staff will also lead to / need a lean organisation so the business is more likely to succeed.

Rewrite when not needed

I would also suggest people and organisations refactor and rewrite when they do not see the obvious reasons for it.

I do not entirely mean rewriting applications on a whim when there is no features to be added or bugs to be fixed, after all I do emphasise the need for delivering business value along with all rewrites. 

But I do think even when the obvious pain is not there that you should try to refactor and update applications. For example if the application is just one or two version behind the latest but still seems fine then still update it. It will be less painful than when it next time is 3-4 version behind with a bigger delta of change.

Rewrite encapsulation

One type of no business value rewrite I would suggest is a good idea is to encapsulate legacy systems API as soon as you come across it.

Even if you are not changing a legacy system but merely reading data from for example, then adding a contemporary facade to it straight away will be of value in the future when you do need to update it.

Experiment and rewrite experiments

Another important staff retention technique on top generally keep up to date with technology is to let them experiment with new technology. 

I would avoid core systems, especially customer facing systems, but internal tools is prime candidates for field testing technology.

This will also lead to discovering technology that you can use in other applications if proven useful that otherwise would have been missed or delayed.

Naturally these experimental applications should also not be abandonware and be rewritten as often as other systems. Though they will probably always be good candidates for next set of technology experimentations.

Rewrite exploration and euthanasia

Certain system never comes involved in new features so will not be included in a normal rewrite. These will consciously have to be found and rewritten without business value. Though probably just as a contemporary facade initially. 

Leaving these as abandonware is not a good idea. Either kill them or update them.

Summary

The main points was already covered in my previous blogs, but I hope people see the value of rewriting frequently to especially keep staff, organisational architecture and company in general up to date for whatever happens in the future.

A utopian expectation of all applications being up to date is too much to hope for. At the other end where companies that rarely and/or minimally update their applications, they are doomed to fail. The grey scale in between decide whether companies will wither and eventually die or survive. 

Don't download the internet. Share Maven and Ivy repositories with Docker containers

Downloading the internet. A term commonly used when building Maven projects. It is due to downloading your applications dependencies and their transitive dependencies. With a normal application those dependencies can add up to quite a few jars and a lot of megabytes to download. Especially on a 'clean' machine without cached dependencies in your local repository.

Running a new container in Docker sometimes feels the same as every image layer it is built on top of also has to be downloaded. Eventually these are also cached in the Docker cache.

If you then have a Maven based application in Docker you have the perfect storm of bandwidth hogging. Especially as your Docker image's Maven configuration will by default not reuse any cached dependencies and instead always download everything from Maven Central. And for each new build or launch it will re-download the internet as it knows of no local cached dependencies.

Work around, not solution

My work around is to mount my local host repositories for Maven, and for its derivative Ivy, as data volumes for the Docker container. This then avoids re-downloading the internet on every further build and launch and will also reuse dependencies I most likely have already downloaded on the host.

It is not the best solution as your images should ideally not be influenced by what you have on your host machine, but it saves a lot of time, and a lot of grief.

Vagrant

When I run my docker containers inside a Vagrant instance I first mount my host repositories by adding these 2 lines to the Vagrantfile:

config.vm.synced_folder "/Users/myusername/.m2", "/home/vagrant/.m2"

config.vm.synced_folder "/Users/myusername/.ivy2", "/home/vagrant/.ivy2"

Boot2docker

Since Docker 1.3 the OSX /Users path has been by default shared with the boot2docker VM on the same path. So to share the maven repositories you need to link the /Users path to your docker user home folder.

   boot2docker ssh;

   ln -s /Users/mysername/.m2 /home/docker/.m2;

   ln -s /Users/mysername/.ivy2 /home/docker/.ivy2

Data volume container

For easy sharing these folders between many docker containers I mount them as a data volume container and naming it ‘maven’.

   docker run -d -P --name maven \

   -v ~/.m2:/root/.m2:rw \
   -v ~/.ivy2:/root/.ivy2:rw ubuntu

I am basing it on the basic Ubuntu image, and it will stop immediately as no process is running. That is fine, the mounted volumes will still work.

Alternatively instead of mounting the host folders you can have a persistent container with these folders exposed as VOLUME in the Dockerfile so it is shared in a similar manner.

Launch container 

   docker run -d --volumes-from maven \
   yourapplicationimage

Obviously you probably have other options on your docker launch, but the important bit here is the ‘--volumes-from maven’ which maps all the volumes from the data volume container called maven into this container.

Now in theory all containers with Maven and Ivy based build tools such as SBT should first look at the mounted Maven and Ivy repositories for their dependencies.

Build problem

Unfortunately if you build new images frequently you will have the same problem still as the above solution is only for running containers not building. During the build stage Docker does not allow you mount any volumes. This is so it is reproducible anywhere.

In your Dockerfile you can ADD or COPY whole repositories into your image, but that will make it very bloated with a lot of irrelevant dependencies unless you somehow construct and maintain a perfect repository. (Ps. don’t do this).

Maven repository manager

One solution that makes sense in general is to add you companies Maven repository manager’s public Maven Central mirror to the image. That way at least you will only download the intranet, not the internet.

For a Maven build add a settings.xml file to your image folder with at least these settings if using Nexus:

<mirrors>
   <mirror>
      <id>Nexus</id>
      <name>Nexus Public Mirror</name>
      <url>http://nexusmachine/nexus/content/groups/public</url>
      <mirrorOf>central</mirrorOf>
   </mirror>
</mirrors>

                

Then add this to your Dockerfile:

   ADD settings.xml /root/.m2/settings.xml

Note this will be overwritten if you later during the run stage mount .m2 folder on top of it, but for most that is fine as the .m2 folder usually contain a relevant settings.xml file anyway.

For a SBT build add this repositories file to your image folder with this content:

[repositories]

local 

maven-local

company-repo: http://nexusmachine/nexus/content/groups/public

scala-tools-releases

maven-central

Then add this to your Dockerfile:

   ADD repositories /root/.sbt/repositories

Repository manager container

A further solution is to run a repository manager as another container and have all Maven/SBT builds refer to it instead. This avoids the involvement of the host computer yet saves any network traffic as everything is localhost.

Pull down something like https://registry.hub.docker.com/u/mattgruter/artifactory/ .

    docker pull mattgruter/artifactory

Configure then run and name it:

    docker run -d --name artifactory \
    mattgruter/artifactory

Modify the above settings.xml and repositories to refer to you locally linked artifactory name as url instead, e.g this one liner:

   company-ivy-repo: http://artifactory/artifactory/repo/,
   [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
   company-repo: http://artifactory/artifactory/repo/

(The example above added it as an Ivy repository which Artifactory supports).

Hopefully these tips will avoid downloading the internet too often and save a few grey hairs.

Learning whilst writing, and relearn later

Martin Fowler recently made a comment that he writes books so that he can learn about a subject.

That resonated well with me, as whilst I don’t write books I do write technical howto documents on my website and my blog. And it is true I mostly do write these documents as I am learning the topic myself.

The main reason I write and share these documents is still to help others. Hopefully someone else will be able to stand on my shoulders and have an easier time learning the same thing. I certainly use a mixture of other’s docs as I learn stuff and hopefully I reference them appropriately in my docs.

Focus and relearn

But I write howtos also to help myself. By documenting each step I keep focus on learning it properly. It also encourages me to achieve certain meaningful levels before I bounce on to the next shiny thing.

An additional unintentional benefit to me is that in e.g. 6 months, 2 years or even just a few weeks later when I need to use/learn the same technology, I have ready made revision notes for me to get up to speed instantly.

Accidental expert

People mistakenly/naïvely think I am expert on the domain of each howto I write.

On some topics covered in a few older docs I may be an experienced fountain of knowledge…, but with most I only scratched the surface. Quite a few I never touched again and have more or less completely wiped any knowledge from my little mind.

So writing these docs can be a little bit of a curse. Of my top five most visited blog posts three of them are related to Hibernate from years ago. I am not an Hibernate expert nor do I want to be, but this could trick people into assuming that I am. My most visited (by a country mile) howto doc is about setting up an email server. I know a bit about the subject but it is not my job nor interest, I merely wrote a good howto in 2003.

But in general sharing is worth it. The amount of thank notes I have received and general good feeling I get from these are gold.

Evangelical

Another benefit to myself is that I now have reference points when I evangelise about a certain topic. I can refer to my own work and words instead of soon to be forgotten spoken words.

I also starting to be confident enough to present, submit CFPs and in general talk about subjects which are often built on documents and blog posts I have previously written.

Dockerise it all - containerised addiction

TL;DR: CDD - Container Driven Development 

(CDL - Container Driven Life)

Occasionally you come across new interesting technology that is easy to use, and suddenly a veil is unveiled and you become aware of so many opportunities and potential. Maybe it was the same the first time you discovered programming, or OO or FP programming, message queues or NoSQL databases, or distributed source control, or moving from IDE only to proper reproducible build tools (Ant, Maven, Gradle, SBT etc). With certain tools it becomes your favourite hammer that you carry everywhere searching for nails to hit it with.

Docker is such a revelation, at least for me. And I am hammering a wide variety of nails with it. And with the hammer-nail pattern/anti-pattern some of those nails really didn’t need hammering but it was fun.

Docker history

Docker was only revealed in March 2013 by Solomon Hykes and other people at dotCloud . It got a lot of publicity in the hacker news section of the world as people could see the possibly potential in this new tangent of virtualisation. It baked and matured through 2013 as people occasionally showed examples of applying it to their workflows, but still it was too rough for most to use.

But during 2014 especially since the summer the  publicity and widespread use of Docker has exploded. Downloads from Docker’s registry hub of ready made images has exponentially rocketed sine the summer of 2014 by 1387%! Now the majority of people I talk to or follow on twitter mention Docker and show examples of how they use it. Conferences and tech news are flooded with new ways to use Docker. And projects that extend or are based on Docker are multiplying all the time.

As someone who has always been interested in virtualisation, automation, reproducibility, build tools and provisioning, introducing containers like Docker has been a welcomed evolution and a revelation. Being already heavily invested in the use of Vagrant for development and systems the migration to/ combination with Docker has been smooth.

Docker addiction step 1. Exploration

First steps are usually to try the easy hello-world-ish examples. Downloading an Ubuntu image and launching bash inside a Docker container. Basically exploring Docker and its commands, which is what I cover in my how to install, basic use patterns and many handy tips with Docker howto.

Docker addiction step 2. Imagination

My howto document also takes you into step 2 which is more about creating your own basic images. Creating a simple Apache or Nginx container with a simple volumised website. Creating images ready with a Java SDK or node.js etc. I have made some simple Dockerfiles of these type of images available at github.com/flurdy/Dockerfiles.

Then you usually also wrap a simple application of your own in an image and running it on other computers. You start to understand how and where Docker can be used.

Docker addiction step 3. Diversity

Once you got one application inside a container you start to experiment with other applications. Might take a look at other types of applications and more likely support applications such as databases inside a container.

You might start to link containers to each other such as an application to a database container and some of the other handy Docker tips in my howto.

Docker addiction step 4. Proliferation

Now you have tried a mixture of applications with Docker you might spread the use to even more core applications and more. You start to link multiple containerised applications to each other.

Replacing third party integration and old legacy applications with a container image, wiring up and switching between different versions of an application via container linking, etc.

As you start to basically recreate environments using Docker containers you might start to facilitate promotion between staging environments only using container images. And eventually production is containerised.

You are probably already sharing and discovering images at work and on the internet. Using the public Docker registry hub and/or internally with tools such as Quay.io

Docker addiction step 5. Eccentricity

When comfortable with writing Dockerfiles and fully aware of the Docker way of layers and process then you start to experiment with more unconventional Docker images.

You might create Android SDK in an image, ready made SSH tunnels to apps or databases in another. You add tools such as a local maven repository manager, local DNS servers, mail servers, etc. into containers.  You realise desktop applications can actually be containerised.

Eventually tools to facilitate easier and more manageable Docker life such as Fig, Panamax, Shipyard, Dokku, etc. are maybe among your toolset.

Docker addiction step 6. Ubiquitous

By now all your core applications are run/runnable inside a container.

They link to databases, message queues, 3rd party adapters, and other smaller applications that are all also inside containers.

Your testing environments, staging environments, production environment, development environment and REPL are all running inside containers.

When you create a new application or touch an old legacy application your first step is to write a Dockerfile and containerise the application. You design applications and processes from the ground up to be containerised.

This is what I call CDD - Container Driven Development.

Possibly by now you are convinced everything belongs inside a container, so a better name is perhaps CDL- Container Driven Life.

Peak Interest: The learning and sharing curve

The interest curve of learning and sharing at work and its eventual peak are partly responsible for higher turnover of staff and reduced productivity if not managed well. 

When you learn and share what you know at work you enjoy it. You also contribute to a very productive environment. However when you no longer learn nor share then you are not enjoying it anymore and become negative and a liability. And then leave. 

About a year ago I read an article that explains everything in this blog post in a much clearer and scientifically researched way with references to specific companies. I haven’t been able to find it again, but when I do I will add a link and note to read it instead.


The problem

When you start a new role you have lots to learn. (Although sometimes you may have too much to learn and you are simply out of your depth and not able to keep you head above water).

At the same time you may have a lot experience and new thinking to share with the others in your team and company.

This learning and sharing is usually very enjoyable and rewarding, and you quickly become more and more productive in your role.

This productivity over time can simplistically be viewed in a graph like this:

However after a while in a role you learn less and less. You have absorbed the majority of the domain knowledge, you know all the people involved, and you are familiar with most of the technology stack (if in a similar line of work as me). 

Your team members and colleagues have already heard most of your experiences and you are starting to even repeat yourself. The rewards for sharing are reduced. So your productivity could be viewed as flattening out:

However what tends to happen is that you enjoy your role less due to not learning and not sharing as much as before, and over time you start to not like your role and become quite negative.

This in turn start to have a negative effect on your productivity over time:

This is what I call peak interest.



The consequence

With this negative productivity trend you may start to become negative person at work. This might then spread to your colleague and the whole team starts to suffer.

In the end you probably will change jobs. So the company will have to recommence the expensive recruitment process to replace you and will loose all your knowledge.

Even if you leave quickly it affects the remaining employees negatively. Naturally they might be upset if you leave on a personal level, but also upset with the company for letting you leave, jealous if you leave for a better package elsewhere, depressed that your knowledge and capacity is gone and they have to figure out how to compensate. If many employees leave quickly all the time then the company will have a bad negative culture.



Known problem

This is in fact a known problem, and whilst I can not refer to any scientific research, I have read about this issue many times. 

Major companies are fully aware of these peaks and take many actions to delay and counteract this curve for their employees.



Not applicable to all

The magnitude and curve gradient are definitely different for everyone and for every situation. And not quite as smooth. Some are quicker learners, some in a difficult role, some join a company where there is little to share, etc.

And then there are some sets of people that this hardly applies to at all. The "lifers". Not meant as a derogatory word but I fail to find a better description of this common characteristics of a set of people. 

These are the people that rarely changes jobs, that are not that interested in learning anything new, nor as keen share what they know. They just get on with their job, and stay there for a long while.

Their interest curve whilst perhaps not entirely linear are certainly very much less curved. And may never peak. However my unscientific opinion is that they also make a much less productive contribution to the company over all. 

Every company will have some lifers. There are many roles where this is a good trait: in jobs that are very 9-17 and never really change. 

However it can be frustrating if you mix lifers and “peakers” in the same team. Developers, artists, any creative or knowledge workers need to work together, learn together and share together. If some are then learning and producing at different speeds you will have conflicts.

That being said, mixing peakers with "ninjas" is a recipe for disaster as well. Ninjas and rockstars do not have an interest curve, they have an interest seismograph. But that is for a different blog post.

Solutions

Companies do try to resist and reduce the effect of these peaks. They try to stretch out the length of the peak and create further peaks.

They can ensure the work environment is nice and comfortable. That bureaucracies are not frustrating, latest tools are available and most of all avoid stressful tasks, overtime and other counter productive situations. This will make the peak last longer and post peak decline slower.

Even then the major peak will eventually happen. Companies can then create further peaks by changing team focus, changing project, changing people, changing teams or changing role type completely. Basically avoid status quo for too long.

What they can then achieve is multiple peaks that keep you at a company for much longer.

So this avoids you having to leave the company to find another enjoyable role. However I do think eventually that undulated curve will start to fade and you might leave your role in the end.

The critical bit is then though that instead of leaving after for example 1 to 2 years you might stay for 5 or 8 years. (Time being relative to personality, industry, culture, etc).



Solution faults

Even if the company is aware of interest peaks, and want to mitigate their effects they might not always be able to.

In a small company there might not be any other team or project to move to. In a big company you might be insignificant or just too many to cater for so you are not moved around as often as needed.

Due to deadlines, business restrictions etc there might not be possible to move at the appropriate times.

Different people will have different gradient of curve and peak and companies will struggle to individually predict this.

These might lead to the negative cycle setting root and you resign.


Other issues are of course that changing everything around all the time is not appropriate either. 

It may not be good to do it too often, to do it to too many people at once, to unevenly change roles for some compared to others.  

People and teams do need some sort of stability so constant change is not good either.



Personal experience

I know that I go through these interest curves, and I have especially been taking notice of mine and others lately.

Whilst I do not like to change projects too often as I like to get a thorough knowledge of the domain and project, I know I also need new challenges once I have mastered a domain, technology and/or team to a sufficient level.

When I have worked with startups the curve has always been very steep, the peaks very high, but also the peaks come quickly and the need to change status quo come sooner.

When I worked with large enterprises the gradient was less steep as there was many things that took a long while to do, the technology was less challenging and pace slower. However I also reached the peaks later. But I think the productivity magnitude was much less.

As a consultant these peaks are not so much of an issue due to the normal change of clients and assignments. Some long term clients have even been helpful by moving me to new projects with them in the same assignment although perhaps less on purpose for my benefit, more for their business reasons. 

My beautiful better half reminded my last year when I was moaning about being bored at work. She pointed out that exactly a year before I had told her that I really need to step up my game as there was so much to learn in my new role and that I was loving it. I had clearly reached and passed my peak for that role. 



Conclusion

Peak interest happens to most people and companies. It is nothing to be ashamed about. It is in fact a nice and beneficial trait which needs to be managed.

Company need to embrace interest peaks. Let people move around the company. No questions asked. Encourage it. Better than haemorrhaging good employees, or let a bad vibe fester.

Avoid status quo. For instance make sure technology stack and skills keep continuously evolving even if not always great business value. Staff will stay interested and not need to leave find new challenges.

As an individual if you start to get bored in role, quickly request a change in your role. New project, new technology, new people, new role, whatever it is that will be a nice new challenge.



Disclaimer

Of course productivity and reasons people leave jobs are very much affected by other factors. I do believe however that peak interest of learning and sharing does contribute to both enough for it to be a significant factor.

I do not have any specific scientific research (atm), just a personal opinion backed by reading over time of similar material. If you agree, disagree or have have any research to back it up then feel free to comment or link back.

Paper Cuts and Broken Windows

It is the little things that matter

We are often reminded of the facts of keeping focus, deliver a minimum viable product, lean startup principles, only invest in what returns business value and so on. Whilst all these are definitely true, but to some extent they are also false.

Delivering business value, concentrating on one task is a virtue I do insist on, however we should not abandon all other tasks and common sense. Secondly it is actually very important we do the low priority and the little ROI tasks as well. Why? Because of Paper Cuts and Broken Windows.   

Note: This may apply to other professions and aspects of life in general but I am only thinking of IT development projects.


TL;DR: Fix your bugs asap, evolve systems continually, keep code clean and take pride in it

The problem: Paper Cuts

Many little bugs that in total hurts a lot.

A paper cut is a simple bug or an outstanding minor feature that is deemed not important enough to fix/implement right now. Some may have best intentions of implementing it soon/eventually but we all know from experience that it is most likely to never happen. 

Over time these paper cuts adds up, soon everything you touch is already full of bugs, painful procedural steps or legacy code, and in the end your team and project may start to venture close to death by a thousand paper cuts or at least feel like it. 

The problem: Broken Windows

If something is already broken, it does not matter if you break something else

Similar to paper cuts, broken windows is when a system/project/code base has some or many things wrong with it so that the current maintainers no longer feel any pride in it. It is then simple to not fix things, to ignore bugs, poor code etc.

If a continuous integration server notification of a broken build or a Nagios alert occurs for that system, people are more concerned about getting rid of the notification than actually fixing the root cause.

In an application suffering from broken windows any new feature will not be done to their best possible standard or refactored for cleaner more maintainable code.  It will not be fully tested, in fact may introduce a few known bugs and limitations that will be ignored.

Also if some epics are never completely finished, it may leave an impression that you do not need to finish stories and epics properly and as such broken windows of missing features start to be an accepted practice.

Combined problem

Essential paper cuts and broken windows are about the same. Paper cuts are how much it is hurting you by slowing you down and making people resistant to do anything with it. Broken windows is about lack of pride and willpower to make an application better as it feels already partially broken.

Or worse, if your customers start to have the same paper cuts and broken windows experience then they will simply stop using the application or even your company.

Derivative problem: The costly rewrite

A system suffering from Paper Cuts & Broken Windows will eventually be so despised by the developers that they will campaign for it to be rewritten from scratch. Which leads to a long period of delivering no business value, just pure expense. Refer to my post: Do Not Rewrite. 

But the new system will eventually also quickly suffer from paper cuts and broken windows if the processes are the same.

Solutions?

There is no right and wrong way to do prevent this. Eventually it is inevitable that a system will suffer from paper cuts and broken windows. However there are several ways to minimise the risk of it happening quickly, increase morale to reduce its spread and significantly delay the rot until the system has been used for so long that there is so little business value left and that it can be shut down.

Prevent death by paper cuts

Fix bugs ASAP

You cannot fix every bug and implement every feature as you will be trying to reach the impossible perfection which would cost an unlimited amount of time and money.

However the cost (in time and therefore money) of fixing a bug increases by magnitudes the longer you leave it, due to the costs of context switch, environment setup, remember data model/logic flow, knowledge share etc. A bug fixed straight away due to a very short feedback loop is invaluable, a bug fixed the same day or next day does take a little time but not much at all. Any longer than that it does become a real exponential costly tech debt.

So try and just fix bugs straight away or at least whilst still on the same specific task. It will make life so much more comfortable. Pushing out buggy features should be deplored.

For minor tasks and bugs that are still skipped or found at a later date they will still cause paper cuts.  A thorough automated and manual QA procedure will reduce this but they will still occur. 

Fix old bugs continually

However I have found continually picking up a few of these left over bugs and tasks will slowly reduce the amount of them on the backlog and therefore help prevent too many paper cuts.

When finishing a large complex story, try to pick up 1 or 2 bugs and minor tech debts before the next big story is started. If you finish another feature an hour before lunch, meeting or end of the day then instead of half starting another story fix a quick bug instead. This continuous self healing done in what is usually mostly lost productivity time will be invaluable over time. It is a beneficial minor procrastination.

If it turns out that bug/feature is a much larger task than you have time for now then simply just add that information to your issue tracker so that at least you have done some backlog grooming that makes it future planning easier. It may then be a valid candidate for one of the few paper cuts that is never fixed, but at least now with more data.

Development principles

Another way of preventing paper cuts is obviously to never introduce them in the first place. A bug free system does not exist, but you can reduce the frequency and impact of them by applying good procedures, architecture and code style. 

Design a system as simple as possible and writing code that is clean ie KISS and SOLID principles, so that a method, class, application or system only does one thing and is very easy to understand. This significantly reduces the risk of introducing unwanted secondary effects and greatly enhances the ease of maintaining it. 

Applying functional programming techniques such as avoiding mutability of objects and removing state where it is not needed will also reduce risk of unwanted secondary effects or architecture issues when scaling a system.

Designing systems and features using TDD reduces risk of bugs, ensures test coverage and also avoids implementing unnecessary features.

Introducing continuous integration delivers a very quick feedback loop so secondary effects are found whilst the developer are still aware of the context. Avoiding time spent on unintegrated feature branches is also advisable (ie max hours, never days). 

Prevent lack of pride due to Broken Windows

Reducing paper cuts will prevent the broken windows feeling. Applying the principles mentioned above (KISS, SOLID, immutability, TDD, CI, etc) will reduce the risk of broken windows. And by insisting on the same principles of good code standards and sensible processes people will take more pride in their work.

But broken windows will happen. Due to unpreventable technology evolution, staff turnover and lack of knowledge transfer or just plain mistakes. 

How do you fix these “windows” and when?

However windows can be fixed. And with more and more windows fixed people take pride in their systems and their work again.

Embrace automation and continuous delivery. If most parts of the process of maintaining a project is automated then any barrier to take the time and energy to fix a broken window will be very low and much more likely to happen voluntarily. 

I wrote a post about a project I was on where we transformed most of what we routinely did into one button clicks on our CI server. The returned value of that investment was great, most tasks was no longer a chore, removed many bottlenecks, the risk of process typos or forgotten steps was minimised and business value feature delivery time was reduced immensely.

A good practice is to apply the same principle of picking up and fixing a broken window issue as you do with bugs, ie as soon as possible or just after finishing another story.

Another practice is to fix issues as part of story, especially if it touches the same areas. If for example a deploy often fails on a server, when you deploy to that server just fix it properly. Do not continue with your head in the sand. 

With these you will slowly mend your most of your broken windows.

Little known time sinks  

It is the little used and little known applications and features that becomes major time sinks when changes are needed. And as such is also the parts the team will avoid to fix. Try to prevent these from becoming this by always evolving architectures and killing features.

Evolve systems

Create your architecture from the start or bit by bit of an existing system into modularised projects and components. That way you can refactor and improve one small part at the time without big and long costly rewrite committal periods of no business value, as detailed in Do Not Rewrite. 

Do not leave old systems to rot with outdated technologies and ever shrinking pool of possible maintainers. Keep evolving it, especially their integration points by for example by applying Micro Service Architecture and/or Strangler Application pattern.

Trim the fat, kill features

If an existing feature is no longer really needed, then be quick to kill it. Euthanasia to unimportant or historic features will make maintenance much easier, reduce risk and speed up delivery of new business value features.

Kill entirely an old application if nearly all its original features have already been migrated away or removed.

Bandwidth

To be able to do this you need the time to do this.

You need a strong tech lead to insist on the team to fix bugs and remove tech debt continuously, and can protect his team if they are pressured to cut corners.

You need non blinkered, smart product owner and/or project manager that understands the long term value of not doing the short term features only. A PO/PM that values the velocity of feature counts only and accepts bandwidth is also used on tech debt to achieve this. The PO/PM should not micromanage tasks and tech debt, only priorities at the epic and story level.

Henrik Kniberg’s Lean from the Trenches book describes the value of feature count above any other metric, and whether you solve tech debt (or not) and how much is up to the team and no-one else.

Summary

To reiterate the TL;DR at the top: Fix your bugs asap, evolve systems continually, keep code clean and take pride in it.