Pair with people you like and code reviews with people you don't like

Pair with people you like and code reviews with people you don't like (misquoted from someone way smarter than me).

That sums it up very well for how I feel about both practices.

With large teams, especially if distributed or partially outsourced, code reviews can ensure code quality and are quite essential. It will allow you to share some knowledge and instill best practices. It will be a reminder for people that their code will be viewed by others so don’t take shortcuts.

However code review can also be a total bottleneck if over-bureaucratic. It will add an overhead for all work. If there are some high and mighty gate keepers that will stop you from pushing your code frequently then you have a complete velocity block. If your code is shit then fair enough, but if it is merely nit picking or just disagreements between styles then it is very costly. If however if any compulsory reviewer is not in your office, country, time zone or just very busy then that adds a large delay in the feedback loop.

Another issue with code reviews is that quite a few reviews are of low quality due to lack of context. They do not necessarily know all the discussions, history of why a piece of code works this way, project code style, etc. Especially if they are of the tainted ivory tower architect affliction.


In smaller, agile and especially collocated teams code reviews will flag issue unnecessarily late in the process. Just pair from the start instead to ensure no short cuts or dodgy code slips through, and automatically spread the knowledge. If you do not trust two of your developers combined then you do have a serious problem.

You can though in addition have small and short swarming/tripling/quadrupling sessions in front of 1 computer to look at especially important issues when they are worked on, not afterwards.

If you do neither code reviews nor pairing then you are in trouble.

Expanded from my own Hacker News comment https://news.ycombinator.com/item?id=6693064

Don't hire me

There are many reasons not to hire me

I will say no, a lot. I am pragmatic and may find a better solution by asking what it is you actually want to achieve. Or just No

I will say its crap if it is, immediately

I will have an opinion, always. Even when based on very little facts. Sorry

I will change my opinion 180 degrees quickly if a good team convinces me I am totally wrong. This happens a lot, thankfully

If obedience is important to you don't hire me

I am in no way politically correct (not an excuse to be an asshole either) and I make terrible jokes and often “too soon”

I do not have a filter for whom to sugar coat a situation to. I will say the same whether you are a team member, manager, CEO, client or shop keeper. And always blunt and honest

I will be honest with estimates. As in they are total guesswork and only worth it to gage magnitude

If you prefer polite sales type people that tell white lies or total tales then I am not your man

I don’t clock in, I’m definitely not a morning person. If you value timeliness above value then I am not your man

I don’t clock out on the dot either but I rarely work overtime and never weekends. Too risky, if the TPS reports can't wait, don't hire me

I like collaborating, pairing, even tripling for important decisions that needs swarming

I like the team taking ownership together, I detest micro managed task delegation

If you need bums on seat and each task accountable to a person then don’t hire me

I will not jump up like a lap dog when people panic, I barely got a pulse at the best of times. If you prefer knee jerk reactions don't hire me

If you want me to be an ivory tower architect, don't. I like to work with the team to find out their best suggestion is and then work with them to implement it

Or if you have another architect of an ivory tower inclination telling me how to implement something - not going to happen

If you want to keep status quo don’t hire me, I will insist on continual evolutionary improvements. Otherwise the systems will rot, become a big ball of risk and your best developers will leave

However if you want a ninja/rockstar, I am not

If you want me to work with a ninja/rockstar I rather not. Some are okay, but most are a PITA and imperceptibly costly for the company

Insist on Windows? Don’t be ridiculous. No serious developer would accept that handicap unless desperate

Shirt & tie? No, not happening. You can’t take techies dressed as used car salesmen seriously

If you prefer drones in suites, that is not me

Many of these are valid reasons not to hire me

Some I hope are reasons to hire me

Your call

@flurdy | linkedin.com/in/flurdy | www.eray.co.uk

Need to ask what each member of your team is doing? Then your team is too big!

In daily (or more frequent) stand ups that is common in agile/scrum based teams many practices the process of asking each and every attendee: what did they do yesterday, will be doing today and if there is any impediments. Many do this by default without considering alternatives.

I have previously blogged that I trust my team members and do not need to know in detail what they did yesterday. I firmly believe it is more productive to focus on the tasks on the board rather than the individual. If you do not trust every member of your team then you have bigger problems.

However task focused only works with a small “1-2 pizza” team. A small team where everyone knows what everyone is doing, where the scrum master / project manager have full understanding of what tasks is currently in progress if not already obvious from the board.

Once the team grows too large you tend to have too many tasks on the board and people are all working on different perhaps even unrelated  tasks. The other team members, and especially scrum master, cannot keep up with all individual and task statuses so you need to do the token ring quiz.

The ring quiz takes time, and tend to let people’s focus drift away from the stand up and not really listen particularly well to what the others have to say after a while (I am guilty of this frequently), especially if some tasks never involve certain other members.

Also the project comes more inclined to individual tasks and not pairing and swarming together on the same tasks. The risk is then some people can disappear between the cracks or loose focus and your project start to become less efficient and agile.

So if you find you stand ups are leaning towards individual “what-you-did what-you-will-do any-impediments” interrogations then it is a strong smell of having a team that is too big.

(“1-2 pizza” is an analogy of how many can share large pizzas together. In my case 3-5 members is a good size, 8-9 is too big, and more than that is simply not agile)

One button - The aim - Nearly there

An aim for a development team (and company) is to achieve one button deploys of your applications to all environment.

By achieving that it involves several solutions which in total results several direct and indirect benefits for the company. Benefits such as:

• Quicker releases of new features
• Frequent releases of new features
• Quick and painless rollback of unwanted releases
• Reliable release process
• Confidence in testing in different environments
• Reduced risk in deployments
• Predictable roadmaps with more stable velocities
• Less context switching for ops and developers
• Developer time focused on delivering features instead of on processes

I joined as a Tech Lead at my current company last year. Since then my team, or more appropriately our fantastic Ops/Devops, Tools as well as other teams, have in a year made great strides towards One button deploys. We are more or less there and the company is rewarded more and more with the benefits mentioned above. We have achieved this by doing the following since last year:

Move to Git and internally hosted repository server

Subversion was once great but no longer. Whilst we used git-svn for a while, we eventually fully moved all applications’ source code to Git.

With the addition of first Stash then GitLab we removed a lot of bottlenecks, restrictions and frustrations. Code could be easily referenced without having to check out. Collaboration via Pull requests for code bases you may not be comfortable with. Code review when needed. Forking and creating new projects became instant. Project discovery and general code share became very very easy.

External configuration and one binary

The main applications in my team used to use Maven profiles to build separate binaries for each environment. This was because the configuration was baked in via Maven filters.

This is not great for QA sign off as you basically have different binaries. It also slows down deployment as you have to rebuild the binary all the time. It definitely makes it difficult to rollback releases or investigate older releases as you have to rebuild an older version from scratch. And it makes it very fragile as each machine might build it differently (different JDK, broken local Maven repositories, etc)

Moving to external configuration meant we had one binary that gets promoted through all environment. The one binary gets upload to the repository manager (initially Nexus then later Artifactory). This binary is then downloaded as part of every deploy job.

Changing configuration is a configuration change and no change to the binary.

Configuration in source control and rolled out with Puppet

With external configuration we enhanced reliability by adding these configurations into source control. This also meant configuration change became very easy. Push changes for the environment to Git and it was then automatically sync and rollout via Puppet. This removed a very annoying and typo prone bottleneck.

Production configuration changes are not activated automatically but are rolled out via a single command.

Process automation, Teamcityfy everything...

We aimed to automate as many processes as possible. Our continuous integration server, TeamCity, have a number of automated builds and a whole range of manual jobs.

Testing and building a binary automatically on every check in. Automatic deployment to development servers on successful builds.

• one button jobs to deploy that binary to other environments
• one button to tag a release
• one button to push release binaries to environments
• one button to create bugfix branch
• one button to rebuild databases
• one button to migrate database schemas
• one button to restart servers
• one button to test 3rd party APIs and environments
• one button to run acceptance tests
• one button to smoke test environments
• one button to trigger load tests (Gatling)

We did use Fabric directly for deploy tasks but Fabric is now triggered from within TeamCity jobs. Deployinator was nice but we phased it out in preference for all deploys in TeamCity for every environment.

Test separation

There was already an extensive JUnit suite of tests in the applications my team was responsible for. However they were a mix of integration tests masquerading as unit tests which slowed down development and feedback loops.

We relabeled unit test that required frameworks (Spring, Hibernate, etc) or databases as integration tests instead. That reduced our feedback loop time.

We added separate verification builds for just integration test. We separated and migrated 3rd party test to 3rd party libraries. We added stand alone acceptance and smoke tests using Cucumber, Specs2 and Selenium.

Test data

We wrote command line and web applications with Node.js and Spray that quickly create test data and tools that review test data. This sped up development testing and proper QA testing.

3rd party mock applications

Mocking out 3rd party systems or even some internal systems entirely by creating test harness applications in integration environments that pretend to be those systems have speeded up our QA process a lot.

Removed restrictions

Restricting who can commit, run certain jobs or see certain data might sound like a sensible option, but in reality it slows down collaboration. We decided we just trust the majority more than we distrust a minority.

A great benefit was that we opened up all Git repositories to everyone. Anyone can commit, although people less involved in a project prefer to use Pull requests.

Team City jobs can be run by anyone. There is an audit trail, but there has never been an issue of anyone ran a job they should not have. It is a great help if people are in meetings or similar that anyone can push code out to the next environment for example.

Production data for some applications due to PCI and CRB restrictions, production credentials and some production jobs for our critical applications are still restricted but we try to minimise this as well.

Feature toggles

Whilst it is a good practice to keep these toggles to a minimum, adding feature toggles that can be overridden via external configuration has been a good change. We can now quickly disable broken new features. We can dark release features or we can canary release features, then via Git & Puppet enable the feature and it becomes available for all.

Content migration tools

Instead of applying content directly and manually as SQL scripts, or scp/ftp/rsync we started writing tools in Play! to help create data sets of new content, then promote those through environment, cross checking which environment the data is in.

Adding scripts that interact with 3rd party portals was also helpful.

This avoid typos, avoids forgetting to run a script in an environment. And greatly speeds up data migration.



Database migration

Many of our applications use NoSQL solutions such as Cassandra and Redis as some of our flows have to handle millions of interactions, but the core data are still mostly in PostgreSQL.

Whilst we use Flyway and DBDeploy to migrate those database schemas and stub data to some environments, we do not use it all the way to production. This is one area we need to improve.

Environment creation

One element that is important and which our Ops team is just starting to roll out is an internal PAAS/IAAS solution. One button to create a new VM environment or one button to create a database, AWS SQS queue etc. Further enhancements, such as one button to clone an existing VM or database will be nice. 

No button

Obviously there were other enhancements that is not really related to "One button", such as phasing out old legacy Java applications with newer Scala applications by applying Strangler Application pattern, replacing Quartz based batch jobs with Akka and Camel, measuring new features effect with AB testing, monitor applications metrics with Graphite, log analysis via Logstash, etc.

Company profitability

All these combined has made certain part of our development and release process so easy and quick. I am sure we have covertly increased the profitability of our company as we now can release quicker, more frequently, with less broken releases or bugs in general.

Future

We still have a lot of work to do. As mentioned we are not quite there with DB migration and Environment creation. We are experimenting with using Vagrant locally and promoting to environments, ZooKeeper for better configuration change without downtime, etc. Eventually we may achieve most of the ideas behind Continuous Delivery.

Do not stand up in the morning, do it at lunch, and again

The Daily Stand Up, a meeting where everyone stands up (to ensure it does not go on for too long), often referred to as the Daily Scrum by projects using Scrum methodology, is often scheduled in every morning.

And by early it is often the first thing in the morning. E.g. 9am.

There are good reasons for people picking that time slot.

* It gives people a clear idea of what they will be working on today.

* It minimises any unfocused time from starting work until the stand up.

* It ensures everyone is present and not being pulled into other meetings, leaving early etc.

However I dispute that.

Everyone may not be present.

* If you are based in a large city, various public transport delays often occur and people frequently do not make it in for the normal start time.

* Modern day work life is not like the old days of set factory shift hours. People do not need to clock in at the same time anymore. Flexible time allows people to start a 7,8,9,10 and change that every day.

* With a smaller world, companies and projects are frequently distributed across different time zones. So the work hours definitely might not match up. (Co-located teams or not discussion is for another time..)

So do you keep running the stand ups without everyone present? Or joining halfway through? Or frequently postpone it until later that day?

The time before the stand up will vary a lot.

* As people start at different times, some will make it in just before the stand up others will have been there an hour or two already.

* Members of the team might need to attend other stand ups before this stand up. E.g. Scrum-of-scrums stand ups,  if the Scrum Master/project owner is split across other teams, or worse team members are split across projects...

So do you just ignore the unfocused time before the stand up?

The task plans will not be clear for the entire day.

* Time before the stand up may be unfocused. If perceived as too small to bother trying to a worthy concentration ‘zone’ or pairing session. I.e. just waste.

* They might finish or become blocked/impeded during the day, and not pick up priority tasks

* Team members might be struggling until the next stand up

Is the unfocused wasted time costly?

So how do we improve this?

I have two suggestions. And they also have other benefits.

Have the stand up time just before lunch.

E.g at 12:45 if in the UK, or 11:15 if in Norway (my two countries of reference)

Everyone will most likely be in the office whether they started early, got stuck in traffic, had to drop off the kids at school, etc.

It definitely ensures the meeting does not overrun as people really do want to go to lunch.

It does have an issue if people go lunch at different times. Lunch times however are more easily adjusted by an hour more or less than the actual working hours. (If still different then you have another problem as I am a big believer in that most of the time team members should want to go to lunch together. Socialising at lunches and other events are great team spirit builders.)

The time before the stand up is now significant so people should not be unfocused before it. They should be able to resume their task, resume pairing, enter their zone and actually do some significant work.

With members in normal working mode they will be able to quickly explain what they have been doing since the last stand up, what the important impediments are etc. I often make it in just before morning stand ups, and rarely have a good memory of what I did the day before (which is a good thing, as it means I was able to switch off from work), but also means my status contribution is not optimal. (Whether stand up are status meeting or not is another discussion).

It also blocks people from scheduling stupid meetings that overlap lunch.

Introduce more stand ups

The daily stand up does not have to be daily. To keep focused on the flow of tasks your team can introduce multiple stand ups during the day.

However I would suggest to keep one as the official one, the one you advertise to externals, the one the project owner attends. The others should be kept as informal meet ups for the members to quickly sync up, to ensure WIP limits are maintained and focused on the most imporant tasks, if any impediment have occurred during the day or if someone is not sure what to pick up/assist with next. Much like the “proper” stand up.

You need to be careful not to introduce too much of a pressure cooker, allow some natural slack time, down time during the day otherwise your team will quickly burn up and become demotivated.

The informal stand ups does not have to be at the board, it can be at a table in the kitchen etc.

Dog food

In my current team we have our official stand up at 10am. Yes not lunch, but it was what the team wanted and you go with the team agreement. Our company does not offer flexi-time, so we all need to clock in 9.30am so with traffic/commute delays people are mostly present at 10am.

We have introduced an informal stand up between the developers and QA around 1400-ish.

And a couple of kitchen breaks during the day, however these naturally also include of general banter.

Follow up

Take a look at “Cocktail Party” of stand ups that Henrik Kniberg describes.

I dont care what you did yesterday, I trust you

A daily stand up is not a status update.

It is not a tool for micro management.

It is not a opportunity for line management, middle management,  project management and other stakeholders to scrutinise status of anything.

It certainly is not a stage to interrogate the work ethics of individuals.

A stand up is for the team,

synchronise on what will happen today,

making sure tasks at the top are moving along to done

and to assist others(swarm) to ensure WIP blockers are removed .

Stakeholders and others can watch standups, it gives them a quick overview of the priority of the team, any impediments etc but it is not for interfering or performance scrutiny.

The board should reflect status at other times. Otherwise the project owner/scrum master can inform people what the backlog priority is.

Any micro status of what happens between the backlog and done is not for external stakeholders.

TL;DR We do not care what you did yesterday. We trust you. You would not be in the team if we did not. Lets focus and help each other on today's task instead.

Heroku commands & tips

I wrote a quick howto of Heroku commands and tips I use all the time.

It covers creating instances, configurations, common add-ons such as databases & email, deployment strategies and more.

As mentioned in the document I tend to have multiple applications per project, so I always append --remote staging for example, resulting in this long deploy command:

git push staging master && \
heroku logs -t --remote staging; \
heroku open --remote staging && \
heroku logs -t --remote staging

For the eagle eyed you will have noticed the ";" which means as the log tailing never really finishes, when you notice the app is up and running you have to manually end it with control+c to proceed with opening up a browser.

Obviously these commands ties nicely into my use of Play on Heroku and the howtos I wrote on integrating both.

They are perhaps obvious commands and basic but I hope they are of use to some people.