Moving to XP
[Thanks to Ron Jefferies (see www.XProgramming.com)
for great feedback on the original article. Ron is a co author of Extreme
Extreme Programming (XP) is one of the Agile Development processes. In this
article I describe XP, when to use it, and offer advice about implementing XP
on both existing projects and new projects.
Agile Development and XP
XP is an agile development methodology that starting spreading in the
development community after Kent Beck’s superb book ‘Extreme Programming
Explained – Embrace Change’ was published in 2000. In Feb 2002 the
was formed, and the term ‘Agile’ was born to describe lightweight processes,
one of which is XP. Other Agile processes include SCRUM and Crystal Clear.
The Agile Manifesto For Software Development
was born. It has 4 core values:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
While there is value in the items on the right, the items on the left are
On the whole programmers tend to love XP. This is because they spend more time
on the things they enjoy, like designing and coding, than the activities they
find boring, like requirements documentation and test plans.
There are contentious debates in the market between the heavy methodology folk
and the Agile folk. XP is often accused of being nothing but glorified hacking.
I strongly believe that it isn’t. However, if you don’t practise all the Core
XPvalues then it can easily become that and you can get into quite a mess,
something I have experienced first hand. There are situations when XP is a good
choice and situations when it is wholly inappropriate.
When Is XP Appropriate?
XP is highly suited to projects where the full requirements are simply not
known up front, are very vague, and are very likely going to change. In these
sorts of situations you often need to get something out quick to the users to
get feedback early and often – one of the tenants of XP. This technique
mitigates the requirements risk. See my article
Mitigating The Requirements Risk With Agile Processes
for a more detailed discussion on this topic.
One could argue that if you don’t know the requirements then you are asking the
users the wrong questions and failing to diagnose their problem. [This often
happens in IT projects where the developers (and even the business analysts)
ask the users what they want rather than trying to diagnose the business
problem. This is a bit like doctor asking you what medication you would like
for your illness, rather than diagnosing the problem and suggesting appropriate
medication.] Anyway, the brutal truth of matter is that in many projects you
don’t know if you are heading in the right direction until you get feedback
from the first and subsequent releases. This is why iterative development is so
widespread. In XP, the iterations can be very small (2 weeks) and there is
always a working system that can be released at any point, as a result of
There is also a very compelling argument to use XP when the requirements are
also known up front. Since XP provides irreplaceable feedback on what is
happening in the project, through delivering tested running software, even if
there is no requirements risk.
It would not be appropriate to use XP in critical applications where a
waterfall development and rigorous formal testing methods would be required to
ensure the safety of the users. Another example where XP isn’t required is the
situation where the requirements are known up front, and hence no requirements
risk. An example might be porting a windows application to a web application.
Granted, you would need some further interaction design doing, but the
functionality set would remain the same.
In summary, XP is well suited to projects where the requirements are unknown,
vague, and are bound to change. The excellent book Questioning Extreme
Programming addresses many of these issues in much more detail.
I discuss how to make XP a success in
Values and Techniques Of XP
XP is based around 4 key values:
Simplicity: Build the simplest thing that will work. No future proofing code.
Communication: High quality and quantity increases understanding.
Feedback: As much as often as possible keeps the direction on course.
Courage: Confidence and courage to embrace those changes when they arrive.
Rather than designing and building a whole bunch of code to meet future
requirements, we build the simplest thing that will work for the immediate
requirements. The code must be clean, and well designed, but there must be no
future proofing. We acknowledge from our experience that on many occasions all
that future proofing code often never gets used as the project and requirements
change. On projects whose direction changes the cost of designing, building,
and then not using future proof code is far higher than just building what you
need when you need it. Keeping it simple lowers the complexity which increases
the chance of higher quality and productivity. In XP this is called YAGNI (You
Aren’t Gonna Need It).
When changes come along that require an enhanced design we use
Refactoring to make the changes, ensuring that everything we has
previously written still works. Refactoring
is the process of improving the design of existing code without changing the
functionality. The thought of ripping up existing working code can give project
mangers a concern (if it ain’t broke don’t fix it etc….). But, because we have
an entire suite of automated tests that run very quickly (<5 minutes), we
can refactor away and evolve the design ensuring that we do not break existing
functionality. To do this effectively you need good design: good abstractions
and good decoupling. Suffice to say, to be an effective XP developer you need
to be good at layered design, good at OOP, and know your patterns. If you
develop using a 2-tier procedural type style then XP really is not for you.
The tests I mentioned above are unit tests that are written before the code
using Test Driven Development (TDD)
– one of the integral parts of XP. The Test Driven side of things ensures that
what you can write is automatically testable and meets the requirements. The
outcome is that you end up with a suite of unit tests you can use as a
regression test harness. Expect to have as many lines of test code as
production code. This is by no means a bad thing. By writing tests first you
often find your initial design needs updating, before you build it, rather than
after a first cut.
When we have a whole suite of tests we then use Continuous Integration to
ensure that the build is never broken and we can release a working version of
the system. This means we have a process running that continuously gets the
latest version of the source code, builds it, and runs all the tests. For a
full description of this see Martin Fowlers excellent article
So, we write unit tests before we write code, then use those when we refactor
to evolve the design to meet the new requirements. What about communication?
Rather than write lengthy requirements documents the documentation is kept
light. XP puts more focus on working code that meets requirements than
documentation. We know things are going to change, and we know the best way of
proving if something works is to actually build it. XP also acknowledges that
the best form of communication is face to face. If the development team needs
to clarify the requirements they talk directly to the customer. XP uses the
concept of On-Site Customer. This means that a customer sits with the
development team at all times. This maximises the quality and quantity of
communication to help ensure the right thing is built.
Pair programming is also an integral part
of XP. This is a contentious part of XP. This is where each line of code is
written by two programmers sitting at a single computer. One is the ‘driver’
(who types at the keyboard) and the other is the ‘navigator’ (who watches the
driver and thinks about the bigger picture). The roles can be interchanged
during a session. Pair Programming removes the need for code reviews at a later
date, since code reviews are done as the code is being written. This removes
bugs and poor design earlier in the cycle. The technique results in higher
quality code being produced quicker. The reason Pair
Programming is so effective is due to the driver and navigator roles
each developer adopts during the session. This is explained in more detail in
the book ‘Pair Programming Illuminated’. Pair programming is not for everyone
and you have to be careful about who is paired together and why they are paired
together. Personally, if I’m with the right partner then I think Pair
Programming is superb.
In terms of design XP uses a Metaphor for each project. This helps keep
everyone on track with regard to design. It’s a way of thinking about the high
Requirements headlines are decided during the Planning Game. This is where the
requirements are written on index cards, with enough detail for the customers
to understand and for developers to put timescale estimates to. The customer
then gets to decide what will go into the next iteration.
To summarise, here is the main list of techniques for XP:
This section gives advice about how to implement XP on both existing projects
and new projects, assuming no existing experience of XP within your
My father taught me from a very young age that the best way to eat an elephant
is in small manageable pieces, and implementing XP is the same. If you tried to
do the whole thing in one go from just reading the theory then it is very
likely that you would fail. If you can employ an XP Champion who has been there
and done it, rather than just read the theory then you will get going faster.
But, it isn’t mandatory. It will take much longer without one though, like
anything you try to attempt without a guiding hand. I’ll talk about timescales
There are some dependencies in the techniques.
So, bearing in mind the above, my advice is to implement the techniques in the
The ability to release a working system at any point is enhanced with
Continuous integration relies on automated tests being in place. Refactoring a
system effectively and efficiently requires solid automated testing.
Getting a good suite of tests is enhanced by using test driven development.
To embrace the changes quickly, the ability to refactor effectively is
Evolving the design using refactoring is aided by the use of a metaphor.
The chances of higher quality code built first time are increased when using
Lightening the requirements gathering needs good commincation. i.e. an On-Site
Automated Testing / Test Driven
Refactoring / Metaphor
Planning Game / On-Site Customer
Doing Partial XP
On the road to full XP techniques get introduced piecemeal. I call this phase
partial XP. During the partial XP phase some some existing techniques will need
to be maintained whilst new ones are brought in. This is what this section is
all about. Let’s now examine the road to full implementation with each of the
Automated Testing / Test Driven Development (TDD)
If you are starting fresh on a project then you can practise TFP from the
start. You will need to learn the techniques of writing tests which is pretty
quick to do. I’d suggest using a version of xUnit which has been ported to most
languages. Kent Beck’s book ‘Test Driven Development’ is a very good primer for
the technique. My article also discusses the
practicalities of implementing TDD.
The real skill in TDD is knowing what to test. By this I mean the scope and
strength (or weakness) of the tests. In a layered system you only want to write
a test that tests the scope of the component under test, and not allow the
assertions to flow further down the architecture. If you are not careful you
can end up with duplication in the test code which will make the tests tough to
maintain, and also longer to run. This skill is something that only comes with
experience. In terms of the test time you need to get the total test time for
all tests very quick. I’d suggest an ideal of less than 2 minutes, a middle
ground of 5, and certainly no more than 10.
There are more projects in progress than there are projects about to start, so
the question remains as to how to introduce TDD onto an existing project.
The first step is learning how to write automated tests. My article ‘Writing
Automated Tests Using NUnit and VB.NET’ will show you how to do this.
Next you need to decide what to test. Start by writing tests for any code that
is new, being enhanced, or being fixed. Over time you will then end up with
tests for the whole code base. To be able to write tests you need to ensure
that your system has sufficient levels of abstraction. Using Model View
Controller is a good place to start. Your functions need to be small with a
single responsibility. If you are suffering from large classes with large
methods (see Blob Antipattern) then you’ll need to split things out (see
Extract Method in Refactoring as a starter).
It is possible to write automated tests for the UI of your application which
I’ve explained in ‘Automating Browser based
tests with IE and NUnit’. You can also do the same for Windows based
apps. However, they tend to take a while to run (>20 minutes), and are
pretty useless as a tool for regression testing during development where
typically you want to run the full suite over 10-20 times a day. They are great
however for automating acceptance tests, and particularly good for testing
previously raised bugs. The great thing about writing tests for previously
raised bugs is that you can guarantee that once a bug is fixed once it will
never raise it’s ugly head again. As an example, on my current project we have
just under 400 unit tests that run in 85 seconds, and 116 front end IE tests
that run in 25 minutes. We have only really just started. On my previous
project I had around 2500 unit tests.
The process of bolting down your existing application with automated tests will
help you learn more about the art of what and when to test. With that
experience and know-how you can then begin to write tests before you write
code. And this is when you make the serious leap into the high quality zone. To
judge when to make that leap watch out for your own personal ‘Moment of Test
First Revelation’. During a design session with other developers you will be
discussing how to implement a certain piece of functionality and someone will
suggest a solution. When you pitch in and say ‘Yes, but how will we test it…’
that is your Moment of Test Driven Revelation. You’ve made the leap and are
ready to start full TFP.
The productivity gains you get from TFP are huge on their own which you can
still reap regardless of whether you are doing full XP or not. And trust me,
once you have made that leap, you will never want to develop any other way
again. I discuss other ways to maximise
productivity in a seperate article.
Refactoring / Metaphor
So, you now have the application bolted down with tests and are starting to
write some tests before you write code. You now have some new requirements to
implement which require some refactoring of existing code.
If you are working alone then you can probably do without the Metaphor concept,
since you will have a good one in your head anyway. The Metaphor concept is for
teams of developers as a useful aid to communication. You need to make sure
that everyone knows what the high level design of the system is. Before you
start refactoring in teams you need to choose a Metaphor. Personally, I’ve not
got stuck into the whole Metaphor thing. I use them all the time when
describing architecture but tend to choose different ones for different
purposes. Instead I tend to use a small example application for an
architectural spike that the developers can refer to, in conjunction with a
good high level class diagram and supporting sequence diagram for the front to
back architecture. This gets supplemented with a document – albeit a very light
There are also different sizes of Refactoring that will need to be done. The
small ones are things like Extract Method, to make the code more manageable or
remove duplication. A larger refactoring might involve putting in a security
pattern for access control. For the larger refactorings I strongly recommend
doing some whiteboard design work before you start. I like to call this Small
Model Up Front. Also, before implementing any new functionality I strongly
advise doing some Small Model Up Front. One of the reasons XP is labelled as
hacky is due to the fact that people think it does not involve design.
[Warning: If you don’t practise refactoring you are hacking. The code
base will rot and turn into a time bomb that is waiting to go off. I know this
from personal experience where a bad decision was made to put off a 2 week
refactoring job to hit the deadline. Further functionality was built on top of
rotting code which resulted in 8 weeks worth of extra effort down the line.
Never put off refactorings. Do them as you need them otherwise they build up
and blow up in your face. It is something that must be done continuously rather
than as separate task on the plan.]
So, to summarise, when refactoring, ensure the team knows the architectural
pattern, and for the larger refactorings or new functionality practise some
Small Model Up Front.
At this point you aren’t doing Pair
Programming yet. So, you should still be doing code reviews instead.
So, when should you start doing Pair Programming? You can actually do this
whenever you want, even as the first thing if you wish. It is a great tool for
teaching other developers. But, as I stated at the start, the best way to eat
an elephant is in small manageable chunks. The most effective mouthful in my
opinion to start on is Automated Testing. Pair Programming is lower on the
Before deciding to do Pair Programming I strongly recommend reading ‘Pair
Programming Illuminated’ so you understand the techniques. Deciding who to pair
together depends on what you are trying to achieve. Also, each developer in a
pair needs to understand the roles and responsibilities of driver and navigator
to make it work most effectively.
So, when should you start Pair Programming? If you have an XP champion on
board, then introduce this right at the start as a means to teaching. When you
have the team up and running with Test Driven Development, it is then a good
time to start pair programming to increase the code quality and turf out those
code reviews. Code reviews are traditionally much too late in the cycle and are
often left out.
When you have all the application bolted down with tests you are doing what
I would call ‘discontinuous Integration’. This is when the developer runs the
tests before and after the coding to ensure they do not break the build before
checking their work back in. Bear in mind that developers are not perfect and
accidents to happen where the build gets broken. This situation can slow down
the development pace whilst all developers down tools to get it working again.
To know immediately when the build gets broken helps the developers identify
where in the code the break was introduced.
Earlier on I talked about automating the system tests and the amount of time
they took to run. Continuous integration resolves this issue, since a
developers time is not taken away whilst their machine is tied up running tests
that take a long time.
There are tools on the market to help with continuous integration. Some are
Draco.Net, Ant, Nant, and Cruise Control by ThoughtWorks. On my current project
we wrote something ourselves to do this, which took about a week. This was
before Cruise Control.Net came along which would have done the same thing for
us. It’s not that tough to write your own. The advantage of writing it yourself
is that you can add lots of project specific detail to it. For example:
updating version numbers in your bug tracking system, emailing people, or
moving code to drop zones. Saying that, I understand Cruise Control.Net
provides a framework for plugging your own pieces in. I’d recommend talking a
look at it. The ThoughtWorks folks know a lot about Agile Development and from
a first glance their tool looks very well thought out.
When should you start attacking Continuous Integration? My advice is to wait
until you’ve got a good automated test driven approach running with a full suite
of tests, both unit test and system tests. Before trying to automate it, start
manually doing a build at the end of each day and running those tests. Doing a
daily build and running the tests is also a good way of getting a crude measure
of your project velocity. i.e how many new tests are added to the system.
Remember that at this point you aren’t writing any code until you’ve written
the tests. When you find you get a couple of weeks worth of green lights at the
end of every night you’ll know your okay, and it is worth automating the
integration. This saves the team time at the end of each day doing a build, and
also means you can build and test many times a day.
Planning Game / On-Site Customer
Once you have a good development process up and running you’ll be in a
situation where you can react quickly, and effectively to requirements changes
without allowing the code base to rot. It’s at that point you can go for
lighter requirements documents and an On-Site Customer. In some projects you
will find it difficult to get an On-Site Customer. If you can’t it is still
very important to get as much access to the primary users as possible. The more
face to face conversational access to users you can get the more understanding
you will get of their business problems. The more understanding you get, the
better you will be at being able to suggest good uses of technology to solve
their problems, and the more change you will get at getting the requirements
right and building the right thing. If you cannot get as much access as you
want then you’ll need to go heavier on the requirements beforehand, perhaps
using tried and tested techniques like prototyping, and big heavy requirements
So, use the planning game when your house is in order and you’re well placed to
react well to changes in requirements. Try to get as close as possible to an
On-Site Customer in order to lighten the amount of up front requirements
How Long Will It Take
You can start tomorrow and start getting immediate results. It will just
get better and better as time goes on. The first step is getting Automated
Testing and Test Driven Development in. Although you can show someone in a week
how to do this using a tool like xUnit, it takes quite while before you get to
the point where you get very good at it. The first time I implemented this on a
team I was learning the technique myself from scratch with no mentor. It took
about 3 weeks to start getting some real value, and about 6 months to get the
whole team seriously well oiled in just Automated Testing. It then took about 1
day to get started on Test Driven Development, and a further 3-4 months to get
really good at it. The benefits of implementing this technique alone were huge.
If you do not have an XP Champion to guide you then expect to take double the
amount of time than if you had one.
You can practise refactoring from the start without having automated tests. It
is just far riskier and takes much longer, due to the length of time it takes
to manually regression test. I’d advise you bolt the system down with automated
tests before you attack any large refactorings.
Once you’ve got Test Driven Development
in you’ve tackled the biggest part. After that I’d suggest 6 weeks to get
Continuous Integration bedded in. You can do Continuous Integration from the
start if you wish, but this will rely on a good development process in place so
that the build is not broken that often. Otherwise, you can spend ages sorting
that out, when the real issue is actually a process one.
In terms of iteration planning. If you can do this from the start then great.
This is again a process issue. If it takes 1 week to integrate and release,
then there is not much point having a 2 week iteration. Once you've got this
licked though, you can iterations of a few hours if you really want. I find one
month is pretty good, with 2 days at the end for tidy up and preparation for
deployment. Deployment then takes about 30 minutes.
Use these estimates as a very rough guide. The time it will take depends on the
size of the project, and whether the design lends itself to easy testability.
If you are thinking about embracing XP on your project then I hope you
found this article useful. XP is both an extremely productive way of working
and also a very enjoyable one. If you wish to email me any comments or
questions based on this please do. I’d also like hear how you get on.
Good luck. Happy developing.