Capgemini Software Engineering

The bigger picture and the smaller details

2019-04-18T00:00:00+01:00

Many people, myself included, say that sprints are the wrong metaphor for software development. When we think about the need to maintain a steady, sustainable pace, and the importance of avoiding burnout, the idea of the sprint can give us the wrong idea. But maybe it makes more sense if we think about how real sprints work, and about all of Scrum’s ceremonies, not just the cargo cult of the stand-up.

When Usain Bolt finishes a race, he doesn’t just start again straight away. He does his victory pose, smiles for the cameras, does some warming down, and has a few days rest. When it’s time for the next race, he spends time preparing himself, both mentally and physically, for the challenge ahead. He does everything he can to ensure that he’s in the best possible state before his next sprint starts, so that he’s able to function at maximum capacity to achieve a good result.

As software development teams, all too often we ignore the importance of these activities between sprints. In particular, the retrospective is frequently undervalued. It’s easy for managers to see the retrospective as a time-consuming navel-gazing session, where the team doesn’t deliver any business value or burn any story points. If the team is under pressure to deliver faster, it’s probably the first thing that gets cancelled.

In our software engineering team manifesto, we laid out some points for how a team should operate, including the idea that we “focus on the details, and on the bigger picture”. Maintaining a healthy balance between these two ways of seeing can be a challenge, and it’s one of the things that people often find difficult when moving from an individual contributor role to take on leadership responsibilities. As developers we’re used to getting deep down into the details of an issue, but as leaders, shaping the direction of a product, we need to take more of an overview. If you’re a technical lead it can be very difficult to switch between these two contexts, especially if you don’t take time to reflect. If the team is working flat-out, it’s easy to fall into the trap of focusing solely on the small details - things like burndown rates or technical implementation challenges - at the expense of considering the bigger picture of the project’s direction of travel. It’s important to schedule time for both modes of thinking, and retrospectives are an ideal opportunity for thinking about the bigger picture.

Another analogy sometimes used for software development is climbing a mountain. When you’re climbing uphill, you have to keep looking at your feet, watching out for uneven ground to make sure you don’t trip up. It’s only when you stop and look around that you get a chance to enjoy the view, and understand the whole point of the endeavour. If you keep plodding on without taking a break, it’s very easy to get disheartened.

That’s what retrospectives are for: a chance to take stock of where we are and how we got there. They offer a moment to pause, take a breath and think about where we’re going next, and whether we’re going the right way. They’re an important opportunity to celebrate what we’ve achieved, and recognise the hard work we’ve put in.

So let’s remember to take a break from time to time, appreciate our colleagues, understand our achievements and our challenges, and think about where we go from here.

The bigger picture and the smaller details was originally published by Capgemini at Capgemini Software Engineering on April 18, 2019.

Project Management Methodology – Probably a necessary evil for large Agile IT Projects

2019-01-24T00:00:00+00:00

For me, who has been practicing the Agile in the IT industry for much of my career, earlier project management methodologies appeared as a big evil monster which try to control everything. After recently being certified as a PRINCE2 Practitioner, I felt like sharing current understanding and experience in a light read with the engineering community which may benefit large enterprise level Agile IT projects.

When looking into PRINCE2, the very first challenge was that it all appeared like geared towards Waterfall and a lot focused on control, for example its principle about Manage by Exception. The other challenge was, some really confusing terms like Configuration Item which can easily be termed as an artefact, basically anything (document, software etc which requires a version and status). Another example of confusing term was Product. In PRINCE2, Product can be of two types i.e. Management or Specialist. In essence Management Products are really documents created to support project like Project Plan and Specialist Products are parts of the final Product and would be handed over to the customer like software etc, though it can be documents as well.

Initially, there was a time when the dominant thought was that PRINCE2 may not be suitable for an Agile IT project. However, after much experience and analysing its potential application to some previous experiences, a light bulb was lit up that a tailored version (Tailor to the Environment) of PRINCE2 may be a necessary evil for the large enterprise Agile IT projects.

Let’s start with some of my experiences about what a typical large Agile IT projects lacks, for example lacking a coherent structure for the whole of the project. It is not to say that projects have no structure at all but more to say that it misses the coherent element and an end to end view of the project. Following are some specific areas:

i. When talking about enterprise level IT projects, Agile does not really discuss all the roles and responsibilities needed for a project. While Scrum is a good approach to structure a delivery team, it discusses less about large enterprise project team. For example, if the Scrum Master is not really responsible for the delivery then who is really responsible for the delivery of the artefact? Who to go to when there is an issue?

ii. On some projects, teams appear a bit distracted and appear disjointed towards the main product and end up spending time on less valuable artefacts or trying to fix a non-existent problem. Sometimes it leads to duplication of efforts or other times creating some unnecessary artefacts which are considered reusable, but no one really reuses them. In some cases, it leads to release nightmare as dependencies are less understood at an earlier stage. A number of companies and projects try to use the mechanism of Scrum of Scrums or something similar, but does it really work?

iii. While inherently Agile is supposed to deliver software as needed by the business but occasionally on long-running Agile projects, the technical delivery may be very good, but business may appear less interested in using it. One of the examples is of a large public sector project which was delivered on time and budget but never went live.

iv. Agile also seems to miss some other cool bits related to project management. For example, there is a principle about reflecting at regular intervals to become more effective which is typically done as retrospective ceremony but there is little or no attention to the lessons from previous projects and how learning can be applied to the future projects.

v. Documentation and reporting progress are also challenging on an Agile project. Agile is heavily influenced with the XP-Extreme Programming. I am an advocate of XP but probably not at the cost of the project or post project life cycle of the application. No documentation may appear very compelling to some during the project but may not look so compelling when looking at the end to end life cycle of the product. There can be situations where you have just code with a sea of fragmented wiki pages. For example, on one project we just had compiled Java code with even no source. Did it really help the next project or was it used more as a kind of mechanism to lock the client so no one else knows how it works and let the client suffer should they decide to go with a different provider?

PRINCE2 seems to appear as a good methodology to manage above mentioned gaps in on a large Agile IT project. For example, it provides a solid structure for the end to end life cycle of a project which helps not only the management team, but it can also help the development and delivery teams to see the relationship with other components and see the non-technical bigger picture. I think PRINCE2 can nicely fill above mentioned gaps:

i. Define Roles and Responsibilities and Manage by Exception principles and related concepts offers well-defined roles and responsibilities which can help to structure project teams relatively better. For example, for a large Agile IT project, a Scrum Master can facilitate a delivery team and a Project Manager can act as a Team Manager. It would allow, the team to focus on the development, a Scrum Master to facilitate them, a Project Manager to take the responsibility of delivery and an Executive to take the responsibility of the overall Business Case. It would put less strain on the Project Manager to facilitate technical aspects as one may not be technical at all. On the other hand, less strain on the Scrum Master as one does not have to be responsible for the delivery but purely facilitation (A true Scrum Master which I have rarely seen in the practical life). This also nicely address the challenge of whom to go to when there is a problem. Please note these are roles and not the person. One person may fill different roles, though some roles cannot be combined.

ii. The Focus on the Product principle and related concepts like Project Product Description, Product Breakdown Structure, Product Description and Product Flow allows keeping team focused on what is needed. It also allows seeing how these teams contribute to the overall Business Case for the project. Even an Executive can see the need of any team and how long it is needed for from a very high level. There is absolutely no harm in using Jira or similar tools to create the same structure but in this instance links are clearer, priorities are more organized, and team has more structured view of the end Product and its link to the Business Case. By all means, it does not suggest that you should describe all details up front before beginning of the project as it would just kill Agile, however you have an earlier structure in place and a Product Description can be matured at even the Work Package level. This also helps to map Product for the release cycle and see what is needed for what stage.

iii. Continued Business Justification and Manage by Stages principles can really help with the projects like large public sector project mentioned above which never went live. While teams Focus on Product and delivery, at a higher level management keep evaluating the business justification at each stage. Should a Product fail to justify its value to the business at any stage, it is relatively easier to identify associated risk and allocated people and resources. It helps management to justify its continuity or premature termination. In other words, Products are only continued when it makes sense to continue not only at the micro (team level) but also at the macro (corporate management level).

iv. Learn from Experience is a core principle of PRINCE2. It mandates and encourages to not only actively capture lesson in the Lesson Log and reflect or refer them in the End of Project Report but also apply lessons from previous projects at early stages of the project during the Starting Up a Project process.

v. PRINCE2 allows mature structure around mandatory documentations like Risk Register, Quality Register, Check Point Reports and in some cases provides a template to start with. It would also help to sort out pain of management reporting. The structure of Project Product Description, Product Breakdown Structure, Product Description and Product Flow, their related Work Package structure and stages allows management to assess the project progress more rigorously. As mentioned earlier, there is absolutely no harm to use tools like Jira for such structure. It is more about structure and reporting rather than what tool you use. Jira can be configured in a way which produces the statistics and reports needed by the management like Check Point Reports. It would allow Project Manager and teams to focus on the delivery rather than wasting much precious time on reporting.

In summary, I see project structure is a necessary evil for the large enterprise Agile IT projects. PRINCE2 appears as a good methodology to provide such structure, though it should be tailored (Tailor to the Environment) to suit the project as needed while maintaining the integrity using its 7 core principles.

Project Management Methodology – Probably a necessary evil for large Agile IT Projects was originally published by Capgemini at Capgemini Software Engineering on January 24, 2019.

It's OK Not to be Agile

2018-10-22T00:00:00+01:00

I love messaging platforms. Slack has made a real difference to the way we can work with remote teams and keep our disparate engineers sharing thoughts and mindsets. We’re also experimenting with Teams, which is proving great, but all these tools sure do make it easy to become distracted from the day job!

This week’s distraction at Capgemini Engineering HQ revolves around a number of new takes on Agile which are being posted to explain why “Agile does not Scale”, and how this or that new and exciting version of Agile can change things. The big beef for me with these articles is that they do not define the problem domain before launching into a solution. So let’s start with the problem!

What is Agile?

I know we do this a lot, but no harm in going back to basics and redefining what we mean by Agile first of all. At Capgemini, when we say “Agile”, we refer to ways of working and methodologies including Scrum, XP, Kanban, DSDM, Lean, possibly including DevOps - which are facilitators to us adhering to the Agile Manifesto. These ways of working enable our customers to better understand how technology can help them, and, in a timely way, produce and improve useful business applications.

What Doesn’t Scale?

Autonomy

Drilling down to what it means to be Agile according to the Manifesto, a big part of this is to enable a team to be autonomous, so that they can deliver software iterations faster. These iterations can then be evaluated and improved. Why faster? Well, they don’t need to wait for some other team / person to approve their decisions. So much time is spent waiting for various forms of signoff; if you can get the decision-maker in your agile team you can move forward so much quicker. This is great for small business units or small companies, where you can easily identify this decision-maker and get their commitment to a project. It is, however, something which does not scale easily. Imagine if the project is something along the lines of: “bring new technology and modern ways of working to the justice system”. This spins out across multiple projects, sites, funding streams, technologies, and may or may not involve software development - how on earth can you maintain autonomy in that kind of environment? And, in fact, should you even be trying? It seems to me that in some scenarios, talking about the project as Agile is really not the right approach. This doesn’t mean in any way that I’m changing my stance as an Agile Evangelist, but more that we should use it in the right place. Agile is a software development methodology; if you’re not developing software - ie you’re not that far along the digital reform path, or you’re installing a custom software package which needs configuring rather than developing, then maybe an agile methodology is not right for you. And redefining your approach as “new Agile” whilst moving away from the Agile manifesto is certainly not helpful.

Dependencies

There are a couple of project types to discuss here:

Integration projects

This is an area I’ve been working in for the last couple of years. Agile doesn’t really provide benefit here for very interesting reasons - the development phase is really tiny, basically a few minutes linking bits of “pipe” together if you’re using a product such as MuleSoft, IBM API Connect, or even Camel (especially the exciting new Camel K). With these platforms, it can take significantly longer to write your Jira task than to actually complete the development phase!

Secondly, there is the fact that an integration project has at least two extra parties (if not more) involved, the source (API caller(s)) and the target(s). These parties are very often different companies, or at least very separate divisions within a company. So again, the decision making process is more complex than the build process and a specific stakeholder / product owner is hard to identify. What if an API caller wants a certain field to be present in their response, which does exist in the target system but is only provided in a convoluted way? Or if it exists in another system? Does the integration layer accommodate this change? Does the target system change for it? Who decides? Who provides the test data / test cases? This multi-party uncertainty often results in the formation of a RACI matrix - cue Agile purists berating the death of their creativity, possibly rightly so - for here, what benefit does the integration team have in being Agile? We are always going to need at least two product owners (source, target) and then of course some kind of mediator when they inevitably disagree… and hence we are unlikely to be able to have all three decision-makers in our sprint team.

Our recent project’s take on Agile for Integration involved, again, going back to the manifesto, and taking the pieces that really helped. In this case, “Individuals and Interactions over Processes and Tools”, “Customer Collaboration over Contract Negotiation”. Despite being split-site and multi-party, we strove to build up a relationship across the integration by getting people together and talking. We were open and honest about the speed at which we could develop on our integration platform, we were open to building disposable pieces. We didn’t wait for requirements documents to be signed off before we built an integration, we were iterative; we built a best-effort flow and then we tested it with the end systems to see whether it provided what they needed. We had standups and also scrum-of-scrum style meetings. This flushed out issues much quicker than if we’d have worked in a more waterfall pattern and forced our parties to try and fully document their API requirements. So yes, we had a RACI matrix and no, we weren’t autonomous; no, I wouldn’t call it Agile, but I feel we were still being true to our roots!

Company-wide Software Installations

Here is a second scenario where Agile is misused, and one with which you are probably familiar:

Programme director: Where is the project plan?

Project manager: We don’t have one, we’re Agile.

Programme director: What is this team doing?

Project manager: They’re configuring the Employee Benefits system.

Programme director: And when will that be live?

Project manager: When we get the data feed from the new payroll system.

Programme director: Well, when do the employees start using the new payroll system?

Project manager: We’ll train them two weeks before the system goes live

Programme director: And when is that?

Project manager: I can’t tell you, as we only plan 2 weeks ahead

The point here is that in large-scale technology programmes, in particular ERP system installations or similar, there is tight coupling between disparate workstreams. The teams can only be autonomous to a certain extent, and can only move so far, before they are blocked by other workstreams. These dependencies are built in to the software package being installed, and the development team have little flexibility to adjust them. Without a long-term project plan forecasting and mapping interdependencies, (which implies rigid requirements), this kind of project can’t succeed. Frameworks like SAFe are designed to give programmes the courage to call themselves Agile, to use an Agile methodology; but how far from the manifesto are they moving? Too far to call themselves Agile? Probably.

What Do We Do About It?

Our take on Agile at Scale within Capgemini Engineering is manifold. We endorse and promote SAFe, partly as a way to reassure our clients that (a) Agile at Scale is possible and (b) a lot of scholarly thinking went into creating the SAFe framework. When we are on the ground, there are a couple of proven foundations that we would encourage. We love the Scrum of Scrums whereby a daily standup, containing a member of each sprint team, is held to communicate important updates. Then there are some architectural paradigms which really help:

Domain Driven Design

We have long been advocates of DDD and it really does help to allow autonomy at a sprint team level if you have put in the effort to identify and reduce dependencies earlier on. How does DDD help? Well, by encouraging a team to define bounded contexts, to get the development and architecture teams closely involved with the business so that they get a proper understanding of the business domains, and getting a focus on continuous integration so that conflicts are identified and resolved constantly - rather than in a big, painful bang at irregular intervals. The focus on defining a Ubiquitous Language; a common, business-wide dictionary of terms, is also essential for cross-communication when developing at scale.

Microservices

A microservice architecture also helps to grant autonomy to the teams building those service(s). It’s a great way to divide up development ito pieces that are, as our blogger Gayathri puts it beautifully, “people-sized”. Using interfaces as the contracts between microservices, and understanding who owns the interface definition (clue: it’s not the implementor) can help get the right product owner, able to make the right decisions, in place to autonomise the microservice dev teams. And microservices should also by definition be independently deployable, so you won’t tread on any toes at a platform/environment level either.

DevOps Tooling

We always advocate an early focus on code deployment mechanisms when working at scale. For a start, it’s often where big bottlenecks can be found. You can streamline deployment down to minutes, but if you have to wait 2 months for a heavy CAB process to grind out a release date, is there any point?! This is often something that gets overlooked by companies who wish to be agile and work at scale, so we point it out as early as possible. And again, it’s fine if you wish to stick to your CAB schedule, but don’t label your project as agile.

Having a standard continuous delivery process across a large-scale project can really help disparate teams to undestand how each other works. A single Jira instance can easily scale, although it may well require a dedicated and skilled resource to make sure it remains legible! And it’s OK to have multiple source control systems, repositories, dashboards and build automation tools, as long as they follow the same basic rules (branching techniques, scheduling i.e. “build on commit”, number and purpose of environments, red dashboard = bad etc).

In Conclusion

It’s easy to struggle to work in an agile way when working at scale. And it’s great to find ways to get large-scale projects moving forwards successfully, but we need to be careful with our use of language and terms. If “Agile” simply comes to mean “Software Development with Sticky Notes”, we will start to lose the understanding of why we wanted to change the way we develop software in the first place. So, it’s fine not to be agile, it’s great to shout about your ways of working, but just don’t feel the need to say that you are, don’t use the term.

Agile is a recognition that, in software development, things will change. We won’t get it right first time because nobody knows what “right” is at the start, and we need to develop ways of working to embrace and absorb that change by granting development teams the freedom to decide how to cope with it. Any references to “fully define”, “controlled schedule” cannot by definition be agile ways of working. Be wary!

It's OK Not to be Agile was originally published by Capgemini at Capgemini Software Engineering on October 22, 2018.

Devoxx UK 2017 Giveaway!

2017-03-31T00:00:00+01:00

Enjoy a free ticket to one of the most sought after developer’s conference of the year, Devoxx UK 2017, courtesy of Capgemini Engineering.

Following a hugely successful 2016 which saw us as Gold level sponsors of Devoxx UK and delivered two brilliant sessions on Harnessing Domain Driven Design for Distributed Systems and NFR Testing and Tuning - A Scientific Approach, we’ve decided to do it again. But this time we’re platinum baby!

Capgemini is set to return to Devoxx UK for a very special conference experience. @Gayathri will be speaking again, this time discussing Distributing Memory in Distributed Systems, and exploring the potential pitfalls and solutions to distributing memory across distributed systems.

Where?

Taking place at The Business Design Centre on the 11th and 12th of May 2017, your only way (apart from buying a late bird ticket) into the ultimate developer’s conference is to win a ticket!

With a ticket you will be able to wonder into sessions such as Microservices for the Masses with Spring Boot, JHipster, and JWT, The Anatomy of Java Vulnerabilities, The reactive landscape, and The DevOps Superpattern. It’s an immersive, not to be missed, multi-disciplinary experience inspired by developers for developers.

Get me in Now:

Our major theme this year is role models. So we would like to know who your role model is and why. Here is an example:

Happy #IWD2017. Every woman who breaks through the stereo type and proves #yeswecan is #MyRoleModel. My mum, dad, hubs and my boss #heforshe.
— Gayathri (@gaythu_rajan) March 8, 2017

All you have to do to bag yourself a ticket is

Follow the @CapgeminiUKppl handle on twitter.
Tweet us @CapgeminiUKppl who your role model is and way using the hashtag #BeYourOwnRoleModel
Not on Twitter? You can always send an email with your full name to technologyconferences.uk@capgemini.com with the subject “Devoxx UK 2017” and tell us who your role model is and why.
Yep, it’s that easy!

You can find out more about Devoxx and take a peek at the vast range of content being delivered here. Final tickets are currently on sale at £510 (inc. VAT). But if you fancy saving the cash and bagging yourself a FREE ticket, enter our competition and tweet us by ~~23:59 on the 1st~~ 09:00 on the 8th of May 2017.
Just keep your eyes peeled in case you win and we get in touch… Good luck!

Terms and conditions:

Capgemini reserves the right to amend the competition end date at any time.
If you are the lucky winner we will notify you via your twitter handle or the e-mail address supplied shortly after the competition end date.
The prize will not be transferable to another person.
No part of a prize is exchangeable for cash or any other prize.
Travel and accommodation is not included.
Competition ends ~~23:59 on the 1st of~~ 09:00 on the 8th of May 2017.

Join us?

We’re involved in tons of conferences / meet-ups and our people attended, speak and help organise them too (Women of Silicon Roundabout, Lead Developer Conference, ContainerSched, FATJIL –just to show off a bit). We’re really keen on developing and expanding the knowledge of our people, it’s part of our culture and it’s who we are. If your interested in joining us our eyes are always open for the best talent on the block.

Devoxx UK 2017 Giveaway! was originally published by Capgemini at Capgemini Software Engineering on March 31, 2017.

INVEST in User Stories

2016-08-12T00:00:00+01:00

An INVEST-able User Story evolves through the journey of a Sprint. Let us follow this journey through the eyes of an Agile Team Member.

This post is directed at Team Members in companies planning on adopting an Agile methodology, especially for those coming from a background of a traditional Waterfall model.

After having implemented Agile in a couple of companies that adapted from a traditional Waterfall model to an Agile Scrum approach as well as initiating the methodology from scratch, I hope the following helps you to channel your time and master the art of investing in User Stories. To elucidate this further, I would like to take inspiration from a real-life scenario and track the evolution of a User Story during the journey of a Sprint and present it from the perspective of an Agile Team Member.

The website of Business XYZ processes huge amounts of payments daily through their checkout process. The business user from the Marketing department requires an urgent “One-click Payment” feature to be implemented within the check-out process that would immensely simplify the experience of the customers to complete their order faster, thereby increasing customer delight and reducing the overall cart abandonment rate.

User summarises, “We need to urgently integrate a ‘One-click Payment’ facility within our Checkout process!”

A User Story must aim to have the following facets:

User Story Template
Acceptance Criteria
Conversation
Sub-tasks

1. User Story Template

A User Story is a short and simple description of a feature (the “what”) told from the perspective of the person who desires the new capability (the “who”), usually the customer of the system (hereinafter referred to as the “customer”) [Reference: User Stories and User Story Examples by Mike Cohn].

If the feature is an internal requirement, the User Story must be told from the perspective of the area of the business that is responsible for managing the efficiency and effectiveness of the customer-facing business of the system (hereinafter referred to as the “user” or “business user”).

Additionally, a User Story must clearly list the direct “benefit” or “value” that the customer or the user will enjoy (the “why”), once the User Story is successfully delivered.

Finally and most importantly, a well-written User Story must always answer the question: “Is the User Story INVEST-able?”

Independent

Negotiable

Valuable

Estimable

Small

Testable

This INVEST mnemonic for agile software projects was created by Bill Wake in his original article, INVEST in Good Stories, and SMART Tasks.

Based on my experience, with tools like JIRA available to keep track and record details for every User Story, the more time you spend in getting your user stories INVEST-able, the better your understanding of the requirements become and the lesser time you spend in the actual coding and testing of those stories. It avoids the team going off course and acts as a guide to help steer the team towards getting their User Story “done” or to take appropriate corrective measures early on, in the journey of the team’s Sprint.

Example template of a User Story:

As a… <”CUSTOMER” or “USER” who directly benefits from the successful delivery of this User Story>

I want to… <perform a “FEATURE” specified in the User Story>

So that… <”BENEFIT” / “VALUE” that the customer or user will enjoy on the successful delivery of this User Story>

Let us write the aforementioned user requirement, following the template of a User Story:

As an XYZ customer ordering using the website, I want to be able to use the One-click Payment option, so that I can skip the payment pages and complete my order faster

Is the above User Story INVEST-able? The simple answer: NO!

This User Story leaves a number of questions unanswered:

Type of Customers: Is this feature available to all new and existing customers?
New Customers: What should be the workflow for a new customer after the One-click Payment option has gone live?
First Visit of Existing Customers without saved card details: What should be the workflow of an existing customer who has not saved any card details and visits for the first time after the One-click Payment option has gone live?
First Visit of Existing Customers with saved card details: What should be the workflow of an existing customer who has already saved their card details and visits for the first time after the One-click Payment option has gone live?
Existing Customers not opted for One-click Payment: What should be the workflow of an existing customer who has not opted to register for the One-click Payment in their previous visit after the feature has gone live?
Existing One-click Payment Customers: What should be the workflow of an existing customer who has successfully registered their card for the One-click Payment in their previous visit?

Let us assume that the Existing One-click Payment Customers use-case received the below answers from the user:

Customers must be presented with a “Buy Now with 1-click” button on the Checkout Screen (where the Checkout Screen is any section that can act as the start of a checkout process).
Clicking on the button must skip the Payment screens and the customer must be presented with the Order Confirmation screen directly.

Based on the above answers, let us rewrite the User Story:

As an EXISTING ONE-CLICK PAYMENT XYZ CUSTOMER ordering from the website, I want to be able to CLICK ON THE ONE-CLICK PAYMENT OPTION so that I can skip the payment pages and complete my order faster

The above User Story is Independent as this feature can be developed and delivered independently after the First Visit of Existing Customers use cases have been done.

The above User Story is Negotiable as this feature can incorporate changes if required.

Let us look at this scenario in our case study:

As always, the business user comes to our Product Owner (PO) once the Sprint has commenced and says, “Whoops! I just got the numbers in from my market research teams and it highlights the fact that 55% of our customers are using their mobiles and tablets (30% of those are Apple device users). Since we are already adding the feature for our web users, could we increase the scope of the delivery of this feature just a teeny-weeny bit so that this is available not only to our web but also to our Apple and Android tablet and mobile apps users as a priority? Later maybe we could make this facility available to our Windows app users since that’s not much of a priority as of right now!”

In such a scenario, is the above User Story Negotiable? Yes! I would negotiate the User Story by breaking it down further as below:

As an EXISTING ONE-CLICK PAYMENT XYZ CUSTOMER ordering from the WEB using a DESKTOP OR LAPTOP, I want to be able to CLICK ON THE ONE-CLICK PAYMENT OPTION so that I can skip the payment pages and complete my order faster
As an EXISTING ONE-CLICK PAYMENT XYZ CUSTOMER ordering from the WEB using an APPLE TABLET OR MOBILE, I want to be able to CLICK ON THE ONE-CLICK PAYMENT OPTION so that I can skip the payment pages and complete my order faster
As an EXISTING ONE-CLICK PAYMENT XYZ CUSTOMER ordering from the WEB using an ANDROID TABLET OR MOBILE, I want to be able to CLICK ON THE ONE-CLICK PAYMENT OPTION so that I can skip the payment pages and complete my order faster
As an EXISTING ONE-CLICK PAYMENT XYZ CUSTOMER ordering from the MOBILE APP using an APPLE TABLET OR MOBILE, I want to be able to CLICK ON THE ONE-CLICK PAYMENT OPTION so that I can skip the payment pages and complete my order faster
As an EXISTING ONE-CLICK PAYMENT XYZ CUSTOMER ordering from the MOBILE APP using an ANDROID TABLET OR MOBILE, I want to be able to CLICK ON THE ONE-CLICK PAYMENT OPTION so that I can skip the payment pages and complete my order faster

The priority of which deliverables need to be included in the current Sprint and which can be excluded can then be negotiated with the business user by the PO as a “Minimum Viable Product” (MVP).

The above User Story is obviously Valuable as the value / benefit has been clearly stated.

Each of the above User Story is easily Estimable as we have now broken it down to an independent, negotiable, valuable, small and testable MVP.

The User Story is Small enough to be “done” within a Sprint.

The User Story is Testable as it is small enough to be developed and tested within a Sprint.

2. Acceptance Criteria

I would use this section of a User Story in conjunction with the PO and a team member who has specialised in Quality Assurance (QA) and / or a team member with a QA mindset. I have seen the quality of deliverables increase immensely, when all Acceptance Criteria is written in a combination of Given-When-Then and checklist formats.

A Behaviour-Driven Development (BDD) approach follows a “Given-When-Then” format to help break the behaviour of the system down to an agreed and specific flow. According to behaviourdriven.org, BDD relies on the use of a very specific (and small) vocabulary to minimise miscommunication and to ensure that everyone – the business, developers, testers, analysts and managers – are not only on the same page but using the same words.

Hence, using the BDD approach ensures that everyone involved with the User Story has a consistent understanding of what is expected to be delivered of the new functionality that would be built from tests that satisfy the Acceptance Criteria.

Use a Given-When-Then format for a one-to-one mapping of acceptance tests that must be met and tested against.

Given: An existing scenario

When: I perform some action

Then: I expect a specific result

Use a checklist format for a scenario where an acceptance test satisfies multiple Acceptance Criteria checklist items.

After discussions with the Team Members, our PO has now negotiated with the business user that in this Sprint, the team would commit to deliver the web version of the User Story.

So let us write Acceptance Criteria specifically for those user stories.

As an EXISTING ONE-CLICK PAYMENT XYZ CUSTOMER ordering from the WEB using a DESKTOP or a LAPTOP, I want to be able to CLICK ON THE ONE-CLICK PAYMENT OPTION so that I can skip the payment pages and complete my order faster

Acceptance Criteria:

Given: An EXISTING ONE-CLICK PAYMENT WEB XYZ Customer, When: The Customer orders through the website and clicks on Checkout using a desktop or a laptop, Then: The Customer must be presented with a One-click Payment button
- Checklists: This test must pass when tested on:
  - Approved versions of IE browsers
  - Approved versions of Chrome browsers
  - Approved versions of Firefox browsers
  - Approved versions of Safari browsers
Given: An EXISTING ONE-CLICK PAYMENT WEB XYZ Customer, When: The Customer clicks on the One-click Payment button using a desktop or a laptop, Then: The Customer must directly be presented the Order Confirmation screen
- Checklists: This test must pass on:
  - Approved versions of IE browsers
  - Approved versions of Chrome browsers
  - Approved versions of Firefox browsers
  - Approved versions of Safari browsers

As an EXISTING ONE-CLICK PAYMENT XYZ CUSTOMER ordering from the WEB using an APPLE TABLET OR MOBILE, I want to be able to CLICK ON THE ONE-CLICK PAYMENT OPTION so that I can skip the payment pages and complete my order faster

Acceptance Criteria:

Given: An EXISTING ONE-CLICK PAYMENT WEB XYZ Customer, When: The Customer orders through the website and clicks on Checkout using an Apple device, Then: The Customer must be presented with a One-click Payment button
- Checklists: This test must pass on:
  - Approved list of Apple mobiles
  - Approved list of Apple tablets
Given: An EXISTING ONE-CLICK PAYMENT WEB XYZ Customer, When: The Customer clicks on the One-click Payment button using an Apple device, Then: The Customer must directly be presented the Order Confirmation screen
- Checklists: This test must pass on:
  - Approved list of Apple mobiles
  - Approved list of Apple tablets

As an EXISTING ONE-CLICK PAYMENT XYZ CUSTOMER ordering from the WEB using an ANDROID TABLET OR MOBILE, I want to be able to CLICK ON THE ONE-CLICK PAYMENT OPTION so that I can skip the payment pages and complete my order faster

Acceptance Criteria:

Given: An EXISTING ONE-CLICK PAYMENT WEB XYZ Customer, When: The Customer orders through the website and clicks on Checkout using an Android device, Then: The Customer must be presented with a One-click Payment button
- Checklists: This test must pass when tested on:
  - Approved list of Android mobiles
  - Approved list of Android tablets
Given: An EXISTING ONE-CLICK PAYMENT WEB XYZ Customer, When: The Customer clicks on the One-click Payment button using an Android device, Then: The Customer must directly be presented the Order Confirmation screen
- Checklists: This test must pass when tested on:
  - Approved list of Android mobiles
  - Approved list of Android tablets

3. Conversation

This section of the User Story summarises all important and relevant email discussions, code discussions, visual representations / diagrams, screen grabs, test results, et al. that have occurred during the uncovering of the User Story and recorded either as comments and / or attachments.

I use this section to record a summary of what discussions and actions have occurred on this User Story during its journey through the Sprint.

Continuing on with our case study, after the delivery of the User Story in the Sprint, it is time for the team’s Retrospective. Let us refer to the “Conversation” of the User Story to record the team’s Retrospective points:

Conversation:

Created this User Story as a part of a change agreed by the PO with the business user after the Sprint had commenced. Team had to spend a time-boxed amount of time in re-planning and re-estimating the story. Work on this story commenced 3 days into the Sprint and delivery has been agreed to the Web platforms only.

Why invest precious time on a Conversation?

To decipher the actual “complexity” of the User Story
To empower the team or any other team working on a similar User Story to make better and more accurate estimations
To save any time that might be spent on potential rework
To act as documentation on the feature delivered
To act as a tool to take corrective measures mid-Sprint rather than render the story as not done at the end of the Sprint

4. Sub-tasks

One has the capability of adding sub-tasks to one’s User Story but this can easily be misused unfortunately. From my perspective, the smaller a User Story, the easier it is for the story to be INVEST-able, thereby rendering sub-tasks as unimportant in most cases! I have only used sub-tasks as a mere logical set of one-liner reminders of tasks that need to be completed for the User Story to be marked as “done”. Usually, I try and avoid sub-tasks as much as possible as it masks the complexity of the task listed and managing them within an already INVEST-able User Story ends up becoming an additional overhead. Moreover, since they are not monitored as closely as a User Story, sub-tasks tend to become complex and quite as often, root to a cascade of problems in the following sprints. I have encountered sub-tasks that should have been user stories in itself (if not “Epics”).

Consequences:

Noticeable long and stagnant parallel lines in the team’s Sprint Burn-up / Burn-down charts
Same user stories carried over across several Sprints
Frustration amongst the Team Members due to failure of meeting commitments Sprint after Sprint
Team losing credibility amongst stakeholders
Nervousness to commit
Unable to calculate the Velocity of the team

Continuing on with our case study, no sub-tasks are required for this story as all potential tasks are already well-covered in the other sections of the story.

User Story AntiPattern: Horror Story

Can you see why the below are “Horror” (user) stories?

“Finish coding for microservice”
“Deploy code to QA”
“We need 3 additional codes to be added to the Shipping module”
“This is an easy JSP change - Integrate a new option in an existing dropdown”

Conclusion

Successfully adopting Agile requires implementing a change in the organisation’s “culture” and this is only as successful as the overall collaborative effort of every member in the organisation investing into this change. Composing INVEST-able user stories form a strong foundation towards paving the way for a successful journey. Implementing this change and Bill Wake’s INVEST mnemonic will help remind you of the mandatory characteristics required for writing good user stories.

Are YOUR user stories INVEST-able?

INVEST in User Stories was originally published by Capgemini at Capgemini Software Engineering on August 12, 2016.

Win Tickets to Devoxx UK 2016!

2016-05-26T00:00:00+01:00

Capgemini UK is proud to announce another fantastic giveaway for the dev community… Tickets to Devoxx UK 2016 worth £474!

We recently announced that we are sponsoring Devoxx UK 2016, and so we are giving away all-inclusive 2-day passes to the conference on the 8th and 9th of June. This includes your choice of conference talks, access to all video content produced, and a chance to get insights into ground breaking new technology and tools that will inevitably make you a better developer!

What is Devoxx?

Devoxx is the leading European non-profit developers conference focusing on the full spectrum of open source concerns. Showcasing the best in technology and talent from the local development community. Its an unrivalled experience, in which developers from across the tech ecosystem come together for content, networking and inspiration. The conference provides a snapshot on future trends and technical innovation with 2000+ attendees immersed in the knowledge of over 50 tech talks and Meetups covering a series of topics including Architecture & Security, Cloud, Containers & Infrastructure and Mobile, IoT & Embedded.

You can find out more about Devoxx and take a peek at the vast range of content being delivered here.

How to win a ticket:

Enter the contest by sending your full name to technologyconferences.uk@capgemini.com with the subject “Devoxx UK 2016”
Winners will be selected at random.

Deadline:

The entry deadline is 11am Wednesday June 1st 2016 and the winners will be announced shortly thereafter.

What you need to know:

No purchase necessary to win.
One entry per email.
If you’ve already purchased a ticket, no worries - if you win, you can give your spare ticket to someone you really like.
Winners will receive a tickets to attend Devoxx UK 2016 on the 8th and 9th June at the Business Design Centre in London.
We will contact the winners by email, so be sure to use your real email address!

Good luck!

Win Tickets to Devoxx UK 2016! was originally published by Capgemini at Capgemini Software Engineering on May 26, 2016.

We're Sponsoring Devoxx

2016-05-09T00:00:00+01:00

We’re Sponsoring Devoxx! (London 8-10 June)

Following on from our sponsorship of Voxxed Days Bristol 2016 and Women of Silicon Roundabout, we’re continuing to put our money where our mouth is and sponsoring yet another major UK tech conference. In the UK JVM team, we care about open source, so much so that we’re taking loads of our team to Devoxx UK; and it’s only one month away!

Devoxx UK is the London flavour of the largest Java Community conference series. It features world class speakers keynoting in such topics as cutting edge JVM development, cloud based systems, and which JavaScript framework might last until Devoxx 2017. This year Devoxx UK is on the 8th to 10th of June, hosted at the Business Design Centre in Islington, London. Over the course of the three days, attendees can choose from five parallel conference sessions, the Hackergarten, open-table Birds of a Feather sessions and rapid lunchtime talks. Oh, and on the Friday there are in-depth Hands-On Labs.

On top of all this however, it’s perhaps the impromptu conversations which are the best part of these events. We’re hoping to have loads of them. The crowd at Devoxx is always incredibly diverse – diverse in background and diverse in opinion - and that is its strength. We know that software is built by people, for people, and by widening the circle of influences and influencers, everyone benefits.

Consequently we, the Capgemini JVM team will be running a trade stand, manned with some of our cheeriest 1x developers. We’ll be showing off some of our latest work (we’re always keen to share what we’ve been up to), answering questions such as “what’s it like to work for the largest SI no-one has ever heard of?” but also listening to you and what you have to share with us. Rumour has it we’ll even be giving out free pick and mix to anyone taking the time out to come visit! (Bring your own breadcrumbs if you want to find your way back to the talks.)

In addition to the stall, a number of our team members are also taking part as speakers. Gayathri Thiyagarajan and Andrew Harmel-Law will be presenting “Harnessing Domain Driven Design for Distributed Systems”. Kevin Rudland and Andrew (again) will be sharing their experiences on “Performance Testing Distributed Systems for the Masses – A Scientific Approach”. I think I’m beginning to spot a trend…

Personally, this is my first major UK conference. I’m really looking forward to catching some of the keynotes in person, rather than on YouTube. I’m also relishing the chance to catch up with friends and colleagues. In Capgemini, we work in distributed teams (mostly) throughout the UK, I’m looking forward to putting names to slack aliases, but also meeting loads of new people throughout the three days.

Don’t forget we’re hiring (see the ad below this post). Here are just a few of the perks:

Surrounded by a growing set of talented and inspiring individuals.
Flexible working time and environment.
Two weeks of training and conference attendance a year. We love sending people to SkillsMatter’s CodeNode in London.
The best tools - we let developers choose Windows, OSX (and in the future Ubuntu – we’re working hard on it) for their corporate machine. We provide tricked out MacBook Pros to our developers if you want one. We also provide IntelliJ IDEA licences for everyone who’s broken away from Eclipse.

We're Sponsoring Devoxx was originally published by Capgemini at Capgemini Software Engineering on May 09, 2016.

Musings on estimation

2016-05-06T00:00:00+01:00

Thinking about writing a blog post about estimating but not sure how long it will take.
— Tom Phethean (@tsphethean) February 25, 2016

I knew when I posted that tweet that I wanted to write a blog post about estimating. After all, I’ve written on the topic before, but wasn’t sure exactly what form the post would take or what message I really wanted to get across. What I did know was that estimation is an important theme, and that I could create some value for both myself and others by putting my thoughts on a page.

Blast from the past

Seeking inspiration, I looked back at a sample of past blog posts I’ve written, measuring word count of posts and looking at time from first commit to when the pull request to publish was opened:

Blog post	Number of Words	Days taken to write	Words/Day
Reflections of Symfony Live London	860	7	122
Keeping Drupal Secure	1303	7	186
Drupal Integration Patterns	1409	25	56
The Lead Developer Conference	1129	2	564
New Year, New Career	230	1	230
Average:	986	8.4	231

Come back in a week

Looking at the data, it appears I can write a couple of hundred words a day. I am likely to turn out a blog post in just over a week. In other words, if you asked me to write a blog post, I could think about blog posts I’ve written before and tell you I’ll take about a week to do it. I’ll mostly meet expectations, but will disappoint you 20% of the time.

Simple, right? If you come back in a week’s time, there should be a fully formed blog post ready for your reading pleasure.

Well… not exactly. There’s some pretty big variances in the details of even that small data set. Let’s explore.

Productivity factors

The data includes one post, written in a day, which was very short and was heavily influenced by a press release which we’d released through another channel. This meant that the amount of thought that had to go into the content was limited because someone else had already done the hard work for me. I could just tweak it a little bit for my intended audience and away we go. Excluding that post lifts the averages to over 10 days for a post, and raises the average word count to about 1200.

This means I’m reducing my average effort by about 20% if I incorporate re-use of work that’s done before.

Let’s look at the other end of the scale. My post on Drupal Integration Patterns was longest in both word count and duration to write. That’s because it was a topic I was wrestling with, and I wasn’t sure how I wanted to get the message across. In writing it, I was referring to a lot of other content and inputs. It was also (relatively) original thought. My perspective on something. I needed to stop and consider what I really did think about the topic, which obviously adds to the effort. It was also written at a time I had a lot of other things going on. Excluding this post drops my days per post to about 5, without substantially altering the average word count.

This means I’m roughly 60% less efficient if I’m not focused on the task, or if I’m creating something I’ve not done before.

Even looking at two posts that should be very similar, “Reflections of Symfony Live London” and “The Lead Developer Conference”, both about experiences at conferences, we can see quite significant differences in productivity. I’ll be honest:

I have no idea why one took longer than the other - or why I was 460% more efficient writing one post than the other.

Writing is only half the job

Of course, the other factor in all of this is that I’m only estimating the things that I’m in control of. I can control the topic I’m writing about. I can decide when it’s done. I’m responsible for organising my time and focusing on getting it done.

What I’m not looking at is the uncontrollable factors. Our publishing process is to open a pull request on our blog GitHub repository. Other authors then review the posts, make suggestions, ask for clarifications, highlight grammatical errors or places the content doesn’t flow. We go through this cycle of review and amend until the post is in a publishable state, agreed by at least two of the reviewers. This keeps quality high (and shout out time - they all do an amazing job) but does mean that the review cycle can vary from a couple of hours to… longer… depending on who is around and available to provide feedback.

Then we have to wait for a publishing spot. We try not to overload our readers with too many posts in a short space of time, and will often have a small queue of posts ready to go just to give us a buffer in case we have a period of low inspiration (touch wood, not so far!). This can mean there’s a few items ahead waiting to be published. For time critical posts, we might jump the queue - if something is related to a specific event or hot topic.

Finally, there’s the act of publishing. Fortunately, we use Jekyll and GitHub Pages, which means our publishing process is very simple. Carry out a final preview of the branch locally, merge the pull request and wait a minute or so whilst the magic happens. However, this isn’t a zero time activity, and we have a smaller number of people with permissions to publish new posts, so this is another factor that could influence how long the new blog post is ready.

So, back to the original question - did you want to know how long it would take to get my post in draft form, or how long it would take to be published?

What’s your point?

I imagine a number of people will be wondering what I’m waffling on about. Fortunately, I do have a point. Everything we have looked at so far has direct parallels to the question we all get on a regular basis - “How long is this piece of work likely to take?”. Hopefully, we can draw some lessons from this:

“did you want to know how long it would take to get my post in draft form, or how long it would take to be published?”

In agile practices, this would be your definition of done. What exactly are you being asked to estimate, and what will it look like when it is done? The time to produce some working code in your local development environment can be vastly different to the time taken to get through code review, testing cycles, user acceptance and deployment. Make sure you know what it is you’re being asked to estimate and what the person asking is expecting.

“I’m only estimating the things that I’m in control of”

We talked about cycles outside your control - like getting peers to look at your code, arranging users to test it. Even once these are in place, you’re losing control of the duration. At best you can time box the activity, but you can’t guarantee what is going to be found. Testing might uncover one bug or one hundred, just like I’ve had blog posts published with an “all OK” as often as I’ve written terribly and had loads of comments to address.

In reality, your best bet for predictability is keeping items small and automating as much as possible. Why wait for someone to point out ten spelling mistakes when you could run a spell-checker? In the same way, automate your code checks - whether through unit tests, code style checks or test environment deployments. The smaller the item under inspection, the less likely something will crop up that will delay things, but if it does then the queue of other things can carry on past it in a steady flow.

The other factor is that developers tend to only estimate for the bit that they think of as “actual dev work” and forget about including time for meetings, documentation, testing, and all the general bits and pieces around the project. The pieces that tend to turn “it’ll take 2 hours” into a full day task, or longer. All of these are forms of waste, all of which can extend the duration of our work, and influence the time to get to “done”.

Remember to think in terms of both effort (or time spent on the activity) and elapsed time that it’s likely to take to complete that effort.

I have no idea why one took longer than the other

We saw, in the blog post examples, evidence of being quicker at things where “prior art” already exists, or things we’ve done before. We also saw that doing something for the first time takes longer. Finally, we observed that sometimes there is no explaining why some things take longer than others even when they appear similar. We’ve all had that bug that crops up in the middle of something routine that sends us down a rabbit hole and expends valuable time.

In software engineering, we often come across all three scenarios and think we can get a feel for how long things can take. However, don’t get too comfortable with this, as focusing on the high level “it’s just another blog post” can quickly make us over confident. Focus on the details, the nuances of this particular scenario, to make sure you challenge yourself to think about how long it will take.

Past performance is not a guarantee of future success.

Finally, we see that looking back at past projects to feed your estimates can be misleading. It only takes one hugely successful project (or one truly disastrous one) to skew your numbers. The numbers alone also don’t take into account the circumstances around them. Did the project take place over a holiday season? Were the team working on multiple things at once? Was the problem domain completely new to the team?

Every blog post, like every project, every feature, every block of code is different. There are similarities, and opportunities for us to learn from what we’ve done before, but making sure we focus on what we’re doing now can we build an accurate picture of what we need to do, and create an appropriate estimate.

Of course, that leads on to the question of whether we should even bother to estimate at all, but I’ll leave that for another time…

Musings on estimation was originally published by Capgemini at Capgemini Software Engineering on May 06, 2016.

Understanding and Addressing our Waste

2016-03-04T00:00:00+00:00

How often do you find yourself having to wait for an event to happen so you can progress what you are trying to do?

Waiting for your application to build?
Waiting for someone to code review so you can commit?
Waiting for a new deployment in the test environment so you can test new functionality?
Awaiting a response from a client?
People cancelling attendance to workshops so you need to reschedule?

These are all types of waste, why should we suffer from them?

Understanding our Waste

What I mean when talking about waste is the time we are taken away from being fully focused on a task. When we give full attention to a task we start to get in to the flow, being focused becomes difficult if for example you are constantly being pulled into lots of meetings or your computer keeps crashing and your waiting for it to catch up.

Now I’m not saying this is the case all of the time but normally there are areas of waste in most projects in some way, shape or form. Off the top of my head some examples of waste:

A problem that is out of our hands (Even though we might have a solution):

The client desktops we need to use are slow and crash frequently, inhibiting our ability to work efficiently.
Restrictions on what tools we can use to communicate with our team (this problem is more prevalent in a distributed team)

A problem that may be within our control:

It takes 4 hours to deploy to an environment so testers can start testing newer functionality (which means either you stop all testers for 4 hours during the working day or make one wait until the next day to test), possible solutions are;
- Set up a CI environment with a streamlined deployment process
- Analyse why your deployment takes so long and start working on a fix for it.
Stand ups are too long or not focused enough, possible solutions are;
- Have the right number of people who are relevant to the stand up
- Ensure people turn up on time
- Try to keep it time boxed so each speaker must be succinct
- Take things offline

These are just a few possible areas of waste on a project, each project is different and will have its own set of challenges and external influences. However for the most part there is always something we can do to improve our projects, thus improving our working day, by reducing waste. People who are familiar with Lean may have heard of the 7 types of waste or Muda (if you haven’t heard of it before, here are a few links for reading: the 7 types of waste, Muda, Seven Wastes Software. I’m keen not to cover it here as I believe it’s deserving of a separate post.)

How do we address our Waste?

Working out what the Waste is

Firstly before we start trying to address our waste we need to work out what exactly it is, for the most part we all love a good moan, so it’s not difficult to get the ball rolling in this area. The important part is to have a team environment where we have a willingness to change and adapt our processes (be ‘Agile’) and create a safe collaborative environment so people aren’t scared to suggest improvements to process.

Trying to work out what our waste is can be a struggle if we don’t have the right focus; luckily most projects that we work on these days tend to be some flavour of Agile. So we have the ability as a team to ask ourselves in a retrospective, what has been inhibiting our productivity or causing us to be frustrated during our work day.

Ultimately it is the project team who know what the areas of waste are; this is due to the fact the team has to face it, day in day out. We tend to know what the solution is but we are a little hazy on how we go about implementing it, hence why it’s beneficial having the time in retrospectives to discuss.

How do we deal with it?

Talking about waste can kick start the thought process on how we could deal with it. Writing down our issues is good practice, as we start to get visibility of them, it’s important to make sure we commit to actually addressing them. There is no point in having a wiki page with 100 issues on and just adding to them each retrospective without tackling any of them.

A key part of looking at a solution is to ensure the action we decide to take, is achievable and measureable. By getting feedback quickly and frequently we can make several small adjustments (but that is also dependent on what the problem is). We need to set a timescale in which we can say if our experiment has been a success, we need more time to judge or we need a different solution (I believe one 2 week iteration is an ideal timeframe). We don’t always have to fix something in one iteration either, it may take a few to mitigate an issue. That is because dealing with waste can be the team adopting a slightly different way of working and that doesn’t happen overnight. We should look to always look to break down the solution we come up with into small, achievable and measureable tasks.

Just because we deal with some of our waste it does not mean we will all start working at 100% all of the time every day, we still need that level of slack in a team where people are not being overworked. An analogy I once read was “you wouldn’t want your CPU working at 100% so why would you want your team doing that?”

When thinking about an area of waste we need to run it through a thought process, see below for an example workflow:

1.) What is the “real” problem?

Identifying what the problem actually is before we think about how to solve it
This is probably the most difficult part of the process as a lot of the time we tend to see symptoms of a problem and not the root cause.

2.) Is the problem within our remit/interest to fix?

There are times when our areas of waste are caused by external influences. Our first port of call should be to attempt to resolve things by speaking to the right people about this issue, however communication channels aren’t always in place to allow these discussions to happen. There will also be situations in which our hands are tied, so we instead capture metrics and attempt to create a well reasoned case.
It is normally in our interest to fix these problems, but the timescale in which they are done can vary. Sometimes it could be better to leave and monitor an issue over a longer period of time, due to the fact it can expose wider issues, again this is normally caused by external influences.
If we can fix a problem and it adds value to the team we should always try to do it, provided it’s not to the detriment of a successful delivery or cost too much.

3.) What is the driver for doing this?

What value will we get out doing this, why do we want to do it? This is perhaps the most important question. We always strive to ‘add value’ in terms of technical stories, it’s the same for improving the way in which we work.

4.) How feasible is this to tackle?

What is the solution to this problem?
- Cost versus benefit (If cost is too high, alternate workflow?)
- Have we got the people to do it?

5.) Is there any risk to implementing the solution?

We need to understand the effects of what we do, if fixing one thing breaks several others is it worthwhile? For instance, if we are updating a class that effects the way a web page works, have we got a comprehensive set of both unit and automated regression tests we can run, to ensure we have not caused any unwanted side effects?

It’s also a good idea but not always practical to run a root-cause analysis on why these problems have occurred as well. (If you are unfamiliar: 5 whys wiki, 5 whys)

Scenario:

An application takes 20 minutes to build and deploy locally and you need to do this to see any change you make (even if it’s a HTML change - bit of a dramatisation as there are normally workarounds).

What is the “real” problem?

The problem may feel like a number of things e.g. a redesign of our system is needed (this may be our root cause), the symptom we see is that when a developers makes a change, it takes 20 minutes to see if it renders as expected on the UI. However the real problem here is that the developer keeps getting disrupted, so they are likely to get frustrated if this happens constantly as it’s delaying them completing a task.

Is the problem within our remit/interest to fix?

Yes, both within in our remit and interest to fix.

What is the driver for doing this?

Reduce the turnaround time for developers to be able to complete the coding for a story and allow developers to get into a flow for a longer period of time, without constant breaks, whilst waiting to redeploy. By reducing this turnaround time it means our testers can get started quicker, meaning we get closer to delivering value quicker.

How feasible is this to tackle?

A solution is to upgrade a core part of the system. However the cost of doing this whilst implementing business critical stories is just not feasible, despite the time we could save. So an alternative solution is to use a tool like JRebel which reloads changes instantly. Whilst this doesn’t actually fix the problem with the application, it does mitigate needing to redeploy for every change. However there are some changes that will require a redeploy and rebuild but this won’t be as frequent. The cost here is a need for licenses and a slight learning curve on how to use the tool, but the benefit outweighs the cost of a license. Also the cost of a license is less than the cost of upgrading the system. We only need one person to understand the tool and then transfer knowledge to the rest of the team.

Is there any risk to implementing the solution?

Little risk, only a bit of a learning curve and knowledge sharing is required for how to use it.

How do we measure the success of our solution?

So, once we’ve worked out what our waste is and we’ve started to address it, the next step is to track our progress. It’s important we capture metrics on this. There are a few ways in which this can be done:

An opinion poll in the stand up ‘How are you feeling about issue X’ and get a rating off each person - this can be a little to repetitive though.
Capture it in the retrospective say, ‘Since our last retrospective has issue X become less of an issue?’ and dedicate sometime in the retrospective to for this issue.

Too often we get bogged down into the “what went well, what didn’t go so well, what can be done better” style of retrospective, I feel like it isn’t always the best format as it is difficult to get the valuable feedback we need from it. Especially when we have a focus on improving our productivity.

It’s also a good idea to keep sight of the issue, even if we feel it has been mitigated, as sometimes it’s easy to fall back into old habits unless we remind ourselves why we made a change. We need to be able to define a state of done for improving our process and addressing the waste. This can be done in a similar fashion to how we create a Definition of Done for our software tasks; it just requires us to think about what our desired outcome should be when we are addressing something, by keeping our tasks small and measurable it becomes easier for us to say when we’ve achieved something.

Conclusion

Ultimately time is a valuable commodity, which we can’t get back, so why should we waste it? Time is money, time that we spend waiting for stuff to happen so that we can be productive is wasting the clients money and our time. This also applies to our own personal lives and processes, even stuff as simple as how we commute to work and making some small, incremental changes can add great benefit.

I believe that we have lots of examples of waste around us in our day to day lives; it’s about being able to identify them and trying to fix them in order to increase the quality of our day. Personally I feel good about a day’s work when I’ve been able to crack on and be productive, however when I’ve just been caught up and not been able to deliver value I find that I leave work feeling frustrated.

At the very least we need to be able to identify our areas of waste, as a result of doing that we can start to work on fixing them iteratively and measuring our progress as our iterations go by. We don’t always need to focus on the big issues either, sometimes it’s just as good getting several small wins. Whilst writing this blog, I was put on to the concept of ‘marginal gains’, which was a major factor in the success of the 2012 British Olympic Cycling Team: “The whole principle came from the idea that if you broke down everything you could think of that goes into riding a bike, and then improved it by 1%, you will get a significant increase when you put them all together”

We should be able to apply this concept to developing our software, everything from gathering requirements, writing code, system testing functionality through to our deployment process (each of these area’s comprise of several smaller things but just for a high level example). If we improved each area by just 1% we would be well on our way to getting rid of some of our waste, delivering value to our customer quicker and improving our working day.

Understanding and Addressing our Waste was originally published by Capgemini at Capgemini Software Engineering on March 04, 2016.

Specialism Constrains Throughput

2016-01-22T00:00:00+00:00

Why Specialisms Within a Team Leads to Constraints on Productivity

Tim, a good developer, has been working on a story. “Fred, I need the build configuration changed to pick up the new component I’ve added.”

Fred responds, “okay, I can fit that in next week.”

“Oh, it’s the demo on Tuesday and we need to deploy to test it before then.”

“Well I’m busy with other tasks at the moment, perhaps you should have planned ahead.”

–

This sounds reasonable on behalf of Fred, the Build Engineer. He has limited availability and needs his work planned out.

But that is exactly the sort of behaviour agile practices are trying to move away from. We want to move towards shorter cycles and faster feedback. We want to be able to do things now. This is the kind of problem the DevOps mentality is supposed to resolve.

Staff Liquidity

I had the privilege to hear Chris Matts speak at Agile Cambridge a couple of years ago. He was talking about options and availability of people. How do we make people available? Get them to do less. This is counter-intuitive - how do we get more done by doing less? Surely we want to give our best people (our specialists) the most work?

It turns out high utilisation doesn’t necessarily lead to high productivity. Plans are fragile as estimates are a guess; plans are fragile as unexpected work arises; plans are fragile as people aren’t always available. Delays lead to blocked work.

Chris’s suggestion was to resolve this by not assigning work to the specialist. Better to get them to mentor and guide others to raise their skill level and to leave the specialist free to tackle emergency issues when they arise. Over time this resolves the dependency on key individuals and, overall, makes them available for high value work.

Expert Level

The Dreyfus Model of Skill Acquisition describes a few levels of skill, with “expert” being the top level.

It is these experts that are in short-supply. It is these experts we want to work on the critical aspects of our work. But it is also these experts that we build dependencies on.

A better model is to raise the level of the team, over time, and only consult “the expert” when needed. Of course this doesn’t mean the expert cannot help out day-to-day, they can even take on some important but lower-priority work. They can also add a lot of value by reviewing work output of others and helping improve quality and technique.

Generally speaking, I’ve found that experts “go their own way” anyway. It’s very difficult to tie-down an expert to a rigid plan. They are best off deciding for themselves where best they can add value.

Skills Matrix

There may of course be a few areas of specialist knowledge. Chris’s suggestion was to have a skills matrix and try to bring everyone up to a certain level. This won’t necessarily be even across the team as people have different interests and skills, but the idea is for everyone to have a broad level of competency and one or two stronger areas.

Let’s say each skill for each person is ranked 1 (novice) to 5 (expert). We probably want to bring everyone up to level 2 for all skills and have a few team members at level 4 for all skills. I haven’t heard the term “generalising specialist” used for a while now (some people thought it an oxymoron!), but I like the term and I think this is what it meant. Not quite “Jack of all trades, master of none”, more good at everything, but great at some.

But note I didn’t suggest anyone at level 5, expert. I think such people are rarely good across the board (how would they have time?) and are in so much demand that it’s difficult to tie them into a team. Better to treat them as an advisor or a consultant.

I do know one person who is actually a specialist-generalist. By this I mean he specialises in being able to pick things up quickly, being flexible, being adaptable, being approachable and most of all a great problem solver. It is this broad range of skills mixed with great soft-skills (which are underrated in his case, even by himself) which mean he is always in demand.

Trust

Part of the problem described in my fictional (!) conversation above is a lack of trust between the build engineer and the developer.

Of course there needs to be a certain level of control, but better to trust people to do the right thing and address the occasional problem rather than never do the task and never learn.

Restarting the Conversation

So how might that conversation go? Perhaps something like:

Tim, a good developer, has picked up a story. “Fred, I’ve changed the build configuration to pick up the new component I’ve added. It’s built and deployed to the test environment so it looks okay.”

Fred responds, “ah, I had noticed the build was taking a little longer than usual, I’ll see if I can do something to speed it up.”

“It’s the demo on Tuesday so we need some stability around then.”

“I’m busy with other tasks at the moment, the build should run okay for now but if there’s a problem give me a shout.”

Specialism Constrains Throughput was originally published by Capgemini at Capgemini Software Engineering on January 22, 2016.

Agile Cambridge 2015

2015-09-25T00:00:00+01:00

Failing Fast - An Autopsy of a Failed Release

I’ve been lucky enough to be selected to speak at Agile Cambridge 2015. My talk is titled “Failing Fast - An Autopsy of a Failed Release”, please read more about this talk on the conference site.

I attended the same conference in 2013. I very much enjoyed the conference and I was inspired enough to submit a proposal the following year. Fortunately this was accepted and I enjoyed the pleasure of speaking about DevOps in the Fellows Dining Room at Churchill College, Cambridge. (A dining room? It’s an impressive room with no doubt an invaluable collection of artwork on the walls, including some very eye-catching Andy Warhol pieces on the walls!)

Speaking about DevOps proved to be a little controversial, but it did prompt some good discussion - kind of my aim for the talk. Essentially I was saying DevOps is a way of thinking, not a separate team. I wanted to define the title “DevOps Engineer”, something I found very difficult to do because of the wide skill-set and level of experience required.

This year I decided to be a little “safer” in my proposal so I decided a case study was the way to go. But then I used the word “Failed” in my title and I seemed to stir-up a few emotions! I can understand that “failure” has negative connotations, but surely not admitting failure early is even worse! “Failing fast” is seen as a benefit of agile practices and adopting such practices is supposed to make catastrophic project failure a thing of the past.

I’ve written a few blog posts now, loosely around agile practices (of course!). Please see my section on the Capgemini Engineering blog to find out more, or indeed to find other posts from my fellow bloggers.

Agile Cambridge 2015 was originally published by Capgemini at Capgemini Software Engineering on September 25, 2015.

Risk Burndown

2015-03-12T00:00:00+00:00

Few Agile approaches actively manage risk. There are some tools to help with risk, such as sprints, Definition of Done, etc. but there are few tools that are explicit about risk.

Risk burndown is an approach I have used to visualise risk.

Risk Value Lifecycle

A key part of Rational Unified Process (and other similar unified processes) is the concept of risk-value lifecycle. It’s not a tool as such, more a thought process alluding to how project engagements should discover and analyse risk.

In RUP, this is an overlay to the phases, especially Inception and Elaboration.

Inception deals with discovery of requirements, or exploring the problem space. The end of the Inception phase is when it is agreed that risks have been discovered and evaluated.

Elaboration deals with the mitigation of risks, or exploring the solution space. The end of the Elaboration phase is when the risks have been mitigated. This is usually when a candidate architecture has been agreed and proven through working software.

To relate this to the risk-value lifecycle, the end of Inception is when all the risks have been discovered and the end of Elaboration is when all risks have been mitigated.

(Of course this is an ideal view. Further risks may be identified after the end of Inception and risks may not be completely mitigated at the end of Elaboration. These are guidelines and are not to be taken too rigidly - it just has to be a consensus view that the project can proceed knowing the level of risk.)

The phase after Elaboration is Construction. By actively discovering risks and mitigating them the project team have built a good foundation for the remainder of the project. This phase deals with scaling and the team can now concentrate much more on delivering value.

Types of Risk

Donald Rumsfeld gave a famous known knowns speech a few years ago. This was largely derided as incomprehensible rhetoric, but he actually made a good point. There are risks out there that we know about, but there are risks out there that we do not know about. Projects must take deliberate action to discover these risks. This action must not be based solely on discussion, it must be based on trying out aspects of the solution.

That said, how does one know the problem space has been explored? Unfortunately I do not have a definitive answer to this, but some pointers are:

Define the boundary of scope. This will determine the space to be explored. (Anything outside the boundary is waste!)
Define the entire space within the boundary. This is likely to be in high level stories (or themes, epics) on the backlog.
Be precise about the boundary of each of these stories to prevent items slipping between the gaps.
Estimate the stories. This is likely to be imprecise for high-level stories, but should drive out some risks. E.g. t-shirt sizing may be good enough.
Assign a risk to all stories. This should be a number, very much in the way stories points are used. The higher the number, the higher the risk. It might be enough to use the estimate itself and a flag to indicate the story carries risk. A risk-matrix approach of balanced severity/likelihood could be used too.

Until this space is explored thoroughly, the project might be carrying a large unidentified risk.

Of course that only identifies risk. The next step is to mitigate risk. In traditional project management this means assigning some sort of action to reduce the impact or a pointer to some other action that will lower that impact. In the Product Backlog risk is lowered by delivering that story, or more likely, extracting the risky parts of the high level story into a smaller story to be delivered high up in the backlog. In some cases this may be a spike or an experiment of some sort. (RUP aims to mitigate features that are identified as architecturally significant - again by taking an action to deliver part or all of that feature.)

Risk Burndown

Applying the method described above would result in a backlog with stories broken down with risky elements extracted into smaller stories.

Warning: Product Owners like to order the backlog by business priority.

Fair enough, let’s deliver the most important features first. Of course what I’m driving at is that business priority isn’t the only important factor, especially early on in the development. Early on we need to perform experiments to prove the architecture. Equally we may explore user experience and other important aspects - but these are usually more obvious than the abstract concepts of architecture and non-functional requirements. Product Owners need assistance here; the Architect is an obvious source of help but a visualisation may help too.

The Risk Burndown is simply a plot of the sum of the risk in the Product Backlog. It is analogous to the Product Burndown, a plot of the sum of story points remaining in the backlog (which might equate to value???).

This gives the Product Owner a visualisation of the risk, an important tool for discussion with the Architect and engineers. Of course the Product Owner may have good reason to prioritise on factors other than technical risk - but at least they are armed to make an informed decision.

The chart above is an example I used on a previous engagement. The vertical scale was number of stories (but could easily have been story points). Risk was marked as count of stories carrying significant risk (in this case marked as “architecturally significant” in the backlog) multiplied by 10 (weighted so the plot line would show up as significant compared to the other lines). (Applying the 80:20 rule, a factor of 5 would be expected: as a rule of thumb about 20% of the backlog carries risk. In our case a factor of 10 seemed to work.)

The risk-value crossover was somewhere around the 10th April, though don’t read too much into this. It’s not a milestone in itself (especially as an arbitrary weighting of 10 was applied to risk in this case!) it just indicates a good time to hold conversations around mitigation of risk vs speed of delivering value.

Conclusion

Lightweight frameworks (e.g. Scrum) are not explicit about managing risk. Backlogs and project management techniques may not cover the detail that the delivery teams themselves require.

The Risk Burndown approach described in this article is one such technique, allowing the current risk level to be visualised. This allows the right questions to be asked during the lifecycle of the project. Very early in the project the trend would be for risk to increase as the risks are uncovered. As the development starts to gain pace the risks should plateau and start to drop.

As always, transparency allows us to inspect. Inspection allows up to adapt.

Risk Burndown was originally published by Capgemini at Capgemini Software Engineering on March 12, 2015.

Little's Law and KanBan

2014-12-12T00:00:00+00:00

This post delves into Little’s Law and how it relates to KanBan for software development. Understanding these concepts and their relationship will help agile practitioners improve the delivery pipeline, in particular by knowing what to measure and inspect. (Software engineers will also spot the parallels with scaling a solution: capacity, response latency and throughput.)

Little’s Law

Little’s Law is often quoted in the context of KanBan. Little’s Law is a mathematical theory of probability in the context of queueing theory:

L = λW.

L = number in the system
λ = throughput
W = response time

In practical terms this means:

response time can be calculated from number in the system and throughput (W = L/λ)
number in the system can be calculated from throughput and response time (L = λW)
throughput can be calculated from number in the system and response time (λ = L/W)

Steady State

The relationships described by Little’s Law only work in a steady state. Steady state is achieved when the arrival rate is the same as the departure rate on average over the long-term.

This means that throughput, arrival rate and departure rate are the same metric.

WIP

Guides to KanBan discussing Little’s Law will introduce the concept of WIP (work in process or work in progress), or more to the point, WIP limits. In order to maintain steady state, a WIP limit is defined before the bottleneck. This ensures the “system” (a particular step in the process) is protected from overload by limiting the arrival rate.

If we take a commonly-used example - a supermarket store - the throughput is the same as number of customers entering the store and is the same as the number of customers leaving the store. Minor disruptions to this flow are okay as Little’s Law holds over the long-term.

But what happens if too many customers enter the store? First, the number of customers in the store will increase. This will lead to a greater demand on the checkouts. More checkouts may need to be opened. Once the capacity of the checkouts is reached, the queues at the checkout will get longer. Eventually these queues will get long enough to fill the capacity of the store and block customers trying to enter the store. This may even block the exit to the store and so customers can no longer leave the store: chaos. Log jam. A broken system no longer in steady-state.

In other words, the arrival rate exceeded the departure rate and the throughput couldn’t be maintained: Little’s Law no longer held.

A WIP limit would have prevented the problem. A example WIP limit would be periodically closing the doors to restrict the number of customers entering the store.

Queue Time

Of course closing the doors only prevents the problem of blocking the entrance. Ultimately this would have led to another problem: queues outside the store or, worse still, turning customers away.

So what else could have been done? In the example the service rate was increased - more checkouts were opened - in an attempt to match the increased arrival rate. To enhance this option further even more checkouts could be installed. Perhaps the capacity of the store could be increased: building work to increase the size of the shop floor.

Queues are a form of waste within Lean. This is because queues are considered a form of inventory - potential value sitting on the shelf. In the example, customers waiting at the checkout are finished shopping (the trolley contains the potential value), the value will only be realised when they have paid for the goods. And of course, nobody likes queueing!

So a better approach might be to improve the efficiency of the checkouts - better training of staff or more efficient checkout systems.

Improvement

The mere presence of the WIP limit tells us there is a weakness in the system. Regularly hitting the WIP limit (or even breaching it) should tell us we have a problem that needs attention. Lean (or rather the Toyota Production System) might refer to this as an “Andon” - a warning lantern of some sort to indicate there is a problem.

Conversely, never getting anywhere near the WIP limit might indicate our system is under-utilised, essentially another form of waste. To much resource has been allocated to the system to match the demand. Perhaps improvement work could be done to release this spare capacity and allocate it elsewhere.

Bottlenecks and Throughput

These constrained systems are often referred to as bottlenecks. However there is actually only one bottleneck step in a steady process. The bottleneck step is the step that determines the overall throughput, that is the throughput of the overall process is equal to the throughput of the bottleneck step - the step with the lowest throughput. This is because the upstream steps are feeding the bottleneck at a rate equal to or higher than it can handle; downstream steps are being fed at the rate of the bottleneck as so cannot achieve a higher throughput.

Addressing any other step in the overall process will have absolutely no effect whatsoever on the overall throughput. There is no point increasing the efficiency of any step other than the bottleneck step in trying to increase throughput.

KanBan and Software Development

One of the challenges software development faces is that the process is rarely steady state: it is notoriously difficult to estimate accurately and it’s unpredictable when problems will occur. The arrival rate into the process is often very variable - feast or famine. Add to that pressures to deliver rapidly and competing priorities it’s easy to see why software delivery is an unpredictable art rather than a nice predictable science.

So what can KanBan and Little’s Law offer to software development?

Firstly, we need to know what our process actually is. How often does the delivery pipeline get mapped out end to end? This is worth doing, even the steps outside our influence. Even if all this achieves is to visualise the process, this is a valuable in itself.

Then we can start to analyse the steps. Collecting metrics will help us do this. Referring to Little’s Law, for each step we need to know any two of:

L = number in the system
λ = throughput
W = response time

…then the third can be calculated assuming a steady state. (On that basis it may be worth tracking overall throughput to check how steady the overall process is.)

High numbers in any of these for a particular step (compared to other steps in the process) will indicate a problem / potential bottleneck.

Assigning WIP limits to steps will also tell us how often desirable throughput rate is breached. If a low-WIP is set then this can also tell us if the step is under-utilised.

For over-utilised steps either make the step more efficient, assign more resource or increase the storage capacity within the step.

Since L is proportional to W, increasing the number in the system is a trade-off that will increase the response time. i.e. increase lead-time.

Achieving Steady State

Given that the demand is fluctuating, estimates are inaccurate at best and problems can occur any time, it is difficult to maintain a steady state. There are two ways to achieve this:

Reduce the demand to a throughput level that will never breach any step in the process.
Increase the throughput level of every step in the system to match the maximum demand.

The first option gives the ultimate control. The process is streamlined, there is little waste in the system. The cost is that throughput is decreased.

The second option gives the best performance. Any level of demand can be met. A good deal of redundancy has been introduced to the system to cope with a sharp increase in demand. The cost is that it may require a great deal of investment to achieve such a system and it may take a significant period of time to get there.

Of course reality is somewhere between the two. Rarely can we afford ultimate performance, but rarely can we constrain the system to a low-level of serviceable demand. The economic version of this is turning customers away - rarely good business sense.

Flexible System

A blockage of some sort is usually a pinch point in software development (e.g. a defect or a failed build). Ideally people will “swarm” to the problem - but this depends on availability of people to be able to swarm. If the system is overloaded people are not available - thus it’s better to build in some spare capacity to free people up should such a situation arise.

Out of software development, motorways are an example. When the system is running at full capacity, it only takes a minor disruption in flow to create a traffic jam. The incident may have been as simple as a driver changing lanes causing the car behind to apply the brakes. This causes other drivers to brake causing a disruption - chaos ensues. (See also traffic compression wave.)

Traffic management has improved with the relatively recent use of variable speed limits. This works in two ways: a slower speed limit is introduced (effectively a WIP limit) but also the hard shoulder is opened up as another traffic lane (capacity is increased). I’m not sure how this is achieved technically - I presume there’s some traffic control centre monitoring traffic flow and throughput - but it will be down to some measure of this metric, plus number in the system and response time. Perhaps if the number of cars passing junction 4 is greater than the number of cars passing junction 5 this indicates a problem between junction 4 and 5? Response: activate variable speed limit traffic control.

Closer to software development, elastic computing works in a similar way. Demand increases, more CPU is added. Demand decreases, CPU is taken away.

Could this be achievable in software development? Detect (or predict) an increase in demand, so increase the available resource? Maybe, it will depend on the lead-time to react. If by resources we mean people then it often takes a while to get more people in, familiarise them and re-establish the team (forming, storming, norming, performing; see also Brooke’s Law). If by resources we mean computing power (e.g. for an automated pipeline) then elastic computing is desirable - so long as it has been designed for.

Measure It!

Of course we cannot make any informed decisions if we don’t have the information. To get the information we need to measure things.

Recommendation is to: start at a high level and map the process end to end; measure enough to find out where the hot-spots are, then delve deeper into those steps.

Automated measurements are better than manual (rapid, repeatable, reliable). Automation is essential if used as an indication for immediate attention.

A high-threshold mark will tell us when the demand on the system is too high. A low-threshold mark will tell us when the demand on the system is too low. Plotting these measures on a time-line would be very useful for analysis - perhaps as input to a retrospective.

Simply providing the metrics may well be enough to grab attention.

About This Post

My motivation for creating this post was not to explain Little’s Law, rather to show how it relates to KanBan. I had struggled for some time relating the concepts as Little’s Law is all about steady state and software development is very rarely steady state. I had understood this related to WIP and WIP limits, but had not appreciated how.

In this post I believe I’ve also introduced a new concept, that of the low-utilisation threshold. Tracking such indicators over time could be very useful for elimination of waste or re-deployment of resources.

Little's Law and KanBan was originally published by Capgemini at Capgemini Software Engineering on December 12, 2014.

Agile Analogies for Software Development

2014-11-18T00:00:00+00:00

I’ve recently been working in a training role, with people who have never been exposed to the software engineering lifecycle before. I was drawn to considering analogies to try and explain to them what we were trying to do. It occurred to me that a bad analogy can be as destructive as a good analogy can be helpful, and with software engineering a lot of the analogies that immediately spring to mind are not necessarily conducive to understanding how a software development project should work.

Other engineering disciplines (mechanical engineering, building construction) often require a waterfall construction method - it’s rare to be able to deliver subcomponents before the whole build is complete. You can’t address the riskiest area of a tower block construction project first if the riskiest area happens to be the roof or the windows. Change is harder and more costly - if you want to move a door, or (heaven forbid) the whole building, you’re looking at huge deconstruction/replan/reconstruction costs. The design documents are not fluid. When would you ever start construction without a detailed, low level plan covering the position of every door and window? It’s just not done.

So, if not the other engineering disciplines, where do we look for a suitable analogy? How about the world of television and film production? Even the end product itself fits better to this analogy. Yes you have a finished product, but it’s ethereal. It can be expanded upon, there can be another “series”, it can be cloned or reworked for different scenarios. Let’s consider the methodology now.

How would the analogy of creating a TV series fit with an agile way of working? Well, you can focus on risk first. Get those scenes with the old bloke / pregnant woman under wraps! You can build a disposable prototype and trial it on a user group; then you can adapt your requirements based on their feedback. You can work iteratively and deliver production-ready output in each iteration; you can go live with a subset of the total (think, for example, of a pilot episode from a series which airs before the entire series has been made). As for documentation, a TV script is a fluid document. We don’t write that we are writing the script, we just write it. You can think of code, or the Software Architecture document, in this way. I think that the analogy of the product owner as the director can also assist in explaining to a non-techy client what will be expected of them in this role. It’s not project management, it’s not technical details (the director doesn’t need to know about lighting rigs).

In my opinion, this explanation would put naive or new-to-software clients in just the right frame of mind to begin an agile project. Do we have a timeline? Well, yes, we know when we’re wanting to go live but we don’t yet know for certain how many “scenes” (think how many user stories) we are going to be able to build by then. Do we have a requirements document? We have our overall outline; we will drill into the detail of each one during its iteration. That way, we can easily incorporate a little change as we understand our product better. Let’s go!

There are certainly a few places where the analogy breaks down. A television programme is seldom interactive or reactive, yet a computer system will be. And a computer system doesn’t often have the requirement for historical consistency - if you kill off a character in a TV series it’s tough to bring them back in later, whereas with programs the whole thing is disposable and can be replaced with version 2.0. But let’s not get carried away with the details, that’s not the point of this exercise.

In summary, if you have a new client who is struggling with the concepts of agile software development, give this analogy a spin and see if it helps with clarity and understanding.

Agile Analogies for Software Development was originally published by Capgemini at Capgemini Software Engineering on November 18, 2014.

Agile and Lean

2014-10-22T00:00:00+01:00

Or should that be Scrum and Kanban?

I was in a short conversation a few days ago around whether we, in the business of software development, want to raise the profile of Lean. Perhaps even create some formal training materials.

The message I was getting was that Agile doesn’t need Lean. This is because Agile practices are built on Lean - especially Scrum!

This is true. Scrum especially is not really a software engineering practice at all. It’s an approach to organising people to deliver something that initially is not very well defined.

I think the fear is that Lean is a process improvement technique for processes that are already reasonably steady state. Software development is NOT steady state. It is unstable all through the lifecycle, especially during the beginning when there are many risks, possibly many unmitigated risks. Lean doesn’t really address risks. Lean is focussed on improvement of an established process.

What Does Lean Bring?

Having reflected on the conversation, this view misses the point somewhat. Often the barriers we face are not in the software development portion of the delivery: rather gathering timely requirements, planning & estimation, testing/QA and cut-over present the challenges. Often these are not addressed as they’re not in our immediate field of view, snowball and present even bigger challenges.

Lean can help here. Lean can map this “process” end to end. It can then help establish metrics to demonstrate where the problems lie.

Lean can also help by bringing a meeting structure to all levels and promote the idea of servant leadership.

A conflict, or perhaps even a paradox, is that the customer is often the supplier too! Maybe we need to redefine this: the customer must set clear parameters for performance (the goal), then the project need to convert these into requirements themselves by consulting the stakeholders on the changes that will bring about positive impacts.

For example, an analysis I performed a while ago on a series of project releases is shown in the charts in the image.

The top image is a Cumulative Flow Diagram (CFD) for a number of releases into production of a product. Each band shows the user stories that were queued or in in-progress at that time. The large steps are where the process was outside of the project team’s direct control. The thinner red stripe was where the team were working in an incremental way to develop and test the software within sprints. The green band after the red stripe shows the software “sitting on the shelf” for the next phase.

Retrospectives would centre on the “red stripe”. Blockers, knowledge transfer, skills, planning, estimates, etc, etc.

Wider Improvement

Perhaps we should have been looking at the larger steps.

At the time there was some thinking around improving the process. The driver was the desire for automated testing, but I had a look a little wider. I speculated what the same delivery might have looked like by addressing the larger steps, resulting in the second chart shown.

There is a clear difference. Immediately there is a noticeable change in the amount of “green” on the chart. This is because the software is not sat on the shelf for so long: it is “released” quicker!

Because the releases are more frequent, less goes into them. This reduces the demand in requirements coming through. Instead of having to define a good proportion of the stories for a large release up front, the number of stories needed is much lower and therefore with a lower lead-time. This is reflected in the thinning of the first dark blue band.

The example demonstrates the potential for using Lean tools, KanBan and CFD, to inspect and analyse the pipeline. Embracing Lean further may well reveal further opportunities for improvement.

Agile and Lean was originally published by Capgemini at Capgemini Software Engineering on October 22, 2014.

Answering: How long will it take?

2014-10-06T00:00:00+01:00

“How long will it take to build my [thing]?”

A seemingly simple question that can strike fear into the heart of many developers, estimation is both essential and thankless at the same time. As a developer, you might feel you can never win - always asked to do things quicker - whilst a project manager usually wonders why developers estimates are inaccurate. In this post I hope to share some techniques that have helped me along the way, and might help reduce the pressure on justifying your estimates!

It’s only an estimate

The first thing to remember, is that an estimate is:

“an approximate calculation or judgement of the value, number, quantity, or extent of something… Synonyms: approximation, evaluation, guess” The Oxford Dictionary

So, estimate equals guess. Done properly, it should be at lease be an educated guess but your estimate is always going to be flawed in some way, something will change, be discovered, or interfere and mean you’re not going to be 100% correct. In fact, we can go further and say: you’ll almost always be wrong.

Getting it wrong doesn’t matter, it’s part of the learning process and feeds back into making future estimates better.

Simply take note of why your estimate was wrong, and make sure that the next time you are coming up with an estimate you consider whether the factors that de-railed your previous ones will apply in your new scenario. The aim should to be “less wrong” next time.

Never answer right away

How many times have you been asked how long something would take, maybe in your daily stand-up, and quickly responded “by the end of the day” or “end of the week”, without really thinking the estimate through? Sometimes you might get lucky and things pan out ok, but many times these instinctive estimates don’t think through the problem properly and often take longer than first given.

Now that’s fine, it was only an estimate after all, but you’ve set an expectation and in all likelihood you’re going to end up working extra hours to make it happen to avoid disappointing someone.

The best tip I ever read, came from reading The Pragmatic Programmer, and went something like:

“You almost always get better results if you slow the process down… Estimates given at the coffee machine will come back to haunt you.”

Step away from the discussion, ask for time to analyse the problem, even if its half an hour to organise your thoughts, before giving an estimate and ensuring it’s an answer you believe in.

Break down the task

For small tasks, arriving at an estimate can be pretty straightforward, but usually we’re not being asked for small task estimates. People want to know how long adding a new feature, or building a new website, will take. The trouble with this is that seemingly simple high level tasks can hide a multitude of traps that could inflate (or deflate) your estimate.

When faced with problems like this, the only way we can process enough information to give anything like an accurate estimate is by breaking the task down into as many constituent sub-tasks as necessary until each sub-task can easily be envisaged and estimated.

Break down the feature into components you’re comfortable estimating. As a hint, if the estimate for any one sub-task is longer than 3 or 4 days, you probably need to break it down further. But having said that, don’t make any task smaller than half a day in your estimates - even that trivial one line code change will need regression testing, unit tests tweaking, functional tests updating, and code review performing, so it’s never “just a 10 minute change”.

One word of warning though - you’re not trying to do a detailed design here, just trying to get tasks or stories at a level granular enough for you to know roughly what you’re going to need to do.

List your assumptions

Every estimate is based on some assumptions you’ve made, so make sure you share these when presenting your estimate back to whoever asked for it. If you think there are no assumptions, then have a look through the list below and see if any would apply:

My estimate is accurate as long as …

…I get uninterrupted time to focus on it
…the environments are configured correctly
…I can re-use component X that we built last time, without significant refactoring
…the interface specification is accurate

Also, make sure you define “done”. Will done mean that you’ve committed the code and run some tests locally, or that you’ll have followed the change through various environments and its ready to be pushed into the next Production release? Regardless of how you define “done”, this is one of your key assumptions, so make sure you and the person you’re giving the estimate to have the same definition of done and therefore share the assumptions.

Add inflation

We’ve covered this a little bit when saying not to estimate any sub-task at less than half a day, in order to factor in all the things we forget about when estimating. However, even when you add up all your sub-tasks (or put them into groups for adding up) we should pad our larger estimates to reflect the inherent uncertainty in the longer term estimate.

For example, if your estimate is around 3-4 days, call it a week. If it’s 3 weeks, then its a month, and so on. Using larger values of time helps set expectations with your stakeholders and also lets you have contingency for the inevitable unforseen delays that will arise. The longer the road, the more likely you’ll find a pot-hole.

The other way to approach uncertainty in your estimates is to provide a timescale range rather than a fixed number, with the size of the range depending on your confidence. For example, if you’re very confident you could answer “6-8 days” but if you’re not certain of your estimate you could say “20-30 days”.

Offer to re-estimate

Often the estimates we give, even when following the guidance above, will be padded out to reflect our uncertainty and unknown complexities. There might be new technologies or approaches, uncertainty over the level work that will be required to meet a performance target, or anything else that you have little experience in on which to base your estimate.

In these cases, and to reflect the uncertainty, you could offer to re-estimate your tasks once you’ve completed a portion of the work. This could be at the end of a first iteration, or initial exploratory phase, during which you’re likely (again, if following good agile practices) to have started to tackle the things which had the highest risk, uncertainty or potential complexity.

Taking this approach allows you to give an estimate, and then use new found experience to re-baseline after a short period in order to adjust your estimates (up or down!) based on what you’ve learnt. It also demonstrates to a project manager that you are mindful of trying to reign in estimates to as much accuracy as possible, but also gives an opportunity to flag areas which turn out more complex that first thought.

Even better would be regular backlog grooming of as much of the backlog as is practical, re-visiting estimates and updating them as you progress through your project and factor in lessons learnt as you go along.

Look back

The number one criticisms of developer estimation is that we never learn from our bad estimates, and so never improve our accuracy. In many ways, this is a false accusation because as we’ve seen no estimate is created equal and the things that trip us up this time might not arise in future tasks, however there is definite value in looking back and comparing your estimate with your actual, making note of the reasons you either missed or achieved your estimate.

If nothing else, this kind of retrospective helps you identify some more assumptions that you made without realising, and will give you a useful tool for future tasks.

Conclusion

Estimates are unfortunately a necessary part of being a developer, and being able to give confident, accurate estimates breeds confidence from the rest of your project team.

If at all possible, never estimate alone - make it part of a team activity. If you’re following good agile practices then you should be doing team planning at the start of each iteration. This can be a great opportunity to try out the principles in this post, and learn from the things other people in your team consider for their estimates.

I hope these suggestions are useful, if you’ve got more tips to share please post in the comments.