5.2 Development processes
The identification of a set of software engineering activities associated with the development of software is the key to meeting the potential diversity of users’ needs. It is the primary mechanism for understanding and harmonising these, sometimes conflicting, demands on the development process. Each activity will undertake some clearly defined process, starting with a number of inputs from preceding activities that are the basis for further work. On completion of an activity there may be one or more outputs (deliverables). The sequence in which these activities are carried out defines an overall process for the development of a software application or system.
Through such activities we shall emphasise a disciplined approach to software development. This has three key elements:
- methods, which describe the technical activities required to develop software
- tools, which allow computer support for the methods
- procedures, which provide the glue that holds the methods and tools together; it includes project and quality management activities.
Project management and quality assurance activities are considered to be part of the life cycle of software development. They form an integral part of every software development project, no matter which particular life cycle is used.
A complete life cycle would take us from those first ideas about the need for a software system to its final withdrawal. We shall assume that a customer has assessed the feasibility of some initial ideas with a potential group of users and has decided that a software system may be required.
Space does not permit us to describe the many and varied methods used to develop software. They can be found in texts on software engineering such as Sommerville (2004). However, each one normally includes the following four activities:
- Analysis This involves identifying the problem and deciding what needs to be done in order to solve it. You need to be able to include all that you consider to be relevant, which is similar to defining a system and its boundary.
- Design You determine how you will solve the problem.
- Implementation You act upon those design decisions.
- Testing You apply what you have done so that you can determine whether or not you have solved the problem.
Some methods subdivide the above activities and some use different names, such as programming for implementation. However, by themselves, the four activities of analysis, design, implementation and testing are not enough to develop a good software system. Other activities are needed to a greater or lesser extent depending on the context, as you will see in this section. For example, you are likely, first, to break most problems up into smaller, more manageable chunks, and then deal with each one separately. If you do this to develop a software system, it will be necessary to bring the chunks together into a unified whole. This process is known as integration and is sometimes identified as a separate activity. Sometimes, the act of delivering a software system is identified separately, especially when there are contractual implications such as payment.
We expect that a software system will change during its operational lifetime. This is the maintenance activity, which allows a software system to evolve in order to:
- correct errors
- adapt the software to a changing environment
- introduce enhancements required by the customer
- improve the software in anticipation of future change.
The four activities of analysis, design, implementation and testing are the backbone of the practices that turn software development into an engineering discipline. They are the activities you will see most often in diagrams depicting the life cycle of software development. Project management and quality management are the two additional activities that hold the process of development together – the all-important glue for software engineering activities. Maintenance will inevitably involve the activities of analysis, design, implementation and testing, and will itself require managing.
When the five activities of analysis, design, implementation, testing and maintenance are arranged into a single sequence, this leads to the classic waterfall model for the development of software. But, even assuming that you can complete each activity completely (and correctly), the waterfall model has drawbacks. For instance, a working version of the software system will not be available until late in the testing activity. For larger projects, this could be significant, since it assumes that the customer is patient and the users are not concerned about the delay required to produce a working response to their requirements. In addition, any errors detected in the working version of the software could be disastrous. Real projects rarely follow a purely sequential life cycle.
An alternative life cycle iterates around one or more of its activities, while maintaining a core that uses a sequence of activities. Iteration allows the developers to improve the outputs from a given activity or set of activities according to the criteria used in the particular iterative life cycle. For example, iteration allows a group of people, usually developers, to perform a review of an activity or the outputs from that activity. In general, reviewing a proposed solution provides the feedback necessary to modify it and solve the problem correctly. Think what happens when you need to tune a guitar or violin. You pluck a string, a note sounds and you adjust the tension on that string. You repeat the process until you get the desired pitch. In practice, of course, you would then tune another string, using the same method, but this may mean that the tuning of the first string has to be repeated. Only when you are satisfied that all strings are individually and collectively tuned would you feel that the instrument, as a whole, has been tuned.
Another way of forming a life cycle involves partitioning a system into independent or semi-independent chunks, known as increments. Increments can be developed sequentially or in parallel, depending on circumstances. For example, a small team might choose to develop increments sequentially, according to a priority agreed with the users. Since it is not always possible to partition the users’ requirements into independent chunks, some additional activities will be needed to manage the dependencies between the chunks. On large projects, a single developer may be allocated to this role. Each activity will involve identifying the interfaces that allow communication between one chunk and another. Such activities would coincide with the start of each increment and continue at regular intervals.
Incremental development is a way of dealing with the complexity of requirements that comes from the notion of breaking up a large problem into a number of smaller, more manageable ones (known as modularisation). As part of the development process, there will be a number of tasks devoted to putting the pieces back together (integration).
A further refinement of incremental development is known as timeboxing, which is the act of setting a fixed-sized period of time in which to achieve an objective. By giving each timebox its own deadline and budget, the intention is to make or deliver something (Stapleton, 1997). The people assigned to each timebox will typically have the freedom to decide how to achieve the objective. As long as there is agreement with the customer or users, a deliverable may be modified in order to meet the deadline.
Whatever the chosen life cycle, it must reflect what actually happens rather than provide a theoretical model of some ideal approach to development. This is not to say that an idealised life cycle cannot be taken as a starting point. But it must be modified as necessary to reflect what actually happens. If this is not done, the project manager will not have an accurate framework for planning and activities will be included in project plans that do not actually happen, while other activities that do take place are excluded.
A method for developing software will provide an integrated set of techniques for carrying out the above activities (as well as any others included in any particular process) in a well-defined way. It will tell you:
- the notation to be used in each activity and the rules for using such notation
- the necessary inputs and outputs (deliverables) required for each activity
- the process to follow; that is, how the activities combine to produce the desired result.
The life cycle provides the framework for the project plan, while the method should allow detailed planning of both the software development process and the quality assurance activities. In principle, a method might be assumed to address every activity and every deliverable product in the life cycle. In practice, this is seldom if ever true. Some methods deal with only part of the development process such as testing, leaving out analysis and design.
By definition, a good software system must be fit for its intended purpose. It should be evident, therefore, that since software is needed for such a variety of purposes there is no single development process that will suit all purposes. For example, consider the following goals that might initiate some development activity:
- to control a series of gates at level crossings
- to control a manufacturing process for chemicals
- to manage an international stock market
- to manage a supermarket
- to manage a public lending library
- to administer the activities of a university
- to help you manage your personal finances
- to book your holidays.
The people who use these systems will have different views about what it means to have a software system that is useful, usable, reliable, flexible, available and affordable. So, it should be no surprise that there are different development methods for different types of system. Indeed, software companies often specialise in developing software for specific kinds of customer, such as education, banking or manufacturing.
The following two examples illustrate how the amount of formality in a development process varies.
Example 9
Government legislation in the UK often requires software systems to be built to support the new administrative functions. When developers compete to supply such software systems, they are asked to conform to government-approved standards. One of those standards is known as PRINCE, which stands for PRojects INControlled Environments and relates to the management of projects and the quality of the developed software. Commonly, developers must be able to show that their development process conforms to the regulations set out in the PRINCE method. In addition, developers may be required to adopt an approved method for their analysis and design activities.
Example 10
In certain financial areas, being first on the market to offer some new service is the way to make the most profit. In other words, the time-to-market is critical: miss out and your competitor takes the spoils. When the need for a software system is established for such instances, development is almost a race. Every aspect of the development process has to be tuned to meet the time-to-market. Thus the aim is to do the simplest things needed that could possibly work when delivered. It also means that communication must be minimised, so there will be very few developers (say, two) who work to a minimal set of agreed procedures. There must still be testing so that the software can perform its intended purpose. Better still, if the amount of actual code (Java or whatever the chosen language) is less, the lower the chances of introducing bugs. So, there are fewer tests to do. But there must be some documentation, unless the developers are not afraid of litigation!
Examples 9 and 10 show that it can be inappropriate to fix a development process for all kinds of product. In both examples, the amount of recorded information during development may be different. However, there is a common need to be able to reconstruct the significant events that happened during the development of the software system as you will see later.
Example 9 illustrates the need for a formal development process. Although it is a slow and deliberate course of action, the resultant software system must be able to deal with all the nuances of the legislation. The developers must be able to show that each aspect of the legislation has been incorporated into the final software system. The control and maintenance of the integrity of a project’s deliverables (ensuring that they do not become corrupted or inconsistent) is the responsibility of the configuration management activity, which is related to the need for integration described above.
In contrast, Example 10 relates to software that is required for a very specialised purpose with a minimal development time. The developers have the benefit of a narrow scope for the software system, so they expect fewer changes to be introduced during the short development time.
In general, the size of a project influences the choice of development process. As illustrated in Example 10, the amount of formality in the process can be minimal. Small teams of up to eight people can communicate and respond to changes informally. In larger projects, such as in Example 9, a well-defined procedure for controlled development is necessary; an informal communication network on its own is not enough.
One approach to the solution of the problem of large projects is to split the group into smaller teams according to the responsibilities set out in a given development process. Just as modularisation is the way to deal with the complexity of a software system, developers can be assigned different activities and deliverables within a given project. Naturally, this partitioning allows developers to specialise and become analysts, designers, programmers and so on.
At some point in its operational lifetime, a software system will have outlived its usefulness. The developers should include a consideration of the expected lifetime of the software that they produce as part of the risk assessment for each project. In the case of Example 10, the time-to-market is a significant factor, whereas the expected lifetime is likely to be short (because of the nature of financial markets). So, a slow and deliberate development process is viewed as inappropriate in that context.
Iteration can also be applied to a life cycle as a whole. Taking a step back to consider a solution (software system) in a broader context allows you to manage the risks associated with a development project rather than ignoring them. For example, if delays cause the team in Example 10 to miss the market window, it does not matter how much software has been developed, as it is no longer of benefit to the customer and the users.
In a typical project, the major risks are around the requirements. Do you understand them? Have you identified them all? Are they changing too frequently? Anything that you can do to increase your confidence that the requirements you have are both necessary and sufficient can reduce the risk of project failure. You have already seen how prototyping can help in requirements analysis. In general, for every decision you make, there is a risk that your decision is wrong or, at least, inappropriate. This is where a review helps. If the risks cannot be overcome, the project cannot succeed.
In any project there is likely to be a trade-off between what can be delivered in a given time and to a specified budget. Often, the number of desirable features of a solution exceeds those that can be delivered for a given price and within a given timescale so choices have to be made. One way to make such choices is to estimate the risk of not delivering each feature. That is, if a feature were not to appear in the final product, what effect would this have on such things as the usability, usefulness, reliability and flexibility of the product?
The phased development life cycle
Phased development is a strategy in which the activities of requirements determination, the evaluation of alternatives, design specification and the implementation of the design are repeated in several iterations. Thus it forms a series of closely linked mini-projects (though each may be of considerable size). At the end of each mini-project some portion of the overall project is implemented and reviewed by the users. The feedback from the review is used to help determine the requirements of the next mini-project. Figure 16 shows diagrammatically the life cycle of a project carried out as four mini-projects, each forming a phase of the overall project.
An example will illustrate this. A large insurance company decided to automate the handling of insurance claims. After initial discussions, they decided to undertake the automation in phases.
There were a number of reasons for this:
- previous attempts to automate major functions had resulted in serious problems arising, which had proved costly to fix
- the resources available to undertake the work were few, so there was no possibility of making a concentrated effort to achieve complete automation in the course of a single project
- the disruption to the normal flow of work in the claims adjustment department would be minimised
- training could occur in phases: people would learn to use the system bit by bit, in a natural way
- feedback at the end of each phase from the users would inform the development and implementation of the next phase.
It was decided that the phases would mirror the way a claim entered the claims adjustment department and moved through it to adjudication and issuing of payment.
The first phase of the project concentrated on the claims form submitted by the claimant. This phase of the project resulted in the creation of a form on a computer terminal screen which could be filled in by an operator talking to the claimant over the telephone. At first, the operators simply took handwritten forms and entered them on the screen form, but quite quickly this changed to the operators using lightweight telephone headsets and entering the information directly on screen. The claim form was then simply printed by the central computer and posted to the claimant for checking and signature.
The second phase made some minor improvements in the form on the screen and the resulting print-out. Furthermore, the central computer, as well as printing the form for checking and signature, passed the form, once approved by the claimant, electronically to the claims adjuster, who no longer worked with a paper copy. The claims adjuster reviewed the electronic form and entered information electronically from quotations from prospective repairers. Any arithmetic needed was performed automatically and then the form was printed out.
The third phase again made minor improvements to the forms and automated the approval by the adjuster and the quality control review by a manager, and simply printed cheques for the claimants in the approved claims, and a printed report of all rejected claims to be used to prepare letters to those claimants.
The final phase used word-processing software to assist the preparation of letters to claimants whose claims either were rejected or required further information: the operator could select standard phrases for inclusion in the letter but could also add customised paragraphs. The letters were automatically printed together with envelopes.
A phased development project life cycle is suitable for projects with a high degree of technological risk attached to them or, as illustrated above, projects where the resources are not available to do everything in one concerted effort. They have the benefit of minimising disruption and of having several stages at which the feedback from users can result in beneficial changes to the end product.
Prototyping
Prototyping is a form of phased development. In prototyping, a model, or prototype, of the proposed system is built quickly and shown to the users in order to obtain rapid feedback. The model may or may not become part of the final system offered to the users. Prototyping reserves the right to throw away a less successful model, build on a more successful model and fine-tune a very successful model. The model may begin as something quite crude (for example, it may ignore function in order to obtain information on, say, ergonomic aspects of the model) and gradually be refined or it may be rebuilt at each iteration or, after several iterations, a ‘formal’ project may be constituted to turn the approved model into a working artefact or system.
The advantages of a prototyping project life cycle are similar to those of a phased development project life cycle. The disadvantage is that it is far more difficult to estimate costs and schedules (due to the experimental nature of the early prototypes) and more difficult to control.
Agile processes
Over the years, a number of proprietary processes have been established for the development of software. Most of them demand the production of particular documents conforming to particular standards, with a significant overhead in record keeping. Following all the rules and having to keep all the various records has proved unpopular with software engineers, mainly because they are distracted from their main pursuit of developing software. This potential for heavy bureaucracy has cost implications, which has not been popular with managers. As a project manager, you do not want to lose control, but you do not want to increase the chances of alienating the software engineers who are your most valuable resource and who you depend upon.
The response has been the emergence of agile methods. People using these approaches realise there is great merit in the informal approaches of small team, small company, software development – what in the United States would be characterised as ‘skunk works’: that is, a small team operating outside normal company policies, with a tight schedule, and working on an ambitious project, often secretively. Somehow these teams develop quality software at low cost. But how do they do it? How such teams achieve this remained tacit knowledge in the past, while agile methods make it visible and generally acceptable to managers and stakeholders.
The Manifesto for Agile Development was drawn up in February 2001 by the Agile Alliance. The original members of the Agile Alliance met to discuss what to do about the bureaucracy of current methodologies and what could be done to release the creative energy of software engineers. They used the concepts of increments, prototyping and timeboxing to establish their principles. For example:
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Working software is the primary measure of progress.
Agile methods do not include prescriptions concerning what should be done when: they do not plan in the conventional sense. They recognise that what a software engineer does is contingent on the situation that each one encounters. As a consequence, agile teams are expected to reflect regularly on how to be more effective and adjust their behaviour accordingly.
Using a life cycle model
Real life is messy and projects, however well organised and run, are likely to run into problems. You may think that there is little value in adopting a life cycle model for a project. In fact, such models are useful in planning, then in guiding the stages of a project and in helping a project manager to manage it, provided that the manager remains aware that the model is only that, a model, and not an exact description of how a project will proceed from start to finish. More important than the life cycle are the milestones and deliverables agreed as part of the project plan.
Software quality
We have as yet no widely agreed measures of the ‘goodness’ of software produced as the result of a project; neither can we measure its quality accurately or positively. Quality measures for software are largely confined to the relative absence of faults. There has been some research effort to quantify and locate the likely source of faults. In one study the percentage of faults by phase was investigated; this is shown in Table 16.
| Phase giving rise to fault | Percentage of total faults |
|---|---|
| Requirements definition | 38 |
| Maintenance (operation) | 32 |
| Design | 23 |
| Installation | 7 |
Requirements definition occurs at the very beginning of a software project. A software engineer or analyst visits the client and users, and interviews and observes them in order to determine what they require. The problem with this phase is the fact that the eventual product, the software, is intangible and invisible. There is not much that can be shown to prospective clients and users. And unless the users are themselves experts, they may not be fully aware of the capabilities and limitations of software, nor may they be fully aware of its complexities and the problems of integrating new software with existing systems. In any case they may have difficulty in expressing their requirements. The analyst or systems engineer, who is rarely an expert in the client’s line of business or the users’ tasks, may have difficulty in understanding fully the requirements they do express.
There are no simple answers for the problems of obtaining a sound requirements definition document. The analyst or software engineer prepares the document and then walks through it with the client and perhaps with some end users, but as there is nothing to try out it is difficult to determine from such a walk-through whether all requirements are covered, and whether those that have been are covered satisfactorily. One possible solution is the prototype approach mentioned earlier: an early prototype, perhaps crudely giving the function and feel of the eventual product, can be tried out to see what its deficiencies, limitations and errors are, and the requirements definition can then be refined.
Software design is plagued by the problem of change: since there is nothing tangible, it is easy to think that nothing has been built, and therefore anything can be changed, and that change is simple. However, the complexity of software, which emerges quickly in the design phase, is such that even relatively simple sounding changes can have knock-on effects that are very difficult to determine until the testing phase. Again, prototyping and phased development are approaches which can help in this respect. However, the problems of controlling this type of development and the increase in the time needed to complete a product can make these approaches unsatisfactory.
Installation, interestingly, gives rise to the fewest significant errors. This is perhaps the one task that is relatively straightforward in a software project, although the project team may not think so when they are scrambling to get everything up and working!
Operation reveals another large proportion of errors. This again is related to the complexity of the software. Hence, maintenance can be a significant activity or even a project in its own right. A fix applied in one area may generate problems in other areas and have cascading knock-on effects.
As well as the problems mentioned so far, you should note that software development involves a series of translations. The requirements document has to be translated into designs, and later the designs have to be translated into a programming language, which in turn gets translated into machine language. At each step, it is possible to make errors: to miss part of what needs to be translated, to insert something that was not there before, to mistranslate. And at each stage, it is also all too easy to make apparently simple errors, such as the substitution of a full stop for a comma, that have enormous effects.
Having read some of the problems associated with software, you may be forgiven if you wonder how any usable software ever gets written. If you have much experience with any software at all, even a simple word-processing or spreadsheet package, you will have found things that do not work as you think they should, or do not work as they are described. Nevertheless, even somewhat flawed software can be an enormous boon in carrying out daily tasks such as letter and report writing and financial record keeping and planning. If your word processor only prints one page at a command instead of a whole document, it may be annoying but it is not threatening. In contrast, the consequences of failure may be much more serious in the software used to fly aircraft, to monitor radar at busy airports, to manage power stations (including nuclear power stations) and even to manage systems in our motor cars.