Category: Waterfall

Software re-engineering of legacy software

Posted on by Steve Padraig in Agile, Software Development Manager, strategy management for it services, Waterfall

There are legacy applications that have been there decades, and much younger applications but a few years.  Both can be subjects of software reengineering for a variety of reasons:

  1. technology no longer supported
  2. to inflexible and/or unable to harness emerging technologies
  3. newer technology can give a better solution
  4. it was just simply not written as best as it could have been
  5. the code has much duplicity, not scalable, and maintenance costs high
  6. badly architected
  7. after much maintenance the code has become disorganized
  8. skills to maintain are becoming rare
  9. compliance with updated or new regulation

What is software reengineering?

Software reengineering is the examination and modification of existing code to continue its use in a more refined state or to migrate it to a new modernized system.

Why not just start again and re-write?

This is the area that generates most debate with agile followers that say “we can just start again and throw away the legacy system when we are finished.  All we need are dedicated users to work along side us”.  This is re-inventing the wheel.

Software often has complex business-critical processes within.  Simply re-writing and you may lose valuable information hidden within the code.  Migration requires more upfront analysis, design, planning, and maybe contract negotiations as we move and probably in many steps to the desired outcome.  In terms of the agile manifesto we have a lot of value on the items on the right which is less agile and more waterfall.  Nothing in the agile manifesto says don’t do waterfall by the way – but it does say be as agile as you can be.

Not everything is obvious via the UI, documented, in someone’s memory, or can be accessed in anyway.  We need to access that hidden information buried inside the code.  Legacy software is also a completed document in its own right – it contains many lessons learned and algorithms that can be modified to fit the new technology.  Starting again and throwing away the legacy system when finished could be a very bad decision.

Many avoid the subject of reengineering but how many of today’s applications start off as a new idea compared to something already existing?  Software reengineering is a very valuable skill for a software engineer to have.

Reverse Engineer

Reverse engineering is the analysis of the existing system and development of high-level plans and designs – the existing system is a very detailed:

  1. functional specification
  2. requirements document
  3. working software

The primary purpose is:

  1. capturing requirements
  2. finding any lost or hidden information, and logic
  3. high level solutions

It is also a feasibility study should/can we

  1. modernize existing code?
  2. migrate to a new system?

If we decide to migrate to a new system then what is the best approach?

  1. The big bang – write a completely new replacement system that we begin using all at once.
  2. Migrate incrementally by modularization – identify modules (if any). We modify legacy code so it is organized into modules where each module represents a function or business area – combine related code parts in modules. We create integration/interface points between these modules so when we introduce a module that is part of the new system, we can redirect those integration points to it. If we are really clever, we use “feature switches” so we can switch between old and new without downtime.

We need to review and document what we will:

  1. Keep
  2. Remove
  3. Improve

We need to look at impact and risk and identify

  1. integration points
  2. interdependencies
  3. interfaces

Do we need to change or normalize the data?

  1. Determine if the interface and relationship with the data permit migration incrementally by modularization or forces a “big bang” approach.  We may need complementary strategies such as migration one region at a time to enable migration incrementally.
  2. Can we use the same database for legacy and reengineered concurrently?  If yes, we may need to make modifications to both legacy and reengineered applications to synchronize with the database as we go.  If no we need a data migration strategy and plan.
  3. If we are migrating incrementally by modularization and require a new database then how do we synchronize data between legacy and reengineered databases?

Forward Engineer

From the high-level plans and designs created in “Reverse Engineering” create low level plans and designs – especially if migrating incrementally by modularization

  1. Database(s)Database designs, migration, synchronization, concurrent support strategies.  Unlike a new product this is live and we have real data that we must not lose.  We have an opportunity to create this right first time – as we have lessons learned from the legacy application. There is little room to evolve a design.
  2. ApplicationObject Orientated Design – design patterns, and application frameworks.  This is not something new, we have an existing application and the majority of requirements.  There is less requirement to evolve an architecture because we know most of the requirements.If we have planned to migrate incrementally by modularization then we know what each of our modules are. We also need to plan for feature switches between them – to turn off modules in the legacy application and on in the reengineered application.

From the low-level plans and designs begin the physical implementation.

If we have planned to migrate incrementally by modularization the first step is to modify the legacy code into modules first or in synchronization with the reengineered application.  If we are creating a new database this would normally be created before the UI.

Deciding between agile and waterfall

Posted on by Steve Padraig in Agile, Software Development Manager, Waterfall

A simple IF statement I use to determine which of agile or waterfall is best suited for a particular project.

IF the end picture is not clear & you don’t know how to achieve it:

  • without continuous feedback
  • most variables unknown
  • few if any requirements set
  • has complexity and novelty
  • defined deadline, scope, and budget not critical
  • team is high-performing, self-organized, allowed  great deal of flexibility, and overlapping skillset.

THEN

Agile

ELSE

Waterfall – there is nothing to be agile about i.e. no flexibility and almost everything already defined

END IF

It is a similar method to the agile manifesto where we value items on the left more.  Many may not like me saying it but the further to the right the closer we are to waterfall.  It could even be waterfall itself.

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

The IF statement shows how far we are to the right before we have started.  There could be many reasons for being nearer to waterfall than agile.

  • A customer already has a comprehensive idea of what they need
  • It is mission critical or scientific making any process fixed, and must practice being risk averse
  • There is simply to much information for the users to provide the software engineers daily that it must be documented
  • It is a re-engineering project and the existing software is in itself comprehensive documentation.

Likewise, the nearer we are to agile there can be many reasons

  • A customer only has a summarized idea of what they want
  • They can afford to be risk tolerant
  • Information can be given on-going and in drips
  • It is a new product, nothing exists yet.

Don’t close your mind

There is nothing in the agile manifesto or principles that says do not do waterfall – and nothing in waterfall that says don’t do agile.  The agile manifesto indicates if you are all the way to the right you are waterfall anyway.  If both are kept as simple a process as they need to be then both can produce great results.

I often think that if we didn’t formally name if we were doing waterfall or agile there would not be so much fuss over it.  Some Software Development Managers & Engineers are believers of one but skeptical of the other…

  • and believe only Agile can work for everything whilst…
  • others believe only Waterfall can work for everything

…but what I have found is not many have used the other – or didn’t use it well.

Conclusion

Great Software Development Managers and Engineers know both waterfall and agile and will select the most suitable.  If they make the wrong selection of tool, process, or technology they will adapt and get on with it.  Great Software Development Managers and Engineers know when to press the reset button and admit when wrong.

Photo by Leah Kelley from Pexels

Mythology versus methodology

Posted on by Steve Padraig in Agile, Software Development Manager, Waterfall

This is not one of those waterfall v agile posts.  This is about debunking the mythology given to each.  I’ve used both extensively and find more similarities than differences, and many things said are just not what I have experienced.

Waterfall

  • Myths
    • Only have one sequence throughout the entire project
    • Customer cannot give feedback until the end of the project
    • Changes cannot be included until the end
  • Facts
    • You can have several sequences overlapping at the same time
    • Customer can provide feedback anytime
    • Changes can be included anytime

I never once worked on a waterfall project with one sequence.  I have worked on a waterfall project 30 months long with sequences 2 weeks long.  Sequences overlapped so when sequence 1 was in implementation stage, sequence 2 was already being designed. The customer gave feedback at the end of each sequence and this was fed back into the project.  Each sequence was iterative until complete.

Waterfall has a history in manufacturing/construction when things were be built in sequential order – basement, floors, then roof.  This is probably where the myth comes from.  When software came along a strict sequential order was no longer required.  We divided into more than one component and joined digital products later.  Manufacturing/construction also followed where many components are prefabricated long distance from where they are joined to together.  Today a roof may be built in a factory before the foundations even laid.

Agile

  • Myths
    • Must use one of the recognized processes such as Scrum and Kanban
    • Iterations must be of constant length
    • You cannot change the contents of an iteration once it is in action
  • Facts
    • Adherence to the agile manifesto and principles and you can add your own simple process to be agile.
    • Use a guide such as 2 weeks for iterations
    • one of the promoted benefits of agile is that you encourage change at any time

The agile manifesto states individuals and interactions over processes and tools – but many of these processes are commercialized versions of agile putting their processes and tools over individuals and interactions – that is how they make money.  I have worked on many agile products where we kept the process simple just using the manifesto and principles.  You only need as simple process as you need. We should own the process not the process own us.

For consistency many argue that all iterations are of the same length but I only ever had a guide for iteration length.  I would add as many user stories to that iteration without exceeding its length.  If that makes the iteration useful then fine, else add more user-stories until it is useful and make that the length of the iteration.   If iterations are constant then this requires the overhead of writing mocks and stubs, and feature switches to spread features over iterations.  The alternative is to change the contents of an iteration.  I always had the teams constantly monitoring progress so they could bring forward highest priority user stories into the current iteration, or push back the lowest priority user stories depending whether time was slack or tight.

Conclusion

Agile and waterfall are two different cultures that clash in software development by those looking for differences. When you focus on similarities not differences they are much closer to each other than you think.

Photo by Vera Arsic from Pexels

Common Problems for Leaders in the SDLC

Posted on by Steve Padraig in Agile, Continual Service Improvement (CSI), DSDM, ITIL and ITSM, Scrum, Service Design, Service Operations, Service Strategy, Service Transition, Software Development Manager, Waterfall
  • Lack of support for multiple streams i.e. production and new development – versioning. Lack of knowledge how versioning works
  • Lack of integration with related projects
  • Development processes overlap into production processes and are not compatible.
  • Lack of clear input and output at each step.
  • Lack of clear ownership
  • Constantly changing release dates – forward and backward
  • Inability to deal with late requests
  • Crunch testing – development overruns into time allocated for testing, testing reduced. Can happen with many processes.
  • Management without technical expertise who override decisions made by technologists, and pressurising technologists to make change to fast

Common Problems for Software Engineers in the SDLC

Posted on by Steve Padraig in Agile, Continual Service Improvement (CSI), DSDM, ITIL and ITSM, Scrum, Service Design, Service Strategy, Service Transition, Software Development Manager, Waterfall

Before getting into defining a Systems Development Lifecycle (SDLC) lets look at some of the main problems experienced by Software Engineers when it is done badly.

  • No time allocated for good design and architecture
    • Code becomes unreadable and unmaintainable
    • Code is disorganised limiting amount of software engineers and/or changes to be active at any time.
    • Makes full and patch releases difficult or impossible without severe downtime or refactoring of code.
    • Simple configuration or system administration requires a code change not variable/parameter change.
    • Makes change extremely difficult – not scalable or object orientated.
      Duplication of solutions i.e. not sharing such as application and hardware servers
    • Makes releases extremely difficult i.e. No thought how to release the project – to many manual processes in a release. Releases take several hours and even days. Build an object to large to get through the door.
  • Lack of necessary documentation or to much unnecessary documentation
  • No or very little enterprise development, testing, or security standards
  • Don’t manage what you don’t understand. Non technical staff getting involved in making technical decisions causes the two most common features in struggling and/or failing technology projects:
    • Pressure from business to implement change fast. Management scheduling without understanding impact and risk
    • Inability of IT to explain and the business to understand the intricate nature of technology projects.