The Agile Revolution: Get ahead with continuous improvement and innovation

Despite the rising complexity of software systems, the rapid technology evolution and the lack of adequate numbers of qualified developers, software vendors are under pressure to produce high-quality software in short time scales at the lowest possible cost. As a result, software companies must adopt the most effective software engineering principles, which optimize the way they organize the different activities needed for constructing and evolving a software system. Conventional Waterfall-like Development (WD) methodologies have conspicuous disadvantages that render them inappropriate for the majority of modern enterprise projects. In particular, they are very inflexible in incorporating the ever changing requirements of the customer, while providing very poor progress visibility, especially in early stages where only documents rather than software are produced. Consequently, they are associated with high risks, which are reduced only during the rather late integration phase. For example, in an 18-months enterprise web site project, the actual product is likely to become visible six or even eight months following the project’s start.

In order to alleviate the problems of WD methodologies, software vendors and integrators are increasingly resorting to iterative Agile Software Engineering (ASE) methodologies, which gradually produce evolutionary and improved versions of a software system. ASE methodologies adhere to the following principles: They put individuals and interactions over processes and tools, they prioritize production of working software over documents, they emphasize collaboration with the customer rather than rigorous negotiation of contracts and they embrace change.

These principles facilitate software system stakeholders to resolve critical risks before engaging in large investments, while enabling objective assessment of progress based on measurable iterations that correspond to working versions of functionalities requested by the customer.  These principles also ease the collection of user feedback as a means of implementing improvements and changes over the initial set of requirements, along with continuous compliance checking.

Agile Methodologies in Practice

From a practical perspective agile methodologies and their iterations are built around four parameters of a software project:

• Scope: Defines what has to be implemented and is usually measured based on a number of user stories (i.e. use cases). This number has to correspond to use cases of analogous granularity in terms of implementation effort.
• Resources: Refers mainly to the human resources (e.g., developers, architects, testers) allocated to the project’s implementation. Software projects tend to be labor intensive and hence the available resources define the number of developers that can be allocated to the implementation.
• Time: Refers to the timeframe for the project’s implementation, measured in weeks or months depending on the project’s scale and duration.
• Quality: Refers to the quality of the software to be produced, which should be never compromised.

ASE methodologies advocate that these four parameters should be visible to all project stakeholders, including the customer, the project sponsor and the development team. Furthermore, they are used to control the flow of the project and the respective iterative processes. In particular, ASE methodologies control three of the four variables in order to achieve a target for the forth one. For example, in order to shorten the project delivery time, project stakeholders may agree to reduce its scope or increase the available resources. Likewise, in order to achieve a larger scope, project stakeholders’ could increase the budget or prolong its duration. In this way, agile methodologies enable the quantitative planning and tracking of the project in a credible and transparent way, which increases stakeholders’ trust.

Any software team adopting an agile methodology works on the basis of periodic cycles (e.g., weekly or bi-weekly) during which the team is able to complete the production of a certain number of user stories (i.e. a “project velocity”).  Following the first couple of iterations, the team is able to provide a good estimation of its velocity, which can facilitate planning, project management, budgeting and other processes that are essential to a system’s implementation.  In order to maximize a team’s productivity agile teams adopt and apply a range of good practices, including continuous integration, test-first programming (i.e. development of tests prior to the code), continuous and automated testing, establishment of automated and responsive build processes, root-cause analysis in order to analyze errors and more.

Agile Software Engineering for Innovation Cycles

ASE methodologies are for several reasons ideal for implementing products as part of innovation cycles. First, innovative products are associated with instability in their initial set of requirements, asking for an approach that is flexible to change. Second, innovation cycles are by their nature evolutionary and leverage user feedback as a means of continuous improvement. Third, agility is required in order to be responsive to competitive pressures and market feedback. Last, agile methodologies increase developers’ engagement and motivation, which is a key to the execution of innovation cycles. ASE methodologies are therefore adopted by the vast majority of teams that are implementing novel software products, including most giant vendors. It should be noted that the popular “lean startup” practices, which are used for product developments by numerous startups around the globe, are fully in-line with ASE. Indeed, lean development practices involve iterative product development on the basis of a “build-measure-learn” loop, which boosts continuous improvement of the product. In this context, the famous MVP (Minimum Viable Product) can be a considered as the starting iteration of the agile approach.

Agile Frameworks and Tools

Nowadays, there is a host of frameworks for agile software development frameworks, including Adaptive software development (ASD), Agile Unified Process (AUP), Extreme programming (XP), Kanban and Scrum. These frameworks feature many similarities, but also differences in aspects such as the organization of iterations, the criteria for measuring the success of each iteration, the roles within the development team, the frequency and organization of meetings, the continuous improvement criteria used and more. Each of the methodologies is supported by a variety of tools that facilitate processes such as integration, automated testing, automated deployment, release planning, team collaboration and more. Recently the Scaled Agile Framework(SAFe) is extensively used as a knowledge base of successful patterns for agile and lean development of software systems. Teams using SAFe get tangible benefits in productivity, quality and faster time to market, which is the reason why the majority of Fortune 100 U.S companies work with it.

Being Aware of the Challenges

While the notions of cycles, iterations and velocities are commonplace for the giant vendors, they are still poorly understood by smaller vendors and their customers (including Small Medium Businesses (SMBs)). A proper planning of the project, including an apt organization of the development and the management team are key prerequisites for the adoption and success of agile methodologies. Likewise, the selection of proper frameworks and tools by both customers and software vendors are essential elements to the ultimate success. Overall, there are still many opportunities for increasing productivity, reducing costs and perfecting the quality of software systems. Project managers, IT managers and CIOs should be proactive and actively participate in the agile revolution.