Business Analysis Benchmark – The Path to Success
The Business Analysis Benchmark is a large scale survey effort designed to assess the link between an organization’s maturity in requirements definition and management and project outcome. This year’s theme is The Path to Success; the study presents detailed findings on the impact of business requirements maturity and analyzes the strategies and tactics needed to implement enhanced requirements maturity.
| “This survey is a testament to the need for investing in your requirements process to deliver value to your stakeholders.” Scott Hebner, Vice President Marketing and Strategy, IBM Rational Software |
The Requirements Maturity Model (RMM) is a means to benchmark an organization’s effectiveness in requirements definition and management by looking at maturity in six underlying capabilities. Like similar standards-based models, it classifies companies based on observed, tangible competency in each capability to make an objective assessment of overall maturity. Using this approach, the report found:
- Requirements maturity improvement is highly correlated with improvement in development effectiveness.
- Requirements maturity cannot be changed through continuous focus on only one underlying capability.
- High requirements maturity companies can be found amongst the followers of many different approaches to development such as Agile, Iterative, Plan Driven (Waterfall),and Prototyping/Visualization centric methods.
The above findings validate the Requirements Maturity Model as a mechanism for identifying the impact of poor requirements practices on companies, quantifying the performance change expected for a particular organization’s situation, and, diagnosing the changes needed should a company choose to pursue a path of improvement. This report identifies both the strategy and tactics of enhancing requirements definition and management maturity. The statistics presented in the Business Analysis Benchmark not only debunk a number of commonly held beliefs about development effectiveness, they show that the average organization wastes a large proportion of its IT development budget due to poor requirements maturity. To be clear, 75% of organizations surveyed waste over one in three dollars spent in IT development and implementation annually as a result of to poor requirements maturity. These findings detail key issues and actions needed to recapture this wastage.
Key Findings of the Business Analysis Benchmark include:
- Requirements maturity has a strong positive correlation to EVERY major measure of development efficiency assessed. On-time performance, on-budget performance, on function performance,
“This [report] was extremely helpful to me, not only to understand the findings relating to my current situation, but also what CEO and CIOs are interested in.”
Carol Deutschlander,
Home Hardware Stores Limitedoverrun magnitudes for each of the above, and project success rates all improve as requirements maturity increases. On average, performance virtually doubled on each of these metrics as organizations progressed from using an ad-hoc approach for requirements definition and management to having institutionalized and consistent competency in all capability areas.
- Average on-time performance of technology projects increased by 161%.
- Time overruns on projects reduced by 87%.
- Average on-budget performance for technology projects improved by just over 95%.
- Budget overruns reduced by just under 75%.
- Percentage of projects that deliver the functionality needed by the business rose by just over 75%.
- Average functionality missed dropped by approximately 78%.
- A total of 74.1 per cent of survey respondents were classified as immature Level 1 or Level 2 organizations (where the highest maturity Level is 5). These organizations waste 39% and 34% respectively of their development budget due to poor requirements definition and management maturity. This wastage, due to poor requirements maturity, will increase to over 50% of IT spending on development and a significant proportion of the maintenance budget in certain circumstances.
- Poor requirements definition and management maturity undermine organizational competitiveness. Organizations with poor requirements maturity expend far more time, budget, and management effort to achieve the same results as organizations with high maturity. For example, organizations with low requirements maturity achieve the business objectives of a project initiative a mere 54% of the time while taking 35% more time to achieve this poorer result. This impact may be so significant over time that it shifts fundamental financial performance metrics such as Return on Assets (ROA).It was found that Level 4 companies, on average, outperform the ROA of their peer group competitors by 10%.
- While this report discusses and busted a number of commonly held beliefs about requirements and development efficiency, two issues garnered significant attention and support from the report’s external review panel:
CIO’s cannot simply attempt to hire great analysts and expect the problem of poor requirements to go away. In fact, lower skilled people in a high requirements maturity company significantly outperform highly skilled people in a low requirements maturity company.
|
“I’ve worked carefully through the Benchmark Study. It’s terrific stuff — some of the conclusions are almost iconoclastic, and yet they make tremendous sense once you analyze them. And the RMM is an excellent tool — of course it does and should heavily parallel CMM / CMMI, but it also provides tremendous value added as you’ve applied/customized it to Business Analysis practice.”Senior Requirements Specialist Major Property & Casualty Insurance Company |
Agile, Waterfall, Iterative, Prototyping/Visualization have immaterial performance differences for any given level of requirements definition and management maturity. There is a raging debate amongst development methodologists, each espousing one method over another. This study finds that changing development methods – in the absence of also improving requirements competence in the areas of process, techniques, staff, technology, organization and deliverables – only nominally improved or reduced overall success rates on projects. Excellent results have been achieved with all these approaches, and the findings of the Business Analysis Benchmark do not endorse any one method over another. The key issue for readers: the overall level of requirements maturity has a MUCH greater effect on project outcome than the development method selected.
The Business Analysis Benchmark describes the issues and impacts of each level of the organization, and the role each plays in moving a company forward along the path of maturity. This report has a preface that describes the survey, maturity model, and basic facts surrounding the impact of requirements maturity on project outcome. The remainder of the report is organized along the lines of a readership group, discussing the key findings as they relate to:
- The CEO: how does requirements maturity impact overall organizational competitiveness?
- The CIO: how does IT Leadership approach the major issues in making requirements definition and management change?
- The PM and BA Leadership: what is the effectiveness of various paths of change, and what are the required activities to bring improvement? In addition to this content, IAG has also asked a series of external reviewers to comment on survey findings. Some of these insights are captured in the call-out boxes in this article.
The Survey – How it was Conducted
Last year, over 22 million business and IT professionals in 80 countries, and using 10 languages, benefited from the statistics generated by the Business Analysis Benchmark.
This year’s survey theme – The Path to Success – identifies a roadmap for maturing requirements definition and management practices. This study is about getting repeatable success on strategic IT projects.
In Q2 of 2009, just under 550 companies chose to participate in the Business Analysis Benchmark – Path to Success survey, leading to 437 qualifying responses. This survey was designed with the intent of assessing the link between an organization’s maturity in requirements definition and management and project outcomes. The Business Analysis Benchmark statistics only include respondents that met the following three criteria:
- The company spends over $1 million annually in application development (net of hardware) or software implementation.
- The individual must have experience with business requirements and project management where net new functionality is added to the business.
- The company must have run at least four projects in excess of $250,000 in the last 12 months.
These criteria ensured that only experienced professionals with knowledge of requirements definition and management issues would be included in survey results. The results are weighted toward medium and large sized commercial companies, in North America. The sampling is summarized below:
|
Position Executive: Head of IT, CIO, Head of Development, Line of Business Executive Number of Employees in Company 1‐99
Energy, Resources & Utilities
United States |
% Respondents 12.2% % Respondents 6.2% % Respondents 3.9% # Respondents 233 |
Don’t forget to leave your comments below
Keith Ellis is the Vice President, Marketing at IAG Consulting (www.iag.biz) – and author of the Business Analysis Benchmark – where he leads the marketing and strategic alliances efforts of this global leader in business requirements discovery and management. Keith is a veteran of the technology services business and founder of the business analysis company Digital Mosaic which was sold to IAG in 2007. Keith’s former lives have included leading the consulting and services research efforts of the technology trend watcher International Data Corporation in Canada, and the marketing strategy of the global outsourcer CGI in the financial services sector.
For access to and to download your free copy of the full Business Analysis Benchmark study, please click on www.iag.biz.