Empirical Studies on Software Quality Mythology

Posted by Gavin Terrill on Oct 08, 2009

Community

Architecture,

Agile

Topics

Software Craftsmanship

Tags

Quality

Microsoft Research has released a summary of the results of empirical studies examining software engineering myths. The work, conducted by Nachi Nagappan, measures the impact on quality that common software engineering practices actually have. The analysis reveals:

• More code coverage in testing doesn't necessarily correlate with a decrease in the number of post-release fixes required, citing many other factors that come into play.
• TDD improves quality but takes longer (pdf): "What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer."
• The use of assertions and code verification decreases bugs. Further, "Software engineers who were able to make productive use of assertions in their code base tended to be well-trained and experienced, a factor that contributed to the end results."
• Organizational structure has a profound impact on quality: "Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent."
• The impact of a team being distributed has a negligible impact on quality.

These research findings are now being used by Microsoft development groups, including helping with risk-analysis and bug-triaging for major projects such as Windows Vista SP2.