Evaluating Impact of Experience in Architectural Design Decision-Making Techniques: An Experimental Study

The software engineering literature describes several decision-making techniques for architectural design. However, the impact of the experience of those who use these techniques on their efficacy has been little explored. This article addresses this research gap through an experimental study to evaluate the impact of experience of software architecture decision-making team members on the efficacy of TaSPeR (Tactics Selection Poker), a software architecture tactics selection technique based on Planning Poker. A set of 24 IT professional developers was divided into two “expert” teams and two “novice” teams (of six people each); each team had to solve two scenarios, one using TaSPeR and one using an ad-hoc technique; the scenarios were crossed to mitigate possible learning effects. The study compared the effectiveness of teams in terms of precision, recall, and accuracy (of selected tactics versus a ground truth); accuracy was found best. Initial results suggest that TaSPeR improves the accuracy of novice teams but hurts the accuracy of expert teams. This result is quite unexpected and begs replication with even larger populations of IT professionals (no easy task). If the results are confirmed, the question that will rise is: if consensus techniques are so good to estimate, why would they hurt design decision-making by expert teams?