Impact Estimation Tables
Impact Estimation Tables (IET) are a tool for impact/effort scoring. They can be used to compare alternative design ideas, to analyse key risks, to estimate and to help with planning, prioritisation and tracking iterative delivery work.
Tom Gilb introduced IET for “analysing design ideas in relation to objectives and related costs” a part of his Evo process (a precursor to modern Agile methodologies), as a more mathematically rigorous alternative to Quality Function Deployment (QFD). In Competitive Engineering, Gilb suggests that impact estimation tables should be used to evaluate any design idea that can have a significant impact (5% or more) on any critical performance or cost requirement of a project.
- The Structure of Impact Estimation Tables
- An Example Impact Estimation Table
- How Impact Estimation Tables differ from ICE and other methods?
The Structure of Impact Estimation Tables
An impact estimation table shows how various potential solutions (“design ideas” in the IET terminology) relate to each other in the context of fulfilling an objective (“performance requirement” in the IET terminology).
In an impact estimation table, different alternative ideas that could achieve the same outcome are listed in columns, and IET scores are listed in rows.
The table begins with a piece of information common for all the options - the problem or the opportunity that will be addressed by the proposed options.
Gilb recommends first deciding how the problem will be measured (“scale”) with a specified unit (such as minutes, dollars, number of users etc). After that, it’s important to specify the “performance requirement change”, by stating the minimum acceptable and the target points on that scale. For some good ways of calculating that, check out the Product Opportunity Assessment questionnaire and the QUPER model.
The following rows contain the predicted impact on the expected scale for each individual design idea and by the error estimate for the impact (“uncertainty” in the IET terminology). First list the estimates in the original scale units, then normalize them to a “percentage on the way to the goal”, for easier comparison later.
The next three rows list the evidence justifying the predicted impact and error estimate, along with the source of that evidence and the credibility rating of the source. The credibility ratings and safety margins for uncertainty need to be set before creating an impact estimation table, and should in general be standardized for a product or team to ensure consistency of scores. Some good potential options are the Confidence Meter model or scores based on Levels of Evidence.
The final three rows contain the estimated development cost, the ratio of the predicted percentage impact to cost, and the adjustment of that score based on the credibility of the evidence.
The credibility-adjusted value is the final score for an idea, and can be used for relative comparison between ideas and for prioritisation.
An Example Impact Estimation Table
Imagine an application that helps event organizers sell tickets for music events. The product managers might want to make it faster to set up ticket sales for larger events.
Three potential options for doing that would be:
- Pre-configured Event Templates: Provide pre-configured templates for common event types (e.g., concerts, festivals) that automatically fill in key details like ticket categories, pricing tiers, and seating arrangements.
- Automated Seat Mapping: Use automated tools to generate seating arrangements and assign tickets based on venue specifications, minimizing manual seat mapping efforts.
- Live Event Support Chatbot: Implement a chatbot that guides organizers through the setup process, answers common questions, and provides real-time support, reducing the need for manual assistance.
| Pre-configured Event Templates | Automated Seat Mapping | Live Event Support Chatbot | |
|---|---|---|---|
| Expected Improvement Range: 20 min reduction (30 min to 10 min) | |||
| Scale Impact | 12 min reduction | 8 min reduction | 10 min reduction |
| Scale Uncertainty | ±3 min | ±2 min | ±5 min? |
| Percentage impact | 60% | 40% | 50% |
| Percentage uncertainty | ±15% (3 out of 20 minutes) | ±10% | ±12.5% |
| Evidence for Predicted Impact | Analysis of setup times using similar templates in beta tests | Time savings observed in similar automated mapping tools | Case studies of reduced support time with chatbot use |
| Source | Internal beta testing reports | Industry benchmark data | Third-party case studies |
| Credibility | High (0.8) | Medium (0.6) | Low (0.2) |
| Estimated Development Cost (in $1000) | 30 | 60 | 100 |
| Performance to Cost Ratio | 2 (60%/30K, 2% reduction per $1000) |
0.66 | 0.5 |
| Credibility Adjusted score | 1.6 | 0.4 | 0.1 |
Gilb suggests documenting guesses and figures produced with insufficient information (for example, marking them with a question mark), noting that an impact estimation table “is there to help you see potential problems, not to cover them up”. Risks and assumptions should also be reported, for example in the footnotes of the table.
How Impact Estimation Tables differ from ICE and other methods?
Compared to similar impact/effort scoring methods, Impact Estimation Tables tend to be a more complex, strict and rigorous. IET includes criteria such as risk, uncertainty estimates and performance/cost ratios which simpler methods usually ignore. Applying the IET requires collecting a lot of data, performing in-depth analysis and applying strict mathematical formulas to compare ideas. In that way, IET suffer a lot less from Mathiness than simpler alternatives. Scoring methods such as ICE are simpler and faster, but are not as rigorous.
The strictness of IET can be a mixed blessing, where on one hand the method forces people to perform more research and explore alternative ideas in depth, but on the other hand precision and mathematical strictness can lead to a feeling of accuracy and a false sense of comfort. The results may be precise, but if the initial estimates are wrong, they will be widely inaccurate. The tables do include two error-estimation factors (uncertainty and credibility), but these numbers can be subjective and difficult to compare.
In general, IET are better suited to in-depth analysis and multi-criteria decision-making for complex and high-risk delivery. Simpler methods are better for quick iterative delivery where the risk of selecting the wrong option can be reduced by A/B testing or product experiments.
Learn more about Impact Estimation Tables
- Understanding Complex Technology Quantitatively: Impact Estimation Tables, From INCOSE International Symposium 9 by Tom Gilb (1999)
- Competitive Engineering: A Handbook For Systems Engineering, Requirements Engineering, and Software Engineering Using Planguage, ISBN 978-0750665070, by Tom Gilb (2005)