Measuring DE Value

The Measurement Problem

Digital engineering initiatives cost real money — tools, training, infrastructure, organizational change. At some point, someone asks: "Is this working? What are we getting for the investment?"

Most organizations answer with the wrong metrics. They report how many models were created, how many engineers were trained, how many tool licenses were deployed. These are activity metrics — they describe what was done, not whether the doing created value. Activity metrics are easy to collect and reassuring to report, but they cannot answer the question that matters: did digital engineering make our engineering better?

Measuring DE value requires understanding the hierarchy of metrics and choosing measurement strategies that connect DE activities to business outcomes.

The Metrics Hierarchy

Level 1: Activity Metrics

Activity metrics measure what the organization is doing. They are necessary for tracking adoption but insufficient for demonstrating value.

Examples:

• Number of system models created
• Number of engineers trained in MBSE
• Number of tool licenses deployed
• Number of model elements (blocks, requirements, interfaces)
• Percentage of programs using model-based practice

What they tell you: Whether people are using the tools and methods. If activity metrics are low, adoption has stalled and nothing else matters until that is fixed.

What they do not tell you: Whether the activity is producing value. A program can create a large model that duplicates existing documents without improving any engineering outcome. High activity metrics with no downstream improvement means the organization is doing more work for the same result.

The trap: Activity metrics are seductive because they always go up once tools are deployed. Reporting rising activity metrics to leadership creates a false sense of progress.

Level 2: Output Metrics

Output metrics measure the direct products of DE practice — the improvements in engineering activities that DE enables.

Examples:

• Time to complete a design review (comparing model-based reviews to document-based reviews)
• Number of defects detected during model-based analysis (before prototyping)
• Time to answer an impact analysis question ("what is affected by this change?")
• Number of requirements traced to both design elements and verification methods
• Time to generate compliance documentation from the model

What they tell you: Whether DE practices are improving the efficiency and quality of specific engineering activities. These metrics show the mechanism through which DE creates value.

What they do not tell you: Whether the improved activities are actually improving program outcomes. A faster design review is only valuable if it leads to better designs or faster delivery.

How to collect: Requires measuring the same activity with and without DE. The ideal approach is a controlled comparison — the same organization performing similar work on comparable programs, one using DE and one using traditional methods. The practical approach is a before-and-after comparison — measuring the same activity before and after DE adoption, with appropriate caution about confounding factors.

Level 3: Outcome Metrics

Outcome metrics measure the engineering results that DE outputs enable — the actual improvements in engineering performance.

Examples:

• Percentage reduction in requirements-related rework (requirements defects caught in the model rather than in integration test)
• Reduction in integration defects (interface mismatches detected in the model before hardware arrives)
• Time saved in design iteration cycles (faster trade studies, automated analysis)
• Improvement in design review effectiveness (percentage of reviews that result in actionable decisions rather than "come back with more information")
• Reduction in documentation errors (reports generated from the model are consistent with the design)

What they tell you: Whether DE is improving engineering performance. These are the metrics that demonstrate real value — the engineering is getting better, not just different.

What they do not tell you: Whether the engineering improvement is translating to business impact. Reducing rework by 30% is valuable, but if the program is still over budget for other reasons, the business case remains incomplete.

How to collect: Requires longitudinal data — tracking outcomes over time as DE practices mature. Baselines must be established before DE adoption begins, which means the measurement framework must be designed at the start of the initiative, not retrofitted after the fact.

Level 4: Business Impact Metrics

Business impact metrics measure the organizational value created by improved engineering outcomes — the metrics that executives and customers care about.

Examples:

• Return on investment (total DE cost vs. quantified value of improved outcomes)
• Reduction in program schedule overruns attributable to earlier defect detection
• Reduction in cost of rework and scrap
• Time to market improvement for new products
• Competitive advantage in proposal win rates (demonstrating DE capability to customers)
• Customer satisfaction with engineering deliverables

What they tell you: Whether DE is creating value at the organizational level — the bottom line.

Why they are hard to collect: Business outcomes are influenced by many factors beyond DE. Attributing a schedule improvement solely to DE when the program also changed its staffing, requirements, or management approach is unreliable. The further up the metrics hierarchy you go, the more confounding factors intervene between DE and the measured result.

How to collect: Requires careful attribution analysis — comparing programs with and without DE while controlling for other variables. Case studies with explicit before-and-after analysis. Financial modeling that quantifies the cost of the problems DE prevents (rework costs, integration failure costs, documentation error costs).

Why Organizations Measure the Wrong Things

It Is Easier to Count Inputs Than Outcomes

Activity metrics are collected by tool administrators. Output metrics require instrumenting engineering processes. Outcome metrics require longitudinal tracking. Business impact metrics require financial analysis. The difficulty increases at each level, and most organizations stop at the level they can collect without effort.

The Measurement Framework Was Not Designed Upfront

If you do not establish baselines before DE adoption, you cannot measure improvement after. The most common failure: an organization deploys DE across a program, achieves what it believes are good results, and then discovers it has no pre-DE baseline to compare against. The improvement is anecdotal, not quantified.

Activity Metrics Always Look Good

Once tools are deployed and training is delivered, activity metrics rise. This creates a positive reporting cycle that masks the absence of deeper measurement. By the time someone asks "but is it actually helping?", the window for establishing baselines has passed.

Outcome Metrics Take Time

Engineering programs run for years. Rework reductions, defect rate improvements, and schedule impacts take multiple program phases to manifest. Organizations under pressure to demonstrate value quickly gravitate to metrics that are available immediately — which means activity metrics.

Building a Measurement Framework

Step 1: Define What Success Means Before Starting

Before the DE initiative begins, define success at each level of the hierarchy. What activities do you expect to see? What outputs should improve? What outcomes are you targeting? What business impact do you anticipate?

Step 2: Establish Baselines

Measure the current state of the outputs and outcomes you plan to improve. How long do design reviews take today? How many requirements defects are found during integration test? What is the cost of rework on current programs? These baselines are irreplaceable — without them, post-DE measurement has no reference point.

Step 3: Instrument the Process

Build measurement into the DE workflow rather than treating it as a separate reporting exercise. Automated metrics collection from tools (model quality scores, traceability coverage). Time tracking built into review processes. Defect tracking that categorizes by detection method (model vs. test vs. field).

Step 4: Report at the Right Level for the Audience

Engineers care about output metrics (is this making my work better?). Program managers care about outcome metrics (is this improving my program?). Executives care about business impact (is this a good investment?). Report the right metrics to the right audience.

Step 5: Iterate the Framework

As the DE initiative matures, the metrics that matter change. Early on, activity metrics confirm adoption. As adoption stabilizes, output metrics confirm the mechanism is working. Over time, outcome and business impact metrics become the primary measures. The framework evolves with the initiative.

Assessment