Building a Decision Framework

From Data to Discipline

Having data is necessary but not sufficient. What transforms data into better engineering decisions is a framework — a repeatable, transparent process that connects evidence to choice. Without a framework, even abundant data leads to inconsistent decisions: different teams apply different logic, weight different factors, and document different rationales for similar problems.

A decision framework is not bureaucracy. It is the engineering equivalent of a scientific method — a structured approach that makes decisions repeatable, auditable, and improvable. The same way software engineering moved from "it works on my machine" to CI/CD pipelines, engineering decision-making must move from "we discussed it in the review" to a traceable, evidence-based process.

The Five-Step Decision Process

Step 1: Frame the Decision

Before analyzing anything, define:

What are we deciding? Select a material, choose an architecture, approve a design change, accept a risk. Be specific — vague decisions produce vague analyses.

What are the alternatives? At least two, otherwise it is not a decision. If there is only one option, the decision is "go or no-go," and the analysis is different (risk acceptance, not trade study).

What criteria matter? Performance, cost, risk, schedule, manufacturability, maintainability. Define them explicitly. Criteria that are assumed but not stated will surface during the decision as unstated preferences — the most common source of disagreement.

Who decides? An individual, a review board, consensus? The decision authority must be defined before analysis begins, not after — otherwise the analysis becomes infinite because nobody has the authority to say "enough."

When must we decide? Deadlines constrain how much analysis is feasible. A decision needed in two days gets a screening-level trade study. A decision needed in two months gets exploration-level simulation and sensitivity analysis. Matching analysis depth to decision timeline is an essential skill.

Step 2: Identify Data Sources

For each criterion, identify:

What data already exists? Requirements in the system model, simulation results from previous analyses, test data from qualification programs, operational data from previous generations. Much of the data needed for a decision already exists somewhere in the organization — the challenge is finding and connecting it.

What data needs to be generated? New simulation runs, prototype tests, supplier evaluations, expert assessments. Generating new data takes time and resources — prioritize the data that will most affect the decision outcome (use sensitivity analysis from Lesson 2 to identify decision drivers).

What data is unavailable and requires judgment? Be explicit. A decision that is 80% data-driven and 20% judgment is defensible — as long as the 20% is identified, documented, and the decision-makers understand which parts rest on evidence and which rest on experience.

The system model is the first place to look. MBSE provides structured data that is already connected to requirements, architecture, and verification — reducing the effort to assemble the evidence base for a decision. This is the practical payoff of the investment described in DGTLENG 104 and DGTLENG 201: requirements and architecture captured in the model are not just design artifacts — they are the data infrastructure for decision-making.

Step 3: Analyze

Apply the appropriate level of analysis for each criterion:

Quantitative analysis: Simulation results, parametric calculations, statistical analysis, cost models. These produce numbers with defined uncertainty. Present them as ranges or confidence intervals, not point estimates, unless the uncertainty is genuinely negligible.

Qualitative analysis: Expert assessment, historical precedent, analogy to similar systems. These produce judgments that should be structured — rating scales, Delphi methods, or pairwise comparisons — rather than unstructured opinions.

Risk analysis: Combining likelihood and consequence to assess which alternatives carry more risk. Sensitivity analysis to identify which uncertainties matter most. The simulation pyramid from Lesson 3 guides the fidelity: screening for early discrimination, exploration for trade studies, validation for the selected design.

Present results in a format that enables comparison across alternatives. Decision matrices, spider charts, and trade study tables all work — the format matters less than clarity and traceability.

Step 4: Decide and Document

Make the decision. Then document:

What was decided — the selected alternative and any conditions or constraints on its implementation.

What alternatives were considered — including alternatives that were rejected. Future engineers who revisit this decision need to understand not just what was chosen but what was not chosen and why.

What data supported the decision — traceable to specific model elements, simulation results, test reports, or expert assessments. This is where the digital thread earns its keep: the decision rationale links to the evidence, and the evidence links to the model.

What risks were accepted — every decision involves accepting some risk. Making the accepted risks explicit is essential for downstream engineering. If an alternative was selected despite a lower reliability score because the cost advantage was deemed more important, that trade-off must be visible.

Who approved it — decision authority and accountability. Not to assign blame if the decision turns out poorly, but to ensure that the person with the appropriate authority and context made the call.

This documentation is not a bureaucratic afterthought — it is the most durable output of the decision process. Designs change, requirements evolve, teams turn over. The decision rationale is what enables future engineers to understand the current design, revisit decisions when conditions change, and avoid re-litigating settled questions.

Step 5: Monitor and Revisit

A decision is a prediction: "Given what we know now, this is the best choice." As new data arrives — from testing, operations, or further analysis — the prediction should be checked.

Are the assumptions still valid? The decision was based on a thermal margin of 8 degrees C. The validation simulation shows 5 degrees C. Does this change the decision?

Is the system performing as predicted? Operational data from deployed units shows higher failure rates than expected. Does this change the maintenance strategy? The next-generation design?

Have new alternatives emerged? A new material, a new component, a new architectural pattern that was not available when the original decision was made. Monitoring the landscape for better options is part of the decision lifecycle.

Not every decision needs active monitoring. Routine decisions with low consequence can be decided and closed. High-consequence decisions — architecture selection, technology commitment, safety-critical design choices — should have explicit revisit triggers: defined conditions under which the decision will be re-evaluated.

Decision Anti-Patterns

Even with a good framework, human cognitive biases can corrupt the process. Recognizing anti-patterns is the first step to countering them.

HiPPO (Highest Paid Person's Opinion)

The most senior person in the room declares the answer. Data exists but is not consulted — or is consulted selectively to support the predetermined conclusion. The meeting is structured as a decision review, but the decision was made before the meeting started.

HiPPO works when the senior person has deep, relevant experience in the specific domain. It fails when the situation is novel, when the data contradicts intuition, or when the senior person's experience is from a different context. The higher the stakes and the more novel the situation, the more dangerous HiPPO becomes.

Analysis Paralysis

The team keeps analyzing, requesting more data, running more simulations — but never decides. New questions surface at every review. The analysis expands in scope rather than converging on an answer. Usually driven by fear of making the wrong choice, or by organizational culture that punishes bad outcomes more than it rewards good decisions.

The fix: agree on decision criteria and their evaluation methods before analysis begins. Define what result would be sufficient to decide. Set a decision deadline. When the criteria are evaluated and the deadline arrives, decide — even if the data is imperfect. Waiting for perfect data is itself a decision (to delay), and delay has its own costs.

Confirmation Bias

The team has a preferred option — often the one championed by the most influential member, the one that requires the least change, or the one that was informally selected before the formal process began. Data that supports the preferred option is accepted. Data that contradicts it is questioned, reanalyzed, or dismissed.

Confirmation bias is insidious because it operates within the framework — the team goes through the motions of a data-driven process while unconsciously steering toward a predetermined outcome. Counters include assigning a "red team" to argue for the non-preferred alternative, requiring blind evaluation (scoring alternatives before revealing which is which), and mandating that contradictory data be addressed explicitly in the rationale.

False Precision

Scoring alternatives to two decimal places when the underlying data has 20% uncertainty. Computing a weighted total of 7.43 versus 7.41 and declaring a winner. The precision of the output exceeds the confidence of the inputs, creating an illusion of objectivity that masks genuine uncertainty.

False precision is seductive because it feels rigorous. A spreadsheet with six decimal places looks more scientific than one with rough estimates. But the precision is cosmetic — it does not reflect the actual quality of the information. The counter: propagate uncertainty through the scoring. If the input data is uncertain by plus or minus 20%, the output scores should reflect that range. If two alternatives score within each other's uncertainty bands, they are effectively tied — and the decision must rest on factors beyond the quantitative score.

The Role of Human Judgment

A decision framework does not replace human judgment. It structures and supports it. Judgment is essential at every stage:

Framing: Deciding what to decide, what criteria matter, and what alternatives to consider requires judgment that no data can provide.

Weighting: How much does cost matter relative to performance? How much does schedule matter relative to risk? These are value judgments that reflect program priorities, not technical truths.

Interpreting: When the data is ambiguous, when the sensitivity analysis shows the decision is fragile, when the top two alternatives are effectively tied — judgment breaks the tie.

Context: Data describes what is. Judgment incorporates what ought to be — organizational values, strategic direction, stakeholder needs that are not captured in any criterion.

The goal of data-driven decision-making is not to eliminate judgment. It is to give judgment better data to work with, a structured process to work within, and a traceable record to learn from.

When Data Is Not Enough

Some decisions cannot wait for data. Some decisions involve unprecedented situations where no relevant data exists. Some decisions involve ethical, political, or strategic considerations that transcend quantitative analysis.

In these cases, the framework still helps — framing the decision, identifying what is known and what is unknown, documenting the rationale — even if the analysis step produces "insufficient data, decision based on judgment." A well-documented judgment call is infinitely more valuable than an undocumented one, because future engineers can understand the basis, assess whether conditions have changed, and decide whether to revisit.

Assessment