Automated Verification and Validation

The Milestone Trap

Most engineering organizations treat verification and validation (V&V) as a phase. The design matures. A milestone approaches — Preliminary Design Review, Critical Design Review, Test Readiness Review. The team scrambles to verify that requirements are met and validate that the system behaves as intended. Problems discovered at this stage are expensive to fix because the design has solidified and downstream artifacts depend on it.

This is not V&V. This is inspection at the end of a production line. The defect has already been produced. You are just deciding whether to ship it.

Continuous V&V inverts this model. Verification runs throughout development, not at milestones. Every model change triggers the relevant checks. Defects are caught when they are introduced — when the engineer who made the change is still working on that area and the cost of correction is minimal. The milestone review becomes a summary of what the continuous process already confirmed, not a discovery exercise.

The shift from milestone-based to continuous V&V is not just a scheduling change. It is an architectural change. The pipeline must support verification as a persistent, automated activity — not a batch job someone runs before a review.

Model Rules: Linting for Engineering Models

Software developers use linters — tools that check code against a set of rules — to catch problems before the code ever runs. A linter flags unused variables, inconsistent formatting, potential null pointer dereferences. It enforces the team's coding standards automatically, so code reviews can focus on design and logic rather than style violations.

Engineering models need the same treatment. Model rules are the engineering equivalent of linting: automated checks that verify the model conforms to the organization's standards and constraints.

Structural Rules

These verify the model's internal consistency. Every requirement must be allocated to at least one design element. Every interface must have matching definitions on both sides of a connection. Every function must be realized by at least one component. Parametric constraints must not be violated. These are the rules you would check manually during a model review — but checking them manually is slow, inconsistent, and prone to oversight.

In an MBSE context, structural rules operate on the model's relationship graph. When a requirement is allocated to a function, and that function is realized by a component, a structural rule verifies that the entire chain is intact. Breaking any link in that chain — deleting a function without reallocating its requirements, modifying an interface on one side of a connection without updating the other — triggers a rule violation.

Domain Rules

These encode engineering domain knowledge. A thermal interface material must have a conductivity value within a physically plausible range. A structural safety factor must meet the minimum specified by the applicable standard. A power budget allocation cannot exceed the available power supply capacity. Domain rules catch values that are syntactically valid in the model but physically nonsensical or contractually non-compliant.

Completeness Rules

These verify that the model contains all required information for its current maturity level. At the concept phase, a component needs an estimated mass. At the preliminary design phase, it needs a mass with tolerance and source reference. At the detailed design phase, it needs a mass from analysis with validation against test data. Completeness rules adapt to program phase — they do not penalize a concept-phase model for lacking detailed analysis, but they flag a CDR-phase model that still relies on engineering estimates.

Cross-Domain Consistency Rules

These are the hardest to define and the most valuable. They verify consistency across discipline boundaries: the mass used in the structural model matches the mass in the system-level mass budget. The thermal dissipation assumed in the thermal model matches the power consumption in the electrical model. The control bandwidth in the dynamics model is compatible with the sensor response time in the avionics model.

Cross-domain rules run against model relationships that span discipline boundaries — exactly the relationships that the digital thread is supposed to maintain. When these rules pass, you have evidence that the disciplines are working from consistent data. When they fail, you have caught a cross-discipline inconsistency before it propagates into analysis results.

Scenario Libraries

Individual rules verify individual properties. Scenario-based verification tests the system's behavior across defined operating conditions. A scenario library is a curated, versioned collection of test scenarios that the pipeline executes automatically.

What Goes in a Scenario Library

Nominal operating conditions: the system operates within its designed envelope. Boundary conditions: the system operates at the edges of its envelope — maximum temperature, minimum power, maximum load. Off-nominal conditions: the system responds to failures, degraded modes, and unexpected inputs. Regression scenarios: specific conditions that previously revealed problems, preserved as permanent checks to prevent recurrence.

Why Libraries Matter

Without a scenario library, the engineer who runs a verification chooses which cases to test. Different engineers choose differently. Critical edge cases are forgotten when the engineer who remembered them changes roles. A scenario library captures institutional knowledge about what to test and preserves it independently of any individual. When a new failure mode is discovered in test or operation, the corresponding scenario is added to the library. The library grows smarter over time.

Model-Driven Scenarios

In an MBSE context, scenarios can be derived from the model itself. The system model defines operating modes, interfaces, and functional flows. Each operating mode implies a set of conditions to verify. Each interface implies a compatibility check. Each functional flow implies an end-to-end scenario. The mapping from model elements to scenarios is itself part of the model — and changes to the model can trigger new scenarios to be added or existing scenarios to be updated.

Performance Baselines

Verification tells you whether the design meets requirements. Baselines tell you whether the design is getting better or worse over time.

A performance baseline is a snapshot of key metrics at a defined point: thermal margin at PDR, structural safety factor at CDR, mass at each design iteration. The pipeline stores baselines and compares every new result against them. This is the regression comparison from Lesson 2, elevated to a systematic practice.

Baseline Management

Baselines must be deliberately established and deliberately updated. The pipeline should never silently overwrite a baseline. Updating a baseline is a decision: "This is the new reference point against which future changes will be measured." That decision should be recorded — who authorized the baseline update, why, and what the previous baseline was.

Trend Detection

Individual comparisons tell you whether this run is better or worse than the baseline. Trend analysis tells you whether the design is systematically drifting. A thermal margin that decreases by 0.5% on each design iteration individually passes every check. After twenty iterations, the cumulative 10% decrease is a problem that no individual check would have caught. The pipeline should compute and display trends, not just point comparisons.

Measuring V&V Effectiveness

Automating V&V is not the end goal. Effective V&V is. The pipeline should measure whether the verification process itself is working.

Defect Escape Rate

Of the defects discovered in downstream activities (integration, test, operation), how many should have been caught by the automated V&V pipeline? A high defect escape rate means the rules and scenarios are incomplete — the pipeline is checking things that are fine while missing things that matter. This is the most important metric because it directly measures the pipeline's value: defects caught early are cheap, defects that escape are expensive.

Time to Detection

When a defect is introduced, how long until the pipeline detects it? In a continuous V&V model, the answer should be measured in minutes to hours — the time between the model change and the next pipeline execution. If the answer is days or weeks, the pipeline is not running frequently enough or is not triggered by the right model changes.

False Positive Rate

Of the violations flagged by the pipeline, how many turn out to be non-issues after investigation? A high false positive rate destroys trust. Engineers stop paying attention to pipeline results when most flags are noise. False positives typically indicate that rules are too strict, thresholds are too tight, or rules do not account for context (flagging a concept-phase estimate against a CDR-phase standard). Tuning rules to minimize false positives is as important as writing the rules in the first place.

Pipeline Execution Time

How long does the full V&V pipeline take to execute? If it takes eight hours, engineers will not wait for results before moving on. They will make the next change on top of an unverified previous change, and defects will compound. Pipeline execution time directly determines whether continuous V&V is practical. A pipeline that runs in thirty minutes enables a tight feedback loop. A pipeline that runs overnight is effectively a nightly batch job, regardless of what you call it.

The Cultural Shift: Quality Built In

The deepest change continuous V&V requires is cultural. In a milestone-based model, quality is the responsibility of the review board. Engineers design. Reviewers check. The implicit message is: "Your job is to design. Someone else's job is to catch your mistakes."

Continuous V&V eliminates that separation. The pipeline runs on every change. The engineer sees the results immediately. Quality is not checked at the end — it is built in from the beginning. The engineer who introduces a problem is the first to know, and the cost of correction is minimal because nothing downstream has been built on top of the error.

This requires engineers to see V&V not as a bureaucratic hurdle but as a tool that helps them do better work. When the pipeline catches a cross-discipline inconsistency within an hour of its introduction, the engineer avoids weeks of work based on wrong assumptions. When the scenario library catches a regression that would have been discovered at system test, the engineer avoids a costly redesign cycle.

The organizations that succeed with continuous V&V are those that treat verification results like a developer treats unit test results: immediate, personal, and actionable. The organizations that struggle are those that treat verification as someone else's job and the pipeline as someone else's tool.

Move through the three stages to see how V&V maturity affects every dimension of engineering quality assurance. Most organizations are in the first or second stage. The third stage requires investment in model rules, scenario libraries, baseline management, and cultural change — but the payoff in reduced rework and faster design cycles is substantial.

Assessment