Orchestration and Monitoring

Beyond Static Pipelines

The pipelines described in the previous lessons follow a fixed script: a model change triggers the same sequence of stages every time. Consistency checks, parametric evaluation, targeted simulation, regression comparison, dashboard update. The stages may run in parallel where dependencies allow, but the set of stages is predetermined.

This works for mature, stable workflows where the cost of running every check on every change is acceptable. But as the system model grows, as the number of simulations increases, and as the pipeline serves more disciplines, the fixed-script approach hits limits.

A structural parameter change does not need to trigger a control system stability analysis. A documentation update does not need to trigger any simulation at all. A minor tolerance change on a non-critical dimension does not need a high-fidelity FEA run that consumes eight hours of compute time and a solver license. A static pipeline either runs everything — wasting resources and slowing feedback — or relies on engineers to manually select which checks to run, which defeats the purpose of automation.

The next evolution is adaptive orchestration: pipelines that analyze the change, classify its impact, and select the appropriate checks. This is where engineering automation starts to leverage AI — not to replace engineering judgment, but to route changes to the right verification activities.

Static Pipelines: The Baseline

A static pipeline has a fixed configuration that maps triggers to stages. "Any model change triggers all stages." Or slightly more refined: "Changes to structural elements trigger structural stages. Changes to thermal elements trigger thermal stages." The mapping is defined at pipeline setup time and does not change without human intervention.

Strengths. Static pipelines are predictable. Every engineer knows exactly what will run when they commit a change. The pipeline behavior is auditable — the configuration file describes the complete execution plan. Debugging is straightforward because the execution path is deterministic.

Weaknesses. Static pipelines are either too broad (running unnecessary checks) or too narrow (missing checks when the predefined mapping does not cover a new type of change). They cannot adapt to the magnitude of a change — a single parameter tweak and a major architectural restructuring trigger the same pipeline. They cannot learn from past executions — a check that has never found a problem continues to run on every change.

Change Classification

The first step toward adaptive pipelines is classifying the change before deciding what to run. Change classification answers three questions:

What changed? Which model elements were modified — requirements, design parameters, interfaces, allocations? This is structural classification based on the model's element types and relationships.

How much changed? A parameter adjustment within its existing range is different from a parameter that crosses a constraint boundary. A component swap that maintains all interfaces is different from a swap that redefines interfaces. Magnitude classification determines whether the change is routine or significant.

What is affected downstream? A change to a component's mass affects the mass budget, the structural analysis, and possibly the dynamics simulation. A change to a component's thermal dissipation affects the thermal analysis and possibly the power budget. Impact classification traces the change through the model's relationship graph to identify which downstream analyses are affected.

In an MBSE context, impact classification uses the digital thread directly. The model defines which requirements are allocated to which functions, which functions are realized by which components, which components participate in which interfaces. Tracing a change through these relationships identifies the downstream impact — and the downstream impact determines which pipeline stages need to run.

Change classification can be implemented at multiple levels of sophistication:

Rule-based classification. Static rules defined by engineers: "Changes to elements of type X trigger stages A and B." Simple, auditable, but brittle when the model evolves.

Graph-based classification. The pipeline traverses the model's relationship graph from the changed element, following allocation, realization, and interface links to identify affected downstream elements. More dynamic than rules, but limited to the relationships explicitly defined in the model.

AI-assisted classification. A trained model analyzes the change in context — considering not just the model relationships but also the history of past changes, which checks found problems, and which combinations of changes tend to produce downstream issues. This is the frontier of engineering pipeline orchestration.

Resource Optimization

Engineering simulations are expensive. They consume compute time, solver licenses, and human attention (for result interpretation). A pipeline that runs every simulation on every change is wasteful. Resource optimization ensures that the most valuable checks run first and that resources are allocated where they provide the most information.

Prioritization

Not all checks are equally valuable for every change. A change to a structural parameter makes the structural FEA high priority and the electromagnetic compatibility analysis low priority. Prioritization ranks the checks by their relevance to the specific change and executes them in that order. If the pipeline has a time budget (say, one hour for fast feedback), prioritization ensures the most relevant checks complete within that budget.

Parallelization

Independent analyses should run concurrently. If the structural FEA and the thermal analysis do not share inputs or outputs, they can execute simultaneously on different compute resources. The pipeline's DAG structure (from Lesson 2) enables this: stages with no unmet dependencies execute in parallel. Effective parallelization can reduce end-to-end pipeline time from the sum of all stage durations to the duration of the longest critical path.

Surrogate Models for Screening

A high-fidelity structural FEA takes hours. A surrogate model — a simplified mathematical approximation trained on previous FEA results — takes seconds. The pipeline can use surrogates as a screening step: run the surrogate first, and only trigger the full-fidelity simulation if the surrogate indicates the result might be near a constraint boundary or significantly different from the baseline.

Surrogates are not replacements for full-fidelity analysis. They are filters that prevent unnecessary full-fidelity runs. A surrogate that predicts the structural safety factor is well above the requirement with high confidence eliminates the need for an eight-hour FEA run on a routine parameter change. A surrogate that predicts the safety factor is near the boundary triggers the full run for confirmation.

The trust question with surrogates is critical: how accurate must the surrogate be, and what is the consequence of a false negative (the surrogate says "safe" but the full simulation would have flagged a problem)? Surrogate accuracy must be validated against full-fidelity results, and the validation must be ongoing — as the design evolves, the surrogate's training data may become stale.

License and Compute Management

Commercial solver licenses are scarce and expensive. A pipeline that queues a simulation and waits two hours for a license is wasting calendar time. Resource-aware orchestration knows which licenses are available, which compute nodes have capacity, and routes jobs accordingly. Low-priority screening runs yield to high-priority regression checks. Jobs that can run on alternative solvers (when multiple tools can perform the same analysis) are routed to whichever solver has available licenses.

Trust Building: From Advisory to Automated Gating

The most sophisticated pipeline is worthless if engineers do not trust it. Trust is built incrementally, not declared by management mandate.

Stage 1: Visibility

The pipeline runs and publishes results, but does not gate anything. Engineers can see what the pipeline would have flagged, but their workflow is not affected. This stage builds familiarity. Engineers learn what the pipeline checks, how results are presented, and how to interpret flags. They discover false positives and help tune the rules. They discover true positives and begin to see the pipeline's value.

Stage 2: Advisory

The pipeline results are formally reviewed as part of the engineering process, but do not block progress. A pipeline flag generates a notification that the responsible engineer must acknowledge — "I have seen this flag and here is why it is acceptable" or "I have seen this flag and I am correcting the issue." Advisory mode creates accountability without creating a bottleneck. It also generates data: how often do engineers override flags, and how often does an override lead to a downstream problem?

Stage 3: Automated Gating for Routine Checks

Pipeline checks that have proven reliable — low false positive rate, high true positive rate, well-understood by the engineering team — become automated gates. A consistency check that has run for six months with a false positive rate below 2% can gate model commits: if the check fails, the commit is rejected until the issue is resolved. The key criterion is demonstrated reliability, not theoretical correctness. A rule that is theoretically sound but produces 30% false positives should not gate anything.

Stage 4: Comprehensive Automated Gating

As trust accumulates and rules are refined, more checks become gates. Cross-domain consistency rules gate design changes that affect multiple disciplines. Regression checks gate changes that degrade key performance metrics beyond a defined threshold. Human review gates remain for high-consequence decisions — design release, test readiness, certification — but the routine verification is fully automated.

The progression from visibility to comprehensive gating typically takes one to two years. Organizations that try to jump directly to automated gating encounter resistance, workarounds, and abandoned pipelines. The trust-building path is slower but sustainable.

Monitoring: Keeping the Pipeline Healthy

A pipeline is infrastructure. Like all infrastructure, it degrades if not monitored and maintained.

Pipeline Health Metrics

Execution success rate. What percentage of pipeline runs complete without infrastructure failures? A pipeline that fails 20% of the time due to license timeouts, tool crashes, or environment issues is unreliable — and unreliable infrastructure is unused infrastructure. Target: above 95%.

Stage duration trends. Are individual stages getting slower over time? A thermal simulation that took 20 minutes six months ago and now takes 45 minutes may indicate model growth (more elements to simulate), infrastructure degradation (slower compute nodes), or configuration drift (unintended changes to solver settings). Trend monitoring catches slow degradation before it crosses a threshold.

Queue wait time. How long do pipeline runs wait before starting? Long queue times indicate resource contention — too many pipelines competing for too few licenses or compute nodes. Queue wait time directly adds to the feedback loop delay: a pipeline that executes in 30 minutes but waits 2 hours in the queue delivers a 2.5-hour feedback loop.

Result stability. Do pipeline results for the same model version produce the same output across runs? Flaky checks — those that sometimes pass and sometimes fail on the same input — destroy trust faster than any other pipeline problem. Flakiness typically indicates non-deterministic tool behavior, environment sensitivity, or race conditions in parallel execution.

Model Change Monitoring

The pipeline does not operate in isolation. It operates on a model that is being actively developed by a team. Monitoring the model change patterns provides context for pipeline results.

Change velocity. How many model changes per day or per sprint? A sudden increase in change velocity may indicate a design phase transition (concept to preliminary) or a deadline-driven push. The pipeline must handle the increased load without unacceptable queue times.

Change distribution. Which model areas are changing most frequently? If the thermal subsystem is changing rapidly while the structural subsystem is stable, the pipeline's thermal stages are providing the most value. Resource allocation should follow change activity.

Change coupling. Do certain model elements always change together? If a mass parameter and a CG location always change in the same commit, the pipeline can treat them as a single change for classification purposes. Coupling analysis can also reveal unexpected dependencies: if changes to the power system frequently trigger failures in the structural checks, there may be an undocumented coupling that the model does not capture.

Feedback Loop Closure

Monitoring data is only valuable if it drives action. The monitoring system should generate alerts for actionable conditions:

• Pipeline success rate drops below threshold: investigate infrastructure issues
• Stage duration increases beyond historical range: investigate root cause (model growth, configuration drift, resource degradation)
• False positive rate increases for a specific rule: the rule may need recalibration because the design has evolved
• Defect escape detected: analyze why the pipeline did not catch it and add or modify rules to prevent recurrence

Each alert should identify the responsible party and provide enough context for diagnosis. An alert that says "pipeline failed" is useless. An alert that says "thermal simulation stage timed out after 3 hours (normal runtime: 25 minutes) on model version 4.7.2, which added 340 new thermal elements compared to 4.7.1" enables immediate diagnosis.

Assessment