Two different metrics quantifying model and simulator predictive capability are formulated and evaluated; both metrics exploit results from conducted validation experiments where simulation results are compared to the corresponding measured quantities. The first metric is inspired by the modified nearest neighbor coverage metric and the second by the Kullback?Liebler divergence. The two different metrics are implemented in Python and in a here-developed general metamodel designed to be applicable for most physics-based simulation models. These two implementations together facilitate both offline and online metric evaluation. Additionally, a connection between the two, here separated, concepts of predictive capability and credibility is established and realized in the metamodel. The two implementations are, finally, evaluated in an aeronautical domain context.

Establishing interoperability is an essential aspect in the often pursued shift towards Model Based System Engineering (MBSE) of, for example, aircraft. If models are to be the primary information carriers during development, the applied methods to enable interaction between engineering domains need to be modular, reusable, and scalable. One possible solution is to rely on available open-source tools and standards. In this paper, the standards Functional Mock-up Interface (FMI) and System Structure and Parameterization (SSP) are exploited to exchange data between the disciplines of systems simulation and geometry modeling. A method to export data from the 3D Computer Aided Design (CAD) Software (SW) CATIA in the SSP format is developed and presented. Analogously, FMI support of the Modeling & Simulation (M&S) tools OMSimulator, OpenModelica, and Dymola are utilized along with the SSP support of OMSimulator. The developed technology is put into context by means of integration with M&S methodology for aircraft vehicle system development deployed at Saab Aeronautics. Finally, the established interoperability is demonstrated in an industrially relevant use-case. A primary goal of the research is to prototype and demonstrate functionality, enabled by the SSP and FMI standards, that could improve on MBSE methodology implemented in industry and academia.

Modeling and Simulation (M&S) is seen as a means to mitigate the difficulties associated with increased system complexity, integration, and cross-couplings effects encountered during development of aircraft subsystems. As a consequence, knowledge of model validity is necessary for taking robust and justified design decisions. This paper presents a method for using coverage metrics to formulate an optimal model validation strategy. Three fundamentally different and industrially relevant use-cases are presented. The first use-case entails the successive identification of validation settings, and the second considers the simultaneous identification of n validation settings. The latter of these two use-cases is finally expanded to incorporate a secondary model-based objective to the optimization problem in a third use-case. The approach presented is designed to be scalable and generic to models of industrially relevant complexity. As a result, selecting experiments for validation is done objectively with little required manual effort.

The aeronautical industry is becoming increasingly reliant on Modeling and Simulation (M&S) for use throughout all system development phases, for system verification, and end-user training. To justify and to utilize the full potential of today’s model-based approach, the development of efficient and industrially applicable methods for credibility assessment of M&S efforts is a key challenge.

This work addresses methods facilitating credibility assessment of simulation models and simulator applications used in aircraft system development. For models of individual aircraft subsystems, an uncertainty aggregation method is proposed that facilitates early model validation through approximate uncertainty quantification. The central idea is to integrate information obtained during component level validation directly into the component equations, and to utilize this information in model level uncertainty quantification.

In addition to methods intended for models of individual subsystems, this work also proposes a method and an associated tool for credibility assessment of large-scale simulator applications. As a complement to traditional document-centric approaches, static and dynamic credibility information is here presented to end-users directly during simulation. This implies a support for detecting test plan deficiencies, or that a simulator configuration is not a suitable platform for the execution of a particular test. The credibility assessment tool has been implemented and evaluated in two large-scale system simulators for the Saab Gripen fighter aircraft. The work presented herein also includes an industrially applicable workflow for development, validation, and export of simulation models.

Uncertainty Quantification (UQ) is vital to ensure credibility in simulation results and to justify model-based design decisions – especially in early development phases when system level measurement data for traditional model validation purposes are scarce. Central UQ challenges in industrial applications are computational cost and availability of information and resources for uncertainty characterization. In an attempt to meet these challenges, this paper proposes a framework for early and approximate UQ intended for large simulation models of dynamical systems. A Modelica simulation model of an aircraft environmental control system including a liquid cooling circuit is used to evaluate the industrial applicability of the proposed framework.

To better utilize the potential of system simulation models and simulators, industrially applicable methods for Verification, Validation and Uncertainty Quantification(VV&UQ) are crucial. This paper presents an exploratory case study of VV&UQ techniquesapplied on models integrated in aircraft system simulators at Saab Aeronauticsand in driving simulators at the Swedish National Road and Transport Research Institute(VTI). Results show that a large number of Verification and Validation (V&V)techniques are applied, some of which are promising for further development and use insimulator credibility assessment. Regarding the application of UQ, a large gap betweenacademia and this part of industry has been identified, and simplified methods areneeded. The applicability of the NASA Credibility Assessment Scale (CAS) at the studied organizations is also evaluated and it can be concluded that the CAS is consideredto be a usable tool for achieving a uniform level of V&V for all models included in asimulator, although its implementation at the studied organizations requires tailoringand coordination.

A common viewpoint in both academia and industry is that that Verification, Validation and Uncertainty Quantification (VV&UQ) of simulation models are vital activities for a successful deployment of model-based system engineering. In the literature, there is no lack of advice regarding methods for VV&UQ. However, for industrial applications available methods for Uncertainty Quantification (UQ) often seem too detailed or tedious to even try. The consequence is that no UQ is performed, resulting in simulation models not being used to their full potential.

In this paper, the effort required for UQ of a detailed aircraft vehicle system model is estimated. A number of methodological steps that aim to achieve a more feasible UQ are proposed. The paper is focused on 1‑D dynamic simulation models of physical systems with or without control software, typically described by Ordinary Differential Equations (ODEs) or Differential Algebraic Equations (DAEs). An application example of an aircraft vehicle system model is used for method evaluation.

Simulation  models of physical systems, with or without control software, are widely used in the aeronautic industry in applications ranging from system development to verification and end-user training. With the main drivers of reducing the cost of physical testing and in general enhancing the ability to take early model-based design decisions, there is an ongoing trend of further increasing the portion of modeling and simulation.

The work presented in this thesis is focused on development of methodology for model validation, which is a key enabler for successfully reducing the amount of physical testing without compromising safety. Reducing the amount of physical testing is especially interesting in the aeronautic industry, where each physical test commonly represents a significant cost. Besides the cost aspect, it may also be difficult or hazardous to carry out physical testing. Specific to the aeronautic industry are also the relatively long development cycles, implying long periods of uncertainty during product development. In both industry and academia a common viewpoint is that verification, validation, and uncertainty quantification of simulation models are critical activities for a successful deployment of model-based systems engineering. However, quantification of simulation results uncertainty commonly requires a large amount of certain information, and for industrial applications available methods often seem too detailed or tedious to even try. This in total constitutes more than sufficient reason to invest in research on methodology for model validation, with special focus on simplified methods for use in early development phases when system measurement data are scarce.

Results from the work include a method supporting early model validation. When sufficient system level measurement data for validation purposes is unavailable, this method provides a means to use knowledge of component level uncertainty for assessment of model top level uncertainty. Also, the common situation of lacking data for characterization of parameter uncertainties is to some degree mitigated. A novel concept has been developed for integrating uncertainty information obtained from component level validation directly into components, enabling assessment of model level uncertainty. In this way, the level of abstraction is raised from uncertainty of component input parameters to uncertainty of component output  characteristics. The method is integrated in a Modelica component library for modeling and simulation of aircraft vehicle systems, and is evaluated in both deterministic and probabilistic frameworks using an industrial application example. Results also include an industrial applicable process for model development, validation, and export, and the concept of virtual testing and virtual certification is discussed.

Simmuleringsmodeller av fysikaliska system, med eller utan reglerande mjukvara, har sedan lång tid tillbaka ett brett användningsområde inom flygindustrin. Tillämpningar finns inom allt från systemutveckling till produktverifiering och träning. Med de huvudsakliga drivkrafterna att reducera mängden fysisk provning samt att öka förutsättningarna till att fatta välgrundade modellbaserade designbeslut pågår en trend att ytterligare öka andelen modellering och simulering.

Arbetet som presenteras i denna avhandling är fokuserat på utveckling av metodik för validering av simuleringsmodeller, vilket anses vara ett kritiskt område för att framgångsrikt minska mängden fysisk provning utan att äventyra säkerheten. Utveckling av metoder för att på ett säkert sätt minska mängden fysisk provning är speciellt intressant inom flygindustrin där varje fysiskt prov vanligen utgör en betydande kostnad. Utöver de stora kostnaderna kan det även vara svårt eller riskfyllt att genomföra fysisk provning. Specifikt är även de långa utvecklingscyklerna som innebär att man har långa perioder av osäkerhet under produktutvecklingen. Inom såväl industri som akademi ses verifiering, validering och osäkerhetsanalys av simuleringsmodeller som kritiska aktiviteter för en framgångsrik tillämpning av modellbaserad systemutveckling. Kvantifiering av osäkerheterna i ett simuleringsresultat kräver dock vanligen en betydande mängd säker information, och för industriella tillämpningar framstår tillgängliga metoder ofta som alltför detaljerade eller arbetskrävande. Totalt sett ger detta särskild anledning till forskning inom metodik för modellvalidering, med speciellt fokus på förenklade metoder för användning i tidiga utvecklingsfaser då tillgången på mätdata är knapp.

Resultatet från arbetet inkluderar en metod som stöttar tidig modellvalidering. Metoden är avsedd att tillämpas vid brist på mätdata från aktuellt system, och möjliggör utnyttjande av osäkerhetsinformation från komponentnivå för bedömning av osäkerhet på modellnivå. Avsaknad av data för karaktärisering av parameterosäkerheter är även ett vanligt förekommande problem som till viss mån mildras genom användning av metoden. Ett koncept har utvecklats för att integrera osäkerhetsinformation hämtad från komponentvalidering direkt i en modells komponenter, vilket möjliggör en förenklad osäkerhetsanalys på modellnivå. Abstraktionsnivån vid osäkerhetsanalysen höjs på så sätt från parameternivå till komponentnivå. Metoden är implementerad i ett Modelica-baserat komponentbibliotek för modellering och simulering av grundflygplansystem, och har utvärderats i en industriell tillämpning i kombination med både deterministiska och probabilistiska tekniker. Resultatet från arbetet inkluderar även en industriellt tillämplig process för utveckling, validering och export av simuleringsmodeller, och begreppen virtuell provning och virtuell certifiering diskuteras.

To support early model validation, this paper describes a method utilizing information obtained from the common practice component level validation to assess uncertainties on model top level. Initiated in previous research, a generic output uncertainty description component, intended for power-port based simulation models of physical systems, has been implemented in Modelica. A set of model components has been extended with the generic output uncertainty description, and the concept of using component level output uncertainty to assess model top level uncertainty has been applied on a simulation model of a radar liquid cooling system. The focus of this paper is on investigating the applicability of combining the output uncertainty method with probabilistic techniques, not only to provide upper and lower bounds on model uncertaintiesbut also to accompany the uncertainties with estimated probabilities.It is shown that the method may result in a significant improvement in the conditions for conducting an assessment of model uncertainties. The primary use of the method, in combination with either deterministic or probabilistic techniques, is in the early development phases when system level measurement data are scarce. The method may also be used to point out which model components contribute most to the uncertainty on model top level. Such information can be used to concentrate physical testing activities to areas where it is needed most. In this context, the method supports the concept of Virtual Testing.

This paper describes a framework for development and validation of multipurpose simulation models. The presented methodology enables reuse of models in different applications with different purposes. The scope is simulation models representing physical environment, physical aircraft systems or subsystems, avionics equipment, and electronic hardware. The methodology has been developed by a small interdisciplinary team, with experience from Modeling and Simulation (M&S) of vehicle systems as well as development of simulators for verification and training. Special care has been taken to ensure usability of the workflow and method descriptions, mainly by means of 1) a user friendly format, easy to overview and update, 2) keeping the amount of text down, and 3) providing relevant examples, templates, and checklists. A simulation model of the Environmental Control System (ECS) of a military fighter aircraft, the Saab Gripen, is used as an example to guide the reader through the workflow of developing and validating multipurpose simulation models. The methods described in the paper can be used in both military and civil applications, and are not limited to the aircraft industry.