A framework for equipmentsystems-of-systems effectiveness evaluation using parallelexperiments approach

1.College of Aerospace Science and Engineering,National University of Defense Technology,Changsha 410073,China; 2.Academy of Equipment,Beijing 101416,China

1.Introduction

A framework for equipmentsystems-of-systems effectiveness evaluation using parallelexperiments approach

Zilong Cheng1,*,LiFan2,and Yulin Zhang1

1.College of Aerospace Science and Engineering,National University of Defense Technology,Changsha 410073,China; 2.Academy of Equipment,Beijing 101416,China

Equipment systems-of-systems(SoS)effectiveness evaluation can provide important reference for construction and optimization of the equipment SoS.After discussing the basic theory and methods of parallel experiments,we depict an SoS effectiveness analysis and evaluation method using parallelexperiments theory in detail.A case study is carried out which takes the missile defense system as an example.An arti fi cial system of the missile defense system is constructed with the multi-agent modeling method.Then,single factor,multiple factors and defense position deployment computational experiments are carried out and evaluated with the statisticalanalysis method.Experiment results show thatthe altitude ofthe secondary interception missile is not the key factor which affects SoS effectiveness and putting the defense position ahead will increase defense effectiveness.The case study demonstrates the feasibility ofthe proposed method.

agent based modeling,parallel experiments,computationalexperiments,missile defense system,effectiveness evaluation.

1.Introduction

Equipmentsystems-of-systems(SoS)effectiveness evaluation has gained increasing attention for its wide application in equipmentSoS planning,constructing and optimizing.The research methods athome and abroad have undergone severalprimary changes,from simple probability index to multi-criteria evaluation,to modern comprehensive effectiveness evaluation index set,and lots ofscholars have made outstanding contributions to these innovations.At present,equipment SoS effectiveness evaluation methods can be divided into two major categories,which are statistics methods and analysis methods.The statistics method evaluates the effectiveness index by adopting mathematical statistics on experimentdata which are obtained from massive exercises and practice.However,the high cost of experiments greatly limits the application of the statistics method.The analysis method can be applied even without the existence of a realsystem.This method ordinarily constructs a mathematicalmodelaccording to the relationship between the ef fi ciency index and the given condition,then obtains an estimated value of the ef fi ciency index through theoretical calculation.Frequently used analysis methods include the structural assessment method,the quanti fi cation scale method,the phase probability method and the fuzzy evaluation method[1].

Building the object model accurately is the basic premise for the application of the analysis method.Traditional equipment always has simpler structures and fewer components,thus we can obtain its relatively accurate mathematicalmodelbased on the theory of reduction and similarity using a top-down decomposition method.With the rapid development of the equipment SoS,modern equipment SoS becomes complicated and possesses characteristics like independence,autonomy and complex interactions which lead to nonlinearity,emergency and uncertainty of the system behavior.The traditional reduction theory is unable to meet the requirement of complex system study with determinacy simulation.In order to explore a new complex system research method,we must combine it with the holism theory which is attained by using bottom-up likelihood calculation.In the process of exploration,a series of practical methods appeared. Kangaspunta et al.[2]developed a portfolio methodology which guides decision-making by identifying which weapon systems combination is ef fi cient with respect to multiple evaluation criteria in different combat situations atdifferent cost levels.Jiang et al.[3]adopted an evidential reasoning approach to aggregate the capability measurement information from sub-capability criteria to topcapability criteria,then the assessed weapon systems were ranked and analyzed according to utility intervals.Ender et al.[4]proposed a modeling and simulation frameworkthatsupports the architecture levelanalysis of the ballistic missile defense system.Neuralnetwork surrogate models were also used to representhigh-or medium-fi delty modeling and simulation tools which enable the model to be executed rapidly with negligible loss in fi delity.The above mentioned research has tried to combine the traditionalreductionism theory with the holism theory and realize the expanding from the deterministic research method to probability computationalexperiments.

Adopting the complex system theory to study equipment SoS effectiveness has become a hot topic since the coming of the 21th century.However,the internalinteraction behavior and the emergence mechanism of complex system progress slowly,which call for innovation of the research idea.Parallel experiments theory[5,6]is a pioneer exploration in the equipment SoS experiments domain.The next section discusses the basic methods and steps of SoS effectiveness evaluation underthe parallelexperiments framework fi rstly.A case study of the missile defense system using the proposed method is carried out and analyzed in the third and fourth sections.Conclusions are given in the last section.

2.Equipment SoS effectiveness evaluation framework

The parallelexperiments theory devotes itself to the complex interaction and evolution mechanism in the complex system based on the parallel system which is composed of the realsystem and the arti fi cialsystem.Physicalexperiments and computationalexperiments jointly promotthe evolution of the complex system and realize the expanding from technology performance testing to SoS effectiveness evaluation.

There are subtle differences between traditional simulation and parallel experiments,including system composition,system type,system behavior,and research objectives and methods.The main avenue of the traditionalsimulation is to build a simulation system isomorphic with the real system,then,analysis and research on the simulation system deepen the recognition of the real system. Mathematicalmodeling is the basis for traditionalsimulation study,however,lack of effective modeling methods makes the traditional simulation inadequate when faced with complex system research.Guidance and controlofthe real system is the main objective in parallel experiments. Its main idea is constructing an arti fi cial system which is equivalent to the real system,then guiding and controling the realsystem towards the desired evolution direction by parallelexecution.Behavior-based interaction rules can guarantee the reproduction of the complex phenomenon, which is more suitable for the complex system study.

In order to distinguish parallel experiments and traditional simulation methods clearly,some comparisons are presented in the table below.

Table 1 Comparisons ofparallelexperiments and traditionalsimulation

Parallelexperiments are deduced by the real time interaction among agents and data interaction between the real system and the arti fi cial system,while traditional simulation is achieved by similarity modeling which is an important method for arti fi cial system modeling.Thus,traditional simulation can be treated as a proper subset of parallel experiments in a generalized point of view.It is worth emphasizing that modern simulation methods have absorbed a lot of latest research achievements of complex systems and some modi fi ed simulation methods can be used to carry outpreliminary research on complex military systems[7].

The agent-based modeling[8,9]method is an effective means to study complex systems which can display the whole emergence phenomenon by agents with reactive and autonomy characters.Computationalexperiments[10,11] employ an agent-based bottom-up method to constructarti fi cial objects which will be used to represent the complex system.With the help of interactions among arti ficialobjects combined with ef fi cientexperimentalschemes, we can analyze and study the interested objects in depth. Considering the complexity of the composition and interactions of the complex system,the design and realization of computational experiments need the support of a topdown method.Exploratory analysis[12]is an important method forcomplex research which employs the top-down method.A hierarchical solving tree is established by the multi-resolution modeling technique which facilitates the design of experiments.In fact,this method explores extensively in multi-dimension indeterminacy space and produces valuable conclusions through data mining on a series of experiment data.Its research technique works by dealing with the experiment results of indeterminacy factors as a whole,which provides a feasible avenue from the top level design to key factors computation.The relationships among parallel experiments,computational experiments,agent-based modeling and exploratory analysis can be depicted as the fi gure below.

Fig.1 Theoreticalrelationship map of parallelexperiments

Agent-based modeling is the premise of computational experiments while the exploratory analysis method is the guiding theory of large-scale computational experiments. Physicalmodels and exploratory tests jointly constitute the foundation of physical experiments.Parallel experiments are achieved by the interaction between computationalexperiments and physicalexperiments.

Equipment SoS effectiveness analysis and evaluation methods use the agent-based modeling method to constructan arti fi cialsystem,the exploratory analysis method to carry outcomputationalexperiments,and the statistical analysis method to accomplish effectiveness evaluation.Its main steps can be depicted as follows.

2.1 Construction of artificialsystem

The arti fi cial system[13]is a complementary part of the realsystem in parallelexperiments,which comes from the real system but distinguishes from it.Firstly,it is a fungible edition ofrealedition and established based on the real system,including entire essentialfactors and internalinteractions ofthe realsystem.Secondly,the arti fi cialsystem is not only a simple replica butalso a prolongation and promotion ofthe realsystem.In otherwords,the realsystem is merely a possible representation form of the arti fi cialsystem which can evolve the entire possibilities of the given system.Allof these distinctions callforsome new requirements for arti fi cialsystem construction.Established arti ficialsystems are capable of revealing the autonomy and the internal interaction.Evolution tendency of the given system is changeable and non-unique in different situations. Eventually,it brings about the complex phenomenon by interactions among individual agents in some certain circumstances.

The arti fi cialsystem is setup based on a reasonable hypothesis and simpli fi cation ofthe realsystem.Agent-based modeling uses agent modules to represent entities.In the complex system study fi eld,the multiagentsystem(MAS) is an ideal technique to build the arti fi cial system.Therefore,main steps ofarti fi cialsystem construction are as follows.

(i)Synthetically analyze essential components of the real system,comb coupling relations among internal entities,and list interested characteristic parameters or phenomena of the given system.

(ii)De fi ne attributes and behaviors of each entity according to the internallogic relation in the arti fi cialsystem. The designed agentmodule adopts an agent-based modeling method.The MAS can be established by the association ofagentmodules.Update and adjustthe attributes and behaviors ofthe arti fi cialsystem to satisfy the requirement of parallelexperiments.

(iii)Select a suitable multi-agent platform to support parallel experiments.Integrate available agent modules with interactive and logic controlmodules.Set the initialization condition and procedures based on abundantinvestigation.

2.2 Computationalexperiments

Computational experiments[14]are carried out to display complex emergence phenomena and try to excavate the complexity generation mechanism through a series of simulations.Exploratory analysis can explore extensively in multi-dimension space,which is hard for conventional methods.The arti fi cialsystem is the foundation of computationalexperimentswhile exploratory analysis is the guiding theory for the computational experiments.Moreover, complete computationalexperiments also need explicitresearch objectives and ef fi cient experiment schemes.The concrete steps of computational experiments are as follows.

(i)De fi ne experiment scenarios under abundant investigation.Researchers need de fi ne system variables whichare ready to study according to the research objective.Before the experiment,design a reasonable preliminary experimentscheme in terms ofcharacteristics ofsystem variables.

(ii)Computational experiments are conducted by the combination of qualitative and quantitative methods.The qualitative study obtains the primary relation between system inputs and outputs while the quantitative study acquires system quantization outputs in the given scenario which can be used for further evaluation.The emergence phenomenon can be investigated through plenty of experiments data.

(iii)Revise and improve experiment schemes based on the preliminary conclusion and research objective.Iteration and feedback are needed to deepen the cognizance of complex system essence.Explore and fi nd the value range of system variables which can acquire satisfactory solutions.

2.3 Effectiveness analysis and evaluation

The objective ofequipmentSoS effectiveness evaluation is to provide constructive suggestions about planning,constructing and optimizing of SoS in different con fi gurations and application areas.Diversity of computational experiments determines thatthe ultimate goalof So S effectiveness analysis and evaluation is notan optimalorunique solution buta satisfactory solution set.

Effectiveness analysis and evaluation has close relationship with experimental schemes.Different experimental schemes restrict the evaluation strategy while the given evaluation method in fl uences the design of experimental schemes.Faced with plenty of experiments data,the mathematical statistics method will be needed to comprehensively analyze the experiment results.Therefore,the essentialsteps of effectiveness analysis and evaluation are as follows.

(i)Design the reasonable synthetic evaluation index according to the characters of system variables and experimental objectives.The index need re fl ect the proportion and relevance of each essentialfactor.

(ii)Acquire qualitative relationship between system variables and SoS effectiveness through primary qualitative experiments.Summarize and induce quantitative relationship between system variables and SoS effectiveness through exploring experiments in the full exploration space.Afterwards,we can study the relevance and con fidence of the experimentresultby statisticalanalysis.

(iii)Compute the system variable value range which can obtain satisfactory SoS effectiveness according to the given threshold.Put the obtained system variable value into the originalsystem for veri fi cation.Several iterations are needed if the system outputcan notreach the expected value.A case study needs to be realized to validate the effectiveness of the proposed method and offer theory supportfor SoS effectiveness evaluation and optimization.

3.Missile defense artificialsystem modeling

3.1 Framework of artificialsystem

A typical missile defense system consists of the detection radar,the command center,and the attack and interception missiles.The experiment scenario is set as follows.The defense system comprises a guidance radar,and a set of interception and command system.The interception system adopts two missiles to interceptthe attack missile[15]. The defense position is arranged underthe attack trajectory which is obtained by actual measurement of some certain missiles.

The whole defense modelconstitutes the agent module and the function module in logic.The defense system composition framework based on the Repast[16]multi-agent platform is depicted in the fi gure below.

Fig.3 Composition of missile defense system

The agentmodule includes the attack missile agent,the interception missile agent,the detection radar agent and the command center agent.The following table presents a summary of the main properties and behaviors of agent entities.The function module includes the transformation module,the scenario loading module,the trajectory prediction module and the radar cross section(RCS)simulation module.The procedure starts from the master program who loads scenario data.Attack missile,detection radar,command center and intercept missile are invoked in a proper order by the schedule mechanism within the Repastplatform.Experimentdata are saved during the iterations untilthe stop condition is satis fi ed.The system operation process is shown as follows.

Table 2 Properties,behaviors and interactions of agents

Fig.4 System operation process chart

3.2 Missile trajectory model

In order to evaluate the missile defense system more accurately,the range measure value of the attack missile trajectory is loaded into the simulation system,then,only the interception missile modelneeds to be considered.The position and the velocity ofthe interception missile in the inertial coordinate system at the k th simulation step can be calculated with the formulae below:

whereμeis the geocentric gravitational constant,Δt indicates the simulation time step,andindicates the distance to the coordinate origin atthe end of the(k-1)th simulation step.

The calculation formula of position and velocity at the launch pointis as follows:

where vFis the absolute speed at the launch point,A is the angle between the localhorizontalprojection ofthe velocity vector and the north direction,andθFis the angle between the speed vector and the localhorizontal.

3.3 Radar detection model

Under the assumption that there is no disturb equipment, radar detection probability pdcan be calculated using the formula below:

where SNindicates the single pulse signal to noise ratio (SNR)which can be calculated by the radar equation,y0is the detection threshold and satis fi es the following equation:

where n is cumulative pulses in a single radar scan and given by

whereθ0.5indicates the half-powerbeam bandwidth ofthe radar antenna,Ωindicates the radar antenna scan angular velocity,and frindicates the radar pulse repetition frequency.

3.4 Missile interception model

The main objective of computational experiments is to investigate the relationship among the altitude of the interception missile,prediction error and SoS effectiveness.Two interceptmissiles are distinguished by the altitude of the expected interception point.The altitude value of the primary interception missile ranges from 15 km to 20 km while the secondary intercept missile ranges from 10 km to 15 km.Prediction error indicates the error between the expected interception point and actual interception point which ranges from 10 m/km to 60 m/km.Interception probability PInterceptis the function of destroy probability[17]and the distance between the attack missile and the interceptmissile.Itcan be calculated with the formula below:

where r is the distance between the targetand the intercept missile,PDestroyindicates the destroy probability which is determined by the damage ability of the intercept missile. In particular,the destroy probability equals to zero if the distance between the targetand the interception missile is beyond the damage radius rmax.Monte Carlo simulation is adopted to acquire the average hit probabilityˉP which can be calculated with the following formula.

where m indicates the total simulation times and l is the successfulinterception times.

4.Computationalexperiments and effectiveness evaluation

4.1 Single factor computationalexperiments

The control variable method is adopted to carry out single factor qualitative study.Note thatthe hitprobability in this paper is normalized due to con fi dentiality and the normalization hit probability PNcan be calculated with the following formula.

As itcan be seen,HP raises as the PIMA increases gradually.Preliminary analysis of possible reasons is thatthe higher the interception point altitude,the smaller the relative velocity and interception encounter angle which always leads to higher hitprobability.

Fig.5 Relationship between PIMA and HP

Fig.6 shows that HP does notpresentobvious variation tendency with the changing of SIMA.Thus,the relationship between SIMA and HP needs furtherinvestigation using the quantitative analysis method.

Fig.6 Relationship between SIMA and HP

Fig.7 shows thatHP descends as the PE decreases gradually.According to conventional experience,higher prediction error leads to higher interception error,then,the HP descends gradually.

Fig.7 Relationship between PE and HP

4.2 Multiple factors computationalexperiments

In orderto obtain quantitative relationship between system variables and SoS effectiveness,multiple factors computationalexperiments are carried out.Experimentparameters are extracted from the PIMA and SIMA value range with 1 km as the interval while PE is extracted with ten as the interval.At least 216 groups of experiments are needed to ful fi llthree parameters discrete exploratory experiments in the whole range space.Each group will repeat 100 times to obtain statistics probability.The following fi gures show the missile defense system effectiveness surface with PE equaling to 10 m/km,30 m/km and 60 m/km.

Fig.8 HP surface with different PEs

By comparing the hitprobability surface above,we can conclude thatthe missile interception probability increases with the increasing of PIMA,descends with the increasing of PE,and has no obvious relationship with SIMA.

For the sake of analyzing the relationship between PE and system effectiveness objectively,the mean and variance of HP with PE ranging from 10 m/km to 60 m/km are listed in the table below.Note thatthe maximum and minimum of HP are equal to 1 and 0,respectively,because of the normalization operation of HP.As itcan be seen from Table 3,with the increasing of PE,HP increases at fi rst,decreases,and then reaches its maximum effectiveness value 54.69%when PE equals 10 m/km and the minimum effectiveness value 39.11%when PE equals 60 m/km.

Table 3 Hit probability statistics

In the currentexperimentscenario,defense effectiveness willreach the theoretically optimalvalue with PE equaling 10 m/km.This can provide reference for the con fi guration of system parameters.The correlation coef fi cients among PE,PIMA,SIMA and HP are listed in the table below.

Table 4 Correlation coefficients

From Table 4,we can see thatthe correlation coef fi cient between PIMAand HP is 0.867 which means a strong positive correlation,the correlation coef fi cientbetween PE and HP is-0.03 which means a weak negative correlation.The correlation between SIMA and HP is 0.007 and the concomitantprobability equals 0.012.We can make a conclusion that in the signi fi cant level of 0.05,SIMA is not the key factor which affects missile defense system effectiveness[18].Statistics data validate the qualitative conclusion made in single factor computationalexperiments well.

4.3 Defense position deployment computational experiments

In order to analyze SoS ef fi ciency with different defense position deployments,one hundred defense positions are arranged on the square whose boundary is perpendicular to the assaultdirection.Every defense position is deployedwith a same fi ve-kilometer interval.According to the theoretically optimaleffectivenessobtained in the previoussection,PE,PIMAand SIMAare setas 10,17 and 12,respectively.Effectiveness surface can be calculated as shown below in Fig 9.

Fig.9 System effectiveness surface with different defense positions

The sensitive information such as target point latitude and longitude is omitted.The HP values are also normalized.The green box indicates the target point and the block arrow indicates the attack missile assault direction. The fi gure shows that the effectiveness of the missile defense system surface is a multi-peak function and a differentdefense position has a signi fi cantimpact on the effectiveness of the defense system.The defense effectiveness value shows a trend of decrease with the increasing distance between the target and defense positions.What’s more,putting the defense position ahead willpromote the defense effectiveness.

5.Conclusions

Equipment SoS effectiveness evaluation is an important conundrum in the military research domain.The agentbased modeling method is the main means of complex system research currently.The parallel experiment is a new theory for complex system analysis and study,which is based on MAS and parallel execution thoughts.This article discusses the basic theory and steps of parallelexperiments,then takes a missile defense system as an example, and carries out single factor,multiple factors and defense position deployment computational experiments.Experimentresults show that the altitude of the secondary interceptmissile is notthe key factorthataffects SoS effectiveness and putting the defense position ahead will promote the defense effectiveness.

The research result demonstrates the feasibility of the proposed method,agent-based modeling method and computationalexperimentscan provide a new idea forthe study of complex systems,especially in the complex military system domain.Our nextstep willfocus on the equipment SoS effectiveness analysis and evaluation based on Repast fora high performance computing(RepastHPC)platform. It is obvious that the computational experiments can be time-consuming with the increase of system variable dimensions.However,with the rapid development of high performance computers,these problems will be properly solved in the near future.

[1]J.Chen,G.H.Yu,X.G.Gao.Cooperative threat assessment ofmulti-aircraftbased on synthetic fuzzy cognitive map.Journal of Shanghai Jiaotong University(Science),2012,17(2): 228-232.

[2]J.Kangaspunta,J.Liesi¨o,A.Salo.Cost-ef fi ciency analysis of weapon system portfolios.European Journal of Operational Research,2012,223(1):264-275.

[3]J.Jiang,X.Li,Z.J.Zhou,et al.Weapon system capability assessment under uncertainty based on the evidential reasoning approach.Expert Systems with Applications,2011,38(11): 13773-13784.

[4]T.Ender,R.F.Leurck,Brian Weaver,et al.Systems-ofsystems analysis of ballistic missile defense architecture effectiveness through surrogate modeling and simulation.IEEE Systems Journal,2010,4(2):156-166.

[5]Y.L.Zhang.Parallel experiments:theory and methodology ofweapon equipmentsystems-of-systems experiments.Theme Report of the 428th Xiangshan Science Conferences,2012.(in Chinese)

[6]X.R.Yang,L.Fan,Z.K.Wang.An discussion on concepts and methods of weapon equipment system-of-system experiment.Defense Technology,2013,34(3):18-22.(in Chinese)

[7]R.Garza,R.R.Hill,D.D.Mattioda.Using simulation to analyze the maintenance architecture fora USAF weapon system. Journal of Simulation,2012,89(3):294-305.

[8]C.M.Macal,M.J.North.Tutorialon agent-based modeling and simulation.Journal ofSimulation,2010,4(3):151-162.

[9]M.Kumar.Rolesand ontology foragentsystems.GlobalJournal of Computer Science and Technology,2011,11(23):39-46.

[10]F.Y.Wang.Parallel control and management for intelligent transportation systems:concepts,architectures,and applications.IEEE Trans.on Intelligent Transportation Systems, 2010,11(3):630-638.

[11]R.Conte,N.Gilbert,G.Bonelli,etal.Manifesto of computational social science.The European Physical Journal Special Topics,2012,214(1):325-346.

[12]J.H.Kwakkei,E.Pruyt.Exploratory modeling and analysis,an approach for model-based foresight under deep uncertainty.Technological Forecasting&Social Change,2013, 80(3):419-431.

[13]J.M.Epstein,R.Axtell.Growing artificial societies:social science from the bottom up.New York:The Brooking Institute Press and the MIT Press,1996.

[14]D.Lazer,A.Pentland,L.Adamic,etal.Computationalsocial science.Science,2009,323(5915):721-723.

[15]B.U.Nguyen.Assessment of a ballistic missile defense system.Defense&Security Analysis,2014,30(1):4-16.

[16]Repast(Recursive Porous Agent Simulation Toolkit). http://repast.soureforge.net.

[17]X.B.Li,Y.L.Lei,H.Vangheluwe,etal.A multi-paradigm decision modeling framework forcombatsystem effectiveness measurement based on domain-speci fi c modeling.Journal of Zhejiang University Science C,2013,14(5):311-331.

[18]E.P.Cornell,S.Guikema.Probabilistic modeling of terrorist threats:a systems analysis approach to setting priorities among countermeasures.Military Operations Research,2012, 7(2):5-20.

Biographies

Zilong Cheng was born in 1988.He received his M.S.degree in control science and engineering from National University of Defense Technology,in 2012.Currently he is a doctoral candidate in College of Aerospace Science and Engineering, National University of Defense Technology.His research interests include equipmentsystem experimentsimulation and evaluation.

E-mail:kdchengzilong@163.com

Li Fan was born in 1977.She received her Ph.D. degree from College of Aerospace and Engineering,National University of Defense Technology, in 2006.Currently she is working in Academy of Equipment as an associate professor.Her research interests include aircraft design,and complex system modeling and control.

E-mail:fanlinudt@163.com

Yulin Zhang was born in 1958.He received his Ph.D.degree from Zhejiang University,in 1988. Currently he is working in College of Aerospace Science and Engineering,National University of Defense Technology as a professor.His research interests include aerospace complex system control and management.

E-mail:y.l.zhang@tsinghua.edu.cn

10.1109/JSEE.2015.00035

Manuscriptreceived March 24,2014.

*Corresponding author.