Parameter estimation is an important topic in the field of system identification. This paper explores the role of a new information theory measure of data dependency in parameter estimation problems. Causation entropy is a recently proposed information-theoretic measure of influence between components of multivariate time series data. Because causation entropy measures the influence of one dataset upon another, it is naturally related to the parameters of a dynamical system. In this paper, it is shown that by numerically estimating causation entropy from the outputs of a dynamic system, it is possible to uncover the internal parametric structure of the system and thus establish the relative magnitude of system parameters. In the simple case of linear systems subject to Gaussian uncertainty, it is first shown that causation entropy can be represented in closed form as the logarithm of a rational function of system parameters. For more general systems, a causation entropy estimator is proposed, which allows causation entropy to be numerically estimated from measurement data. Results are provided for discrete linear and nonlinear systems, thus showing that numerical estimates of causation entropy can be used to identify the dependencies between system states directly from output data. Causation entropy estimates can therefore be used to inform parameter estimation by reducing the size of the parameter set or to generate a more accurate initial guess for subsequent parameter optimization.
Issue Section:
Research Papers
References
1.
Juang
, J.-N.
, 1994
, Applied System Identification
, Prentice-Hall
, Upper Saddle River, NJ
.2.
Juang
, J.-N.
, and Pappa
, R.
, 1985
, “An Eigensystem Realization Algorithm for Modal Parameter Identification and Model Reduction
,” J. Guid.
, 8
(5
), pp. 620
–627
.3.
Tischler
, M.
, and Remple
, R.
, 2012
, Aircraft and Rotorcraft System Identification
, American Institute of Aeronautics and Astronautics
, Reston, VA
.4.
Cooper
, J.
, and Wright
, J.
, 1992
, “Spacecraft In-Orbit Identification Using Eigensystem Realization Methods
,” J. Guid., Control, Dyn.
, 15
(2
), pp. 352
–359
.5.
Pappa
, R.
, and Juang
, J.-N.
, 1984
, “Galileo Spacecraft Modal Identification Using an Eigensystem Realization Algorithm
,” AIAA
Paper No. 84-1070.6.
Iliff
, K.
, 1989
, “Parameter Estimation for Flight Vehicles
,” J. Guid., Control, Dyn.
, 12
(5
), pp. 609
–622
.7.
Kabaila
, P.
, 1983
, “On Output-Error Methods for System Identification
,” IEEE Trans. Autom. Control
, 28
(1
), pp. 12
–13
.8.
Landau
, I.
, and Karimi
, A.
, 1997
, “An Output Error Recursive Algorithm for Unbiased Identification in Closed Loop
,” Automatica
, 33
(5
), pp. 933
–938
.9.
Iliff
, K. W.
, and Maine
, R. E.
, 1982
, “NASA Dryden's Experience in Parameter Estimation and Its Use in Flight Test
,” AIAA
Paper No. 82-1373.10.
Greene
, W.
, 1980
, “Maximum Likelihood Estimation of Econometric Frontier Functions
,” J. Econometrics
, 13
(1
), pp. 27
–56
.11.
Taylor
, B.
, and Rogers
, J.
, 2014
, “Experimental Investigation of Real-Time Helicopter Weight Estimation
,” J. Aircr.
, 51
(3
), pp. 1047
–1051
.12.
Akaike
, H.
, 1974
, “A New Look at the Statistical Model Identification
,” IEEE Trans. Autom. Control
, AC-19
(6
), pp. 716
–723
.13.
Akaike
, H.
, 1972
, “Information Theory and an Extension of the Maximum Likelihood Principle
,” 2nd International Symposium on Information Theory, Supplement to Problems of Control and Information Theory
, pp. 267
–281
.14.
Baram
, Y.
, and Sandell
, N.
, 1978
, “An Information Theoretic Approach to Dynamical Systems Modeling and Identification
,” IEEE Trans. Autom. Control
, AC-23
(1
), pp. 1113
–1118
.15.
Matsuoka
, T.
, and Ulrych
, T.
, 1986
, “Information Theory Measures With Application to Model Identification
,” IEEE Trans. Acoust., Speech, Signal Process.
, 34
(3
), pp. 511
–517
.16.
Kulhavy
, R.
, 1996
, “A Kullback–Leibler Approach to System Identification
,” Annu. Rev. Control
, 20
, pp. 119
–130
.17.
Kulhavy
, R.
, 1996
, “From Matching Data to Matching Probabilities
,” Recursive Nonlinear Estimation: A Geometric Approach
, Springer Berlin Heidelberg, pp. 13–61.18.
Kulhavy
, R.
, 1998
, “On Extension of Information Geometry of Parameter Estimation to State Estimation
,” Conference on Mathematical Theory of Networks and Systems (MTNS 98)
, Padova, Italy, July 6–10.19.
Chernyshov
, K.
, 2006
, “An Information Theoretic Approach to System Identification Via Input/Output Signal Processing
,” 2006 International Conference on Speech and Computer
, St. Petersburg, Russia, June 25–29.20.
Chernyshov
, K.
, 2009
, “An Information Theoretic Approach to Neural Network Based System Identification
,” 2009 Siberian Conference on Control and Communications
, Tomsk, Russia, Mar. 27–28.21.
Majda
, A.
, and Gershgorin
, B.
, “Improving Model Fidelity and Sensitivity for Complex Systems Through Empirical Information Theory
,” Proc. Natl. Acad. Sci.
, 108
(25
), pp. 10044
–10049
.22.
Granger
, C. W.
, 1969
, “Investigating Causal Relations by Econometric Models and Cross-Spectral Methods
,” Econometrica
, 37
(3), pp. 425
–438
.23.
Granger
, C. W.
, 1988
, “Some Recent Developments in a Concept of Causality
,” J. Econometrics.
, 39
(1–2), pp. 199
–211
.24.
Scheiber
, T.
, 2000
, “Measuring Information Transfer
,” Phys. Rev. Lett.
, 85
(2
), pp. 461
–464
.25.
Honey
, C.
, Kötter
, R.
, Breakspear
, M.
, and Sporns
, O.
, 2007
, “Network Structure of Cerebral Cortex Shapes Functional Connectivity on Multiple Time Scales
,” Proc. Natl. Acad. Sci.
, 104
(24
), pp. 10240
–10245
.26.
Vicente
, R.
, Wibral
, M.
, Lindner
, M.
, and Pipa
, G.
, 2011
, “Transfer Entropy—A Model-Free Measure of Effective Connectivity for the Neurosciences
,” J. Comput. Neurosci.
, 30
(1
), pp. 45
–67
.27.
Sun
, J.
, and Bollt
, E.
, 2014
, “Causation Entropy Identifies Indirect Influences, Dominance of Neighbors and Anticipatory Couplings
,” Phys. D
, 267
, pp. 49
–57
.28.
Sun
, J.
, Cafaro
, C.
, and Bollt
, E.
, 2014
, “Identifying the Coupling Structure in Complex Systems Through the Optimal Causation Entropy Principle
,” Entropy
, 16
(6
), pp. 3416
–3433
.29.
Sun
, J.
, Taylor
, D.
, and Bollt
, E.
, 2015
, “Causal Network Inference by Optimal Causation Entropy
,” SIAM J. Appl. Dyn. Syst.
, 14
(1
), pp. 73
–106
.30.
Cover
, T.
, and Thomas
, J.
, 1991
, Elements of Information Theory
, 2nd ed., Wiley
, Hoboken, NJ
, Chap. 2.31.
Sobczyk
, K.
, and Holobut
, P.
, 2012
, “Information Theoretic Approach to Dynamics of Stochastic Systems
,” Probab. Eng. Mech.
, 27
(1
), pp. 47
–56
.32.
Prokopenko
, M.
, Lizier
, J.
, and Price
, D.
, 2013
, “On Thermodynamic Interpretation of Transfer Entropy
,” Entropy
, 15
(2
), pp. 524
–543
.33.
Hahs
, D.
, and Pethel
, S.
, 2013
, “Transfer Entropy for Coupled Autoregressive Processes
,” Entropy
, 15
(3
), pp. 767
–788
.34.
Materassi
, M.
, Consolini
, G.
, Smith
, N.
, and De Marco
, R.
, 2014
, “Information Theory Analysis of Cascading Process in a Synthetic Model of Fluid Turbulence
,” Entropy
, 16
(3
), pp. 1272
–1286
.35.
Silverman
, B.
, 1986
, Density Estimation for Statistics and Data Analysis
, Springer Science and Business Media
, New York, NY
.36.
Moon
, Y.-I.
, Rajagopalan
, B.
, and Lall
, U.
, 1995
, “Estimation of Mutual Information Using Kernel Density Estimators
,” Phys. Rev. E
, 52
(3
), pp. 2318
–2321
.37.
Kraskov
, A.
, Stogbauer
, A.
, and Grassberger
, P.
, 2004
, “Estimating Mutual Information
,” Phys. Rev. E
, 69
(6
), p. 066138
.38.
Hlavackova-Schindler
, K.
, Palus
, M.
, Vejmelka
, M.
, and Bhattacharya
, J.
, 2007
, “Causality Detection Based on Information Theoretic Approaches to Time Series Analysis
,” Phys. Rep.
, 441
(1
), pp. 1
–46
.39.
Scott
, D.
, 1992
, Multivariate Density Estimation: Theory, Practice, and Visualization
, Wiley
, Hoboken, NJ
, Chap. 6.40.
Barrat
, A.
, Barthelemy
, M.
, and Vespignani
, A.
, 2008
, Dynamical Processes on Complex Networks
, Cambridge University Press
, Cambridge, UK
.41.
Hénon
, M.
, 1976
, “A Two-Dimensional Mapping With a Strange Attractor
,” Commun. Math. Phys.
, 50
(1
), pp. 69
–77
.42.
Reinhall
, P.
, Caughey
, T.
, and Storti
, D.
, 1989
, “Order and Chaos in a Discrete Duffing Oscillator: Implications on Numerical Integration
,” ASME J. Appl. Mech.
, 56
(1
), pp. 162
–167
.43.
More
, J. J.
, 1978
, “The Levenberg–Marquardt Algorithm: Implementation and Theory
,” Numerical Analysis: Proceedings of the Biennial Conference
Dundee, June 28–July 1, G. A. Watson, ed., Springer
, Berlin
, pp. 105
–116
.
This content is only available via PDF.
Copyright © 2017 by ASME
You do not currently have access to this content.
