Contents
Preface v
Acknowledgment vii
Contents viii
List of Tables xiv
List of figures xv
1 Randomness and probability 1
1.1 Randomness , 2
1.1.1 Random phenomena 2
1.1.2 Sample space and random events 2
1.2 Probability 4
1.2.1 Probability defined on events 4
1.2.2 Conditional probability ., 6
1.2.3 Independence 8
1.3 Random variable 9
1.3.1 Random variable and distributions 9
1.3.2 Vector random variables and joint distribution 12
1.3.3 Conditional distribution , 14
1.3.4 Expectations 16
1.3.5 Typical distribution 19
1.4 Concluding remarks 22
2 Inference and statistics 25
2.1 Sampling 26
2.1.1 Sampling distributions for small samples 28
2.1.2 Sampling distributions for large samples 30
2.1.3 Chebyshev's inequality 31
2.1.4 The law of large numbers 32
2.1.5 The central limit theorem 33
2.2 Estimation 34
2.2.1 Estimation 34
2.2.2 Sampling error 35
2.2.3 Properties of estimators » 37
2.3 Maximum Likelihood Estimator , 39
2.3.1 The Maximum Likelihood Method (M-L Mehtod) 39
2.3.2 The Asymptotic Distribution of the M-L Estimator 40
2.4 Hypothesis testing 45
2.4.1 Definitions 45
2.4.2 Testing procedures... , 47
2.5 Concluding remarks 48
3 Random numbers and their applications..................... 49
3.1 Simulating random numbers from a uniform distribution 50
3.2 Quality of random number generators 53
3.2.1 Randomness test 54
3.2.2 Uniformity test 56
3.2.3 Independence test 58
3.2.4 Visual testing 58
3.3 Simulating random numbers from specific distributions 59
3.4 Simulating random numbers for general CDF 61
3.5 Simulating vector random numbers 64
3.6 Concluding remarks 66
4 Approximation and B-spline functions 67
4.1 Approximation and best approximation , 69
4.2 Polynomial basis 72
4.3 B-splines 77
4.3.1 Definitions 77
4.3.2 B-spline basis sets 81
4.3.3 Linear independence of B-spline functions 82
4.3.4 properties of B-splines 82
4.4 Two-dimensional B-splines 87
4.5 Concluding remarks „ , 87
5 Disorder, entropy and entropy estimation... 89
5.1 Disorder and entropy 89
5.1.1 Entropy of finite schemes 92
5.1.2 Axioms of entropy 94
5.2 Kullback information and model uncertainty 97
5.3 Estimation of entropy based on large samples 105
5.3.1 Asymptotically unbiased estimators of four basic entropies 107
5.3.2 Asymptotically unbiased estimator of TSE and AIC 114
5.4 Entropy estimation based on small sample 118
5.5 Model selection 119
5.5.1 Model selection based on large samples 120
5.5.2 Model selection based on small samples 126
5.6 Concluding remarks 128
6 Estimation of 1-D complicated distributions based on large
samples............................................... 129
6.1 General problems about pdf approximation 130
6.2 B-spline approximation of a continuous pdf 132
6.3 Estimation 135
6.3.1 Estimation from sample data 135
6.3.2 Estimation from a histogram 137
6.4 Model selction 140
6.5 Numerical examples 144
6.6 Concluding Remarks 156
Appendix: Non-linear programming problem and the uniqueness of the
solution , 159
7 Estimation of 2-D complicated distributions based on large
samples ., 163
7.1 B-Spline Approximation of &2-Dpdf 164
7.2 Estimation 167
7.2.1 Estimation from sample data 167
7.2.2 Computation acceleration 169
7.2.3 Estimation from a histogram 170
7.3 Model selection 173
7.4 Numerical examples , 174
7.5 Concluding remarks 186
8 Estimation of 1-D complicated distribution based on small
samples... ..,..,.,.. .,.,...,., , 189
8.1 Statistical influence of small sample on estimation 190
8.2 Construction of smooth Bayesian priors 192
8.2.1 Analysis of statistical fluctuations 192
8.2.2 Smooth prior distribution of combination coefficients 194
8.3 Bayesian estimation of complicated pdf 198
8.3.1 Bayesian point estimate 198
8.3.2 Determination of parameter co2 200
8.3.3 Calculating b and determinant of |FTF| 203
8.4 Numerical examples 204
8.5 Application to discrete random distributions 209
8.6 Concluding remarks 210
8.6.1 Characterization of the method 210
8.6.2 Comparison with the methods presented in Chapter 6 211
8.6.3 Comments on Bayesian approach 211
9 Estimation of 2-D complicated distribution based on
small samples 213
9.1 Statistical influence of small samples on estimation 213
9.2 Construction of smooth 2-d Bayesian priors 216
9.2.1 Analysis of statistical fluctuations 216
9.2.2 Smooth prior distribution of combination coefficients 217
9.3 Formulation of Bayesian estimation of complicated pdf. , 219
9.3.1. Bayesian point estimate 219
9.3.2 Determination of parameter a»2 221
9.4 Householder Transform 223
9.5 Numerical examples 225
9.6 Application to discrete random distributions 228
9.7 Concluding remarks 229
Appendix: Householder transform , 230
A.I Tridiagonalization of a real symmetric matrix 230
A.2 Finding eigenvalues of a tridiagonal matrix by bisection method 234
A.3 Determing determinant of a matrix by its eigenvalues 235
10 Estimation of the membership function,,...,. 237
10.1 Introduction.. 237
10.2 Fuzzy experiment and fuzzy sample 242
10.2.1 How large is large? 242
10.2.2 Fuzzy data in physical sciences 242
10.2.3 B-spline Approximation of the membership functions 244
10.3 ME analysis 247
10.4 Numerical Examples 248
10.5 Concluding Remarks 253
Appendix: Proof of uniqueness of the optimum solution 255
11 Estimation of distribution by use of the maximum entropy method
259
11.1 Maximum entropy , 260
11.2 Formulation of the maximum entropy method 265
11.3 B-spline representation of $ (x) 268
11.4 Optimization solvers 270
11.5 Asymptotically unbiased estimate of At 271
11.6 Model selection 272
11.7 Numerical Examples 273
11.8 Concluding Remarks 279
12 Code specifications 281
12.1 Plotting B-splines of order 3 281
12.1.1 Files in directory B-spline 281
12.1.2 Specification 281
12.2 Random number generation by ARM 282
12.2.1 Files in the directory of random 282
12.2.2 Specifications 282
12.3 Estimating 1-D distribution using B-splines 283
12.3.1 Files in the directory shhl 283
12.3.2 Specifications 283
12.4 Estimation of 2-D distribution: large sample ..,.,.... 284
12.4.1 Files in the directory shd2 284
12.4.2 Specifications 284
12.5 Estimation ofl-D distribution from a histogram 285
12.5.1 files in the directory shhl 285
12.5.2 Specifications 285
12.6 Estimation of 2-D distribution from a histogram 286
12.6.1 Files in the directory shhl 286
12.6.2 Specifications 286
12.7 Estimation of 2-D distribution using RBF , 287
12.7.1 Files in the directory shr2 287
12.7.2 Specifications 287
Bibliography 289
Index 295
Another Information Theory Books
Download
No comments:
Post a Comment