Statistical Estimation

Estimation deals with how we use a sample to approximate the unknown parameters of a population. This poster brings together the basic theory and formulas for the properties of estimators, estimation methods and confidence intervals.

Estimators and Properties

• Estimator: a function of the sample, $\hat{\theta }=T(X_1,\dots ,X_n)$
• Unbiasedness: $E(\hat{\theta })=\theta$
• Bias: $b(\hat{\theta })=E(\hat{\theta })-\theta$
• Mean squared error: $MSE(\hat{\theta })=Var(\hat{\theta })+b{(\hat{\theta })}^2$
• Consistency: the estimator converges in probability to $\theta$ as $n\to \infty$

Efficiency and Sufficiency

• Fisher information: $I(\theta )=-E\left[\frac{{\partial }^2}{\partial {\theta }^2}\ln f(X;\theta )\right]$
• Cramér–Rao inequality: for an unbiased estimator, $Var(\hat{\theta })\ge 1/(nI(\theta ))$
• Efficient estimator: attains the Cramér–Rao lower bound
• Sufficient statistic: Neyman factorization criterion
• Rao–Blackwell and Lehmann–Scheffé: from a sufficient statistic to the minimum variance unbiased estimator

Estimation Methods

• Method of moments: we equate the sample moments with the theoretical ones
• Maximum likelihood: the $\hat{\theta }$ that maximizes $L(\theta )=\prod f(x_i;\theta )$
• Likelihood equation: $\frac{\partial }{\partial \theta }\ln L(\theta )=0$
• Invariance property: if $\hat{\theta }$ is the maximum likelihood estimator of $\theta$, then $g(\hat{\theta })$ is that of $g(\theta )$

Confidence Intervals

• Interpretation: over repeated samples, $100(1-\alpha )\%$ of the intervals contain the parameter
• Mean, known variance: $\bar{x}\pm z_{\alpha /2}\sigma /\sqrt{n}$
• Mean, unknown variance: $\bar{x}\pm t_{n-1,\alpha /2}s/\sqrt{n}$
• Proportion: $\hat{p}\pm z_{\alpha /2}\sqrt{\hat{p}(1-\hat{p})/n}$
• Variance: based on the ${\chi }_{n-1}^2$ distribution