Point Estimation: Method of Moments and Maximum Likelihood: every key term you need (+ practice quiz)
25 flashcard terms for Probability & Statistical Inference Topic 6, written to match the course framework. Study them here, then drill them as interactive flashcards, or test yourself with the 15-question quiz — free, no account needed.
A rule turning sample data into a single number intended to stand in for an unknown parameter. The rule is a random variable; the number produced from one data set is the estimate.
Estimand
The population quantity an analysis is actually trying to learn about. Confusing the estimand with the estimator is the source of much muddled reporting, since a good rule aimed at the wrong target is useless.
Bias of an estimator
The difference between the expected value of the estimator and the true parameter. Zero bias means the procedure is centred on the target across repeated samples, not that any one estimate is correct.
Consistency
The property that an estimator converges in probability to the true parameter as the sample grows. A biased estimator can still be consistent if its bias shrinks to zero, which is common in practice.
Efficiency
A comparison of estimators by variance, with the smaller-variance rule preferred among those with the same bias. Efficiency is a relative notion and is usually measured against a theoretical lower bound.
Sufficiency
A statistic is sufficient when the conditional distribution of the data given it does not involve the parameter, so it captures all parameter-relevant information. Estimators may then be built from it without loss.
Factorisation criterion
A statistic is sufficient exactly when the likelihood factors into one piece depending on the data through that statistic and the parameter, and another piece free of the parameter. It avoids computing conditional distributions.
Minimal sufficiency
A sufficient statistic that is a function of every other sufficient statistic, achieving the greatest possible data reduction without losing parameter information. It is the natural summary to base inference on.
Method of moments
Equating sample moments to their theoretical counterparts and solving for the parameters. It is simple and generally consistent, but it can produce estimates outside the allowed parameter range.
Likelihood function
The joint density or mass of the observed data viewed as a function of the parameter with the data held fixed. It is not a probability distribution over the parameter and does not integrate to one.
Log-likelihood
The logarithm of the likelihood, which turns products into sums and is easier to differentiate and to compute stably. Maximising it gives the same answer since the logarithm is strictly increasing.
Maximum likelihood estimator
The parameter value making the observed data most probable under the assumed model. It is typically consistent and asymptotically normal and efficient, though it can be biased in small samples.
Score function
The derivative of the log-likelihood with respect to the parameter. Its expectation at the true parameter is zero, and setting the observed score to zero gives the usual likelihood equations.
Fisher information
The variance of the score, equivalently the expected negative second derivative of the log-likelihood. Larger values mean a sharply peaked likelihood and therefore more precise estimation.
Cramer-Rao lower bound
For unbiased estimators satisfying regularity conditions, the variance is at least the reciprocal of the Fisher information. Attaining it certifies that no unbiased competitor can do better.
Regularity conditions
Technical requirements such as a parameter-free support and differentiability under the integral sign. When they fail, as with a uniform upper endpoint, standard likelihood theory and its bounds do not apply.
Invariance of maximum likelihood
If a parameter has a maximum likelihood estimate, any function of that parameter is estimated by applying the function to the estimate. Unbiasedness enjoys no such invariance and is generally destroyed by transformation.
Asymptotic normality of the estimator
Under regularity conditions the maximum likelihood estimator is approximately normal around the true value with variance given by the inverse Fisher information. This supports large-sample intervals and tests.
Bias-variance trade-off
Accepting some bias can reduce variance enough to lower overall mean squared error. Shrinkage and regularised estimators deliberately exploit this, which is why unbiasedness alone is a weak criterion.
Minimum variance unbiased estimator
The unbiased rule with the smallest variance for every parameter value. Conditioning an unbiased estimator on a complete sufficient statistic produces it, a construction known as Rao-Blackwellisation.
Completeness of a statistic
The condition that no nonzero function of the statistic has expectation zero for every parameter value. It is what turns Rao-Blackwellisation into a uniqueness result rather than just an improvement.
Robustness of an estimator
How little an estimator changes under small departures from the assumed model or under a few extreme observations. The sample median is robust to outliers while the sample mean is not.
Plug-in estimator
An estimate formed by applying the population formula to the empirical distribution of the sample. It is intuitive and often consistent, but Jensen's inequality means nonlinear formulas usually introduce bias.
Bayesian point estimate
A summary of the posterior distribution such as its mean, median or mode. The posterior mode coincides with the maximum likelihood estimate when the prior is flat over the relevant range.
Standard error of an estimate
An estimate of the estimator's own standard deviation, often obtained from the inverse observed information or by resampling. It converts a bare number into a statement with quantified uncertainty.