\documentclass[12pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{bigfoot}
\usepackage{hyperref}
\usepackage{amsmath}
\usepackage{amssymb}

\title{Bayesian Basketball}
\author{Charles Kankelborg}
\date{Revised April, 2024}

\newcommand{\D}{\mathbf{d}} % The data vector.
\newcommand{\M}{\mathcal{M}} % The Model.
\newcommand{\X}{\mathbf{x}}  % The model parameter vector.
\newcommand{\exercise}[1]{\bigskip\noindent \small\textbf{Exercise:\ }{#1}\bigskip\normalsize} % Exercises to ensure understanding

\interfootnotelinepenalty=10000  % prevent footnotes from running onto subsequent pages. 

\begin{document}

\maketitle
\begin{abstract}
    I introduce statistical inference using Bayes' theorem from minimal assumptions. As an example, I solve the
    following problem: A basketball player shoots $N$ free throws, and all of them are good. What is the probability
    that she will sink the next shot?
\end{abstract}

\section{Bayesian Inference}\label{sec:bayes}
In \S\ref{sec:derivation} I briefly introduce and justify Bayes' theorem. Then, in \S\ref{sec:inference}, I lay out the application of Bayes' theorem to parameter estimation.

\subsection{Bayes' Theorem}\label{sec:derivation}
Two logical propositions, $A$ and $B$, are capable of being either true or false. The expression $\Pr(A)$ means the probability that $A$ is true. The expression $\Pr(A|B)$ denotes a conditional probability, which is read as the probability that $A$ is true, given that $B$ is true. Now, the probability that \emph{both} $A$ and $B$ are true is given by a fundamental axiom of probability theory,
\[
    \Pr(A,B) = \Pr(A|B) \,\Pr(B).
\]
In words, the probability that both $A$ and $B$ are true equals the probability of $A$ under the assumption that $B$ is true, multiplied by the probability that $B$ is true. We take this as given, but I invite you to think about it carefully. Equivalently, I could have written
\[
    \Pr(B,A) = \Pr(B|A) \, \Pr(A).
\]
Notice, however, that $\Pr(A,B)$ and $\Pr(B,A)$ mean the same thing, the probability that both $A$ and $B$ are true. Consequently,
\[
    \Pr(A|B) \,\Pr(B) = \Pr(B|A)\,\Pr(A).
\]
 That is, the probability of $A$ given $B$, multiplied by the probability of $B$, is equal to the probability of $B$ given $A$, multiplied by the probability of $A$.  The above equation is Bayes' theorem. It is frequently rearranged to solve for one of the two conditional probabilities,
\[
    \Pr(A|B) = \frac{\Pr(B|A)\,\Pr(A)}{\Pr(B)}.
\]


\subsection{Statistical Inference: Parameter Selection}\label{sec:inference}
While the above expression of Bayes' theorem is technically correct, we still need to develop the conceptual framework to use it for statistical inference. In particular, we will consider the problem of \emph{parameter selection}.

Let us suppose that some data, $\D$, has been collected. We now assume that model $\M$ can be used to account for this data. The model has some number of tunable parameters, given by the parameter vector $\X$. We wish to use the data to asses the probability distribution for those parameters. We begin by using Bayes' theorem:
\begin{equation}\label{eq:bayestheorem}
    \boxed{ \Pr(\X | \D,\M) = \frac{\Pr(\D | \X,\M)\,\Pr(\X|\M)}{\Pr(\D|\M)}. }
\end{equation}
This form of Bayes' theorem will motivate the rest of the paper. Often, $\M$ is omitted to simplify the notation, but it behooves us to remember that the model has been assumed to be true. The thing we wish to calculate, $\Pr(\X | \D,\M)$, is called the \emph{posterior distribution}.
Can we evaluate all the quantities on the right hand side of the equation? The most straightforward part is probably $\Pr(\D | \X,\M)$. After all, if assuming a model and choosing all its parameters does not give us enough information to assess the likelihood that our observations turn out a certain way, then the model is not very useful! 

The factors $\Pr(\X|\M)$ and $\Pr(\D|\M)$ require careful thought. The notation seems to indicate that we are to calculate the probabily of some choice of model parameters without any knowledge of the data, and the probability of the data without having the slightest idea about the parameter values.  That seems like an ill-defined problem, so I need to back up and think about $\Pr(\X|\M)$ and $\Pr(\D|\M)$ in a different way.

\subsubsection{The Prior}\label{sec:theprior}
We cannot calculate the probabilities $\Pr(\X|\M)$ and $\Pr(\D|\M)$ without starting from some assumption, so what is that starting point? The most important point is that $\Pr(\X|\M)$ and $\Pr(\D|\M)$ \emph{must be compatible; that is, they must be derived from exactly the same set of assumptions}. We already know that both assume the same model, but it is usually necessary to add something. The way this is usually accomplished is to begin with an assumed parameter distribution $\Pr(\X|\M)$. We will simply call this the \emph{prior}, a too-short name that fails to express the weight on this starting assumption.\footnote{If it seems to you that assuming a probability distribution for $\X$ is begging the question, then I have done my job! That is exactly the danger we must keep in mind. One of the strengths of Bayesian inference is that it forces us to lay our cards on the table, making all assumptions explicit.}

If something is already known about the likelihoods of parameter choices before the data $\D$ are taken, then that information should be encoded in $\Pr(\X|\M)$. If not, then usually the most honest choice is a \emph{flat} prior, meaning that all parameter values are equally likely. If $\X\in\mathbb{R}^n$, the proper normalization of the prior would be
\begin{equation}\label{eq:priornorm}
    \int dx_1 \int dx_2 \cdot\cdot\cdot \int dx_n \, \Pr(x_1,x_2,...,x_n|\M) \equiv \int \Pr(\X|\M))\,d^nx = 1.
\end{equation}
The parameter integrals above are definite integrals over the appropriate domain for each parameter. Once we have defined a properly normalized multivariate distribution, the calculation of $\Pr(\D|\M)$ will follow directly, as described in section \ref{sec:marginalization}.


\subsubsection{Marginalization}\label{sec:marginalization}
Once we have the prior, it becomes possible to work out $\Pr(\D|\M)$. We do this by \emph{marginalizing} over all possible values of the model parameters:
\begin{equation}\label{eq:marginalizePD}
    \Pr(\D|\M) = \int \Pr(\D|\X,\M)\,\Pr(\X|\M) \,d^nx.
\end{equation}
Notice how the prior, $\Pr(\X|\M)$, was required for marginalization. This establishes the required consistency between $\Pr(\X|\M)$ and $\Pr(\D|\M)$. Notice that if the prior not normalized, the normalization ultimately drops out in Bayes' theorem. 

% \exercise{Show that if the prior is properly normalized, then $\Pr(\D|\M)$ is properly normalized.}

\subsubsection{The flat prior}
Let's think for a moment about a single model parameter, $x$. It is common practice to introduce a prior of the form
\begin{equation}\label{eq:flat1}
    f(x) \equiv \Pr(x|\M) = c,
\end{equation}
where $c$ is a constant over the appropriate domain of $x$. This is called a ``flat'' prior. The idea behind a flat prior is to eliminate bias in parameter selection. Suppose, for example, that $x$ is an astronomical distance whose uncertainty covers many orders of magnitude. It might then be more natural to work in terms of $y=\log x$:
\begin{equation}\label{eq:flat2}
    g(y) \equiv \Pr(y|\M).
\end{equation}
By conservation of probability,
\begin{align*}
    f(x)\,dx &= g(y)\,dy\\
    f(x) &= g(y)\,\frac{dy}{dx} = \frac{g(y)}{x}\\
    \therefore g(\log x) &= cx
\end{align*}
Notice how the change of parameters affects the functional form of the prior. The distribution cannot be uniform in both $x$ and $\log x$. So, which would be free of bias: flat $\Pr(x|\M)$, flat $\Pr(\log x|\M)$, or something else? Sometimes, the answer is obvious in context. But it is always important to spell out the prior clearly.


\subsubsection{Posterior Distribution}
We are now in a position to write down the probability distribution for the model parameters. Once again, this is called the posterior:
\begin{equation}\label{eq:pramdistro}
    \Pr(\X | \D) = \frac{\Pr(\D|\X)\,\Pr(\X)}{
        \int  \Pr(\D|\X') \, \Pr(\X')\,d^nx'}.
\end{equation}
I have left the model $\M$ unstated to keep the notation compact. Primes are used to distinguish all the independent variable that disappears when the definite integral is calculated to marginalize over the parameter space. This notation helps us to see that the denominator does not introduce any dependence on the parameter vector, $\X$. If the prior is flat, then the posterior takes the same functional form as $\Pr(\D|\X)$, and Bayes' theorem serves only to renormalize that distribution over the domain of the model parameter space. Nevertheless, correct normalization is important if we want to calculate a mean, a median, or confidence intervals from the distribution.


The posterior distribution, $\Pr(\X | \D)$, is ostensibly the answer we have been looking for. Or is it? We have a range of possible models, described by the parameters $x_1, x_2,..., x_n$. We also have a probability distribution for those parameters. We might prefer to have a well-defined best choice of parameters to describe the situation, $\X_{\mathrm{best}}$. We will see from the basketball example in the next section that many possible choices of $\X_{\mathrm{best}}$ could be made using $\Pr(\X | \D)$. 

\section{The Free Throw Problem}
We now pose the problem:
\begin{quote}
    A basketball player shoots $N$ free throws,
    and all of them are good. What is the probability that she will sink the next shot?
\end{quote}
This kind of statistical problem has many applications, including the analysis of risk in engineering.\footnote{For example: A new kind of single-use solid rocket booster has had $N$ flights, all successful. What is the likelihood that the next flight will be a success?} Bayesian inference, as described in \S\,\ref{sec:bayes}, gives us a way to attack the problem. 

\subsection{The Model and the Prior}
In order to formulate a prior, I need to decide on a parameterized model. Let's characterize our basketball player by a single parameter, $q$, which is the probability of sinking a free throw. I have no other prior knowledge to incorporate, so I will use a flat prior:
\[
    \Pr(q)= 1, \quad 0 \le q \le 1.
\]
Note the recursive use of the word probability: $\Pr(q)$
is the probability that the player's free throw probability has a certain value, $q$. This distribution is normalized:
\[
    \int_0^1 \Pr(q)\,dq = 1.
\]

\exercise{Choosing a parameterized model restricts me from considering other models. What are some possible considerations that I have ignored by characterizing the player's free throw shooting ability with a single parameter, $q$?}

\subsection{Finding the Posterior Distribution}
The next step is to calculate $\Pr(\D|q)$, the probability that the data would result, as a function of model parameter $q$. The data is that we have observed $N$ good shots in a row; the sole model parameter is $q$. The chance of putting the ball through the hoop on any particular try is $q$, so the chance of two baskets in a row is $q^2$. Similarly, the chance of $N$ baskets in a row is 
\[
    \Pr(\D|\X) = \Pr(N|q) = q^N.
\]
Now, I marginalize over all possible values of $q$ to find $\Pr(\D)$:
\[
    \Pr(\D) = \Pr(N) = \int \Pr(N|q)\,\Pr(q)\,dq = \int_0^1 q^N\,dq = \frac{1}{N+1}.
\]
The posterior distribution is therefore
\begin{equation}\label{eq:posteriorN}
    \Pr(q|N) = \frac{\Pr(N|q)\,\Pr(q)}{\Pr(N)} = (N+1)\,q^N.
\end{equation}

\subsection{Interpretation of the Posterior}
The posterior distribution we have derived is zero for $q=0$ and grows monotonically as $q$ increases to $1$. This makes a certain amount of sense. So long as $N>0$, it has been demonstrated that she can hit the basket. If $N$ is large, our player appears to be a very good shot.  If we could decide on a single representative or ``best'' value for $q$, that would be our answer: the chance of making the next free throw is just $q_{\mathrm{best}}$. Some possible options for interpreting the posterior distribution would then include the usual measures of central tendency for a univariate (single parameter) probability distribution:
\begin{enumerate}
    \item Mode: $q_{\mathrm{best}}= \arg\max \Pr(q|N)$.
    \item Mean: $q_{\mathrm{best}}= \int_0^1 \Pr(q|N)\,q\,dq$.
    \item Median: $\int_0^{q_{\mathrm{best}}} \Pr(q|N)\,dq= \frac{1}{2}$.
\end{enumerate}
The mode is the \emph{maximum likelihood} estimate. Since the distribution is monotonically increasing, the maximum likelihood estimate is $q=1$. The larger $N$ becomes, the more plausible this estimate becomes. However, it seems strange that our result would not depend on $N$. Should we think this player is a perfect shot after only a few successful tries? The other two estimates of $q_{\text{best}}$ are more nuanced and will actually depend on $N$. How can we decide?

Let's look carefully at the original question: ``What is the probability that she will sink the \emph{next shot}?'' I denote the answer as $\Pr(N+1|\D)=\Pr(N+1|N)$, by which I mean ``the probability that the $N+1^{\text{st}}$ free throw will be good, given the data that the first $N$ free throws were good''. I propose to calculate this probability by marginalizing over all possible values of $q$:
\[
    \Pr(N+1|N) = \int_0^1 \Pr(q|\D)\,\Pr(N+1|q)\,dq.
\]
In terms of $q$, the probability of sinking the next free throw is $\Pr(N+1|q)=q$. The answer is therefore:
\begin{align}\label{eq:answer}
    \Pr(N+1|N) &= \int_0^1 \Pr(q|N)\,q\,dq \equiv \left<q\right> \nonumber \\
        &= \int_0^1 (N+1)\,q^{N+1}\,dq = \boxed{ \frac{N+1}{N+2}.  }
\end{align}
Notice that this corresponds to the mean value of the distribution, which is also called the \emph{expectation value} of $q$, written $\left<q\right>$. The reasonableness of this result becomes more apparent when we check extreme values:
\begin{enumerate}
    \item In the limit $N\rightarrow \infty$, the player is a perfect shot: $\left<q\right>\rightarrow 1$.
    \item If $N = 0$, then we have no evidence, and $\left<q\right>=\frac{1}{2}$.
\end{enumerate}


\exercise{Calculate the median value of the posterior distribution, also called $q_{50}$, as in 50$^{\text{th}}$ percentile. How does the median behave in the limits of extreme values of $N$?}

\exercise{Perhaps you are not satisfied by a single ``best'' value for $q$. Using the posterior, calculate $q_{90}$, the smallest value of $q$ to 90\% confidence. In other words, there is only a 10\% chance that $q<q_{90}$.}

\section{Further Exploration of the Problem}
\subsection{Non-flat prior} \label{sec:nonflat}
It might bother us to begin with a flat prior. Perhaps we have some minimal intuition about the likely results. For example, it seems unreasonable to expect that $q=0$ because the player is presumed to have at least some skill. It is also unlikely that $q=1$, because no athlete is perfect. A simple prior that goes to zero at the ends of the interval is:
\[
    \Pr(q) = 6q(1-q).
\]
Note that this is properly normalized. How does the new prior affect the result? We still have $\Pr(N|q)=q^N$. Marginalizing over $q$,
\[
    \Pr(N) = \int_0^1 6q^{N+1}(1-q)\,dq = \frac{6}{(N+2)(N+3)}
\]
The posterior is then:
\[
    \Pr(q|N) = (N+2)(N+3)(1-q)q^{N+1}.
\]
\exercise{Calculate $\left<q\right>$ for the above posterior. Compare to the result for the flat prior.}

\subsection{Subsequent Trips to the Court}
In \S\ref{sec:nonflat}, we modified the prior somewhat arbitrarily. A better use of a non-flat prior is to take account of results from previous experiments. For example:
\begin{quote}
    On Monday, a basketball player hits $N$ free throws out of $N$ attempts. She returns to the court on Tuesday, and hits $K$ out of $K$. What is the probability that she will sink the next free throw?
\end{quote}
Provided that the conditions have not changed, we can take both practice sessions into account in calculating the likelihood of hitting on the next attempt. The simplest approach would be to calculate it all at once. According to equation \ref{eq:answer}, $N+K$ baskets in a row implies that
\begin{equation}\label{eq:answerNK}
    \boxed{  \left<q\right> = \frac{N+K+1}{N+K+2}.  }
\end{equation}

It is more interesting (though more tedious) to use this example to demonstrate how Bayes' theorem takes account of the prior, $\Pr(\X)$. We argue based on equation \ref{eq:posteriorN} that the posterior calculated from the first session should be the prior at the time we make the second trip to the court:
\[
    \Pr(\X) \rightarrow \Pr(q|N) = (N+1)\,q^N.
\]
We must also marginalize over this distribution to get $\Pr(\D)$ for the second session:
\begin{align*}
    \Pr(\D) \rightarrow \Pr(K|N) &= \int_0^1 \Pr(K|q) \Pr(q|N)\,dq \\
        &= (N+1)\int_0^1 q^{K+N}\,dq = \frac{N+1}{N+K+1}.
\end{align*}
Bayes' theorem for the posterior distribution, taking both datasets into account, is:
\begin{equation}\label{eq:combineNK}
    \Pr(q|K,N) = \frac{\Pr(K|q) \, \Pr(q|N)}{\Pr(K|N)}.
\end{equation}
The posterior works out to be:
\[
    \Pr(q|K,N) = \frac{q^K (N+1)\, q^N}{(N+1) / (N+K+1)} = (N+K+1)\, q^{N+K},
\]
which is exactly what we would have obtained by simply substituting $N\rightarrow N+K$ in equation \ref{eq:posteriorN}. The expectation value (mean) of this posterior distribution is obviously what we predicted a moment ago, based on simpler reasoning (equation \ref{eq:answerNK}). This result should increase our confidence that \emph{Bayes' theorem properly takes into account prior information}. 

Often, prior information about a parameter or set of parameters comes in a very different form, and from a very different experiment, than the present experiment. If we are prepared to assume that the same model, with the same parameters, should hold on both occasions, we can use the method illustrated by equation \ref{eq:combineNK} to build a posterior distribution that combines what has been learned from both experiments.


\subsection{Example with Misses}
So far, our star player has never missed. Let us generalize the problem as follows:
\begin{quote}
    A basketball player attempts $N$ free throws, and sinks $M$ of them.
    What is the probability that she will sink the next free throw?
\end{quote}
Of course, $M \le N$. We now have a more complicated data set, $\D$. Assuming underlying probability $q$, we can calculate $\Pr(\D|q)$ using the binomial distribution.

\exercise{Work out the rest of the example. Assuming a flat prior, what is $\Pr(q|M,N)$? What is $\left<q\right>$? Explain the deviation from $\left<q\right>=M/N$.}

\clearpage

\section{Summary}

\begin{enumerate}
    \item Bayes' theorem (equation \ref{eq:bayestheorem}) provides a rigorous way of calculating a posterior distribution, $\Pr(\X|\D)$, that can be used for statistical inference about model parameters $\X$ based on data $\D$.
    \item When we calculate the posterior distribution, it is essential to derive $\Pr(\D)$ and $\Pr(\X)$ using compatible assumptions. That means we marginalize over the model parameter space according to equation \ref{eq:marginalizePD}.
    \item Bayes' theorem allows us to take prior information about the parameters, $\Pr(\X)$, into account.
    \item The interpretation of the posterior depends on the question being asked. In the case of parameter selection, for example, we might prefer the maximum likelihood, the expectation value, or some other descriptor of the posterior. If we are being more careful, we might integrate the posterior to obtain a 1- or 2-sided confidence interval.
\end{enumerate}

\end{document}

**********************************  B O N E   Y A R D  ********************************************

\begin{abstract}
    A basketball player comes to the court to practice. She shoots $N$ free
    throws, and all of them are good. What is the probability that she will
    sink the next shot? I will use this simple problem as a motivation to develop Bayes' theorem and the necessary conceptual framework to apply
    it to practical statistical problems. My aim is to introduce statistical inference using Bayes' theorem from minimal assumptions, and to demonstrate that there is no one recipe for the application of these ideas, even when we are only trying to estimate one parameter. 
\end{abstract}



\exercise{Show that posterior distribution (equation \ref{eq:pramdistro}) is properly normalized:
\begin{equation}
    \int \Pr(\mathbf{a} | \D) \,d^na = 1.
\end{equation}
If we should forget to properly normalize $P(\D|\mathbf{a})$ or $P(\mathbf{a})$, would this affect the normalization of $P(\mathbf{a} | \D)$?
}

\section{How Many Tickets in the Hat?}
\begin{quote}
    A hat is filled with sequentially numbered tickets, from $1$ to $n$.
    The hat is shaken to thoroughly randomize the tickets. From the hat,
    you draw a ticket numbered $k$. What is your best estimate of the 
    number of tickets, $N$?
\end{quote}
This is a famous problem.\footnote{See \url{https://en.wikipedia.org/wiki/Maximum_likelihood_estimation#Discrete_uniform_distribution}.} It is a special case of the so-called German Tank Problem,\footnote{See \url{https://en.wikipedia.org/wiki/German_tank_problem}.} which was important to Allied intelligence in World War 2.
I have posed it because it will demonstrate that we can't rely on just one recipe for Bayesian inference.

The model in this case is represented by the parameter $n$, which we hope to guess as accurately as possible. If there are $n$ tickets, then the probability that we pick ticket number $k$ is
\[
    P(\D|\M) = P(k|n) = \begin{cases}
        \frac{1}{n}, & 1 \le k \le n;\\
        0, & \text{otherwise.}
    \end{cases}
\]
This is normalized in the sense that 
\[
    \sum_{k=1}^{n} P(k|n) = 1.
\]
We have no guidance about $n$, so apparently we need a flat prior,
\[
    P(\M) = P(n) = C,\quad \text{a constant}.
\]
This is problematic, because we can't normalize it:
\[
    \sum_{n=1}^{\infty} P(n) = 1 \implies C \rightarrow 0.
\]  
Nevertheless, let us proceed to marginalize over $n$:
\[
    P(\D) = P(k) = \sum_{n'=k}^{\infty} P(k|n') \, P(n') = \sum_{n'=k}^{\infty} \frac{C}{n'}
\]
Bayes' theorem yields the posterior distribution
\[
    P(n|k) = \frac{P(k|n)\,P(n)}{P(k)} =
        \frac{\displaystyle \frac{C}{n}}{\displaystyle \sum_{n'=k}^{\infty} \frac{C}{n'}}
        = \frac{1}{\displaystyle n \sum_{n'=k}^{\infty} \frac{1}{n'}}, 
        \quad 1 \le k \le n 
\]
This distribution is not normalizable, and both the mean and the median diverge toward infinity. More The only apparent choice using Bayes' theorem is to take the maximum likelihood estimate, $n=k$.
However, this is a lower limit, so it is a biased estimate. If we start from pretending to know $n$, then we can make the argument that, on average, $k=(n+1)/2$. So, perhaps a better estimate would be $n=2k-1$. Nevertheless, we cannot reach this conclusion from the posterior distribution. 

There appears to be no firm argument that any finite result would constitute an unbiased estimate. It could be argued that the defect is in a poorly posed problem, not a limitation of Bayesian inference.

\subsection{Modified Prior}
Suppose that there is a known upper limit on the number of tickets, $n \le N$. How would that change the results?
We still have $P(k|n)=1/n$. Again assuming a flat prior, 
\[
    P(n) = \frac{1}{N}, \quad n \le N.
\]
Marinalizing over $n$, 
\[
    P(k) = \sum_{n'=k}^N \frac{1}{Nn'}.
\]
This and subsequent sums will lead to a rather impenetrable expression for the posterior distribution. Let's assume $k \gg 1$, and treat $n$ as a continuous variable. Inspired by the midpoint rule, I will approximate the above sum as:
\[
    P(k) = \int_{n'=k-\frac{1}{2}}^{N+\frac{1}{2}} \frac{dn'}{Nn'} 
    = \frac{1}{N}\ln\left( \frac{2N+1}{2k-1} \right).
\]
The posterior distribution then becomes
\[
    P(n|k)=\frac{1}{n\,\ln\left( \frac{2N+1}{2k-1} \right)}, \quad 1 \ll k \le n \le N.
\]
Again taking my cue from the decision to replace sums by integrals and using the trapezoid rule, I exceed the integer domain of $n$ by $\frac{1}{2}$ at both limits:
\[
    \left<n\right> = \int_{n=k-\frac{1}{2}}^{N+\frac{1}{2}} \frac{dn}{\ln\left( \frac{2N+1}{2k-1} \right)}
    = \frac{N-k+1}{\ln\left( \frac{2N+1}{2k-1} \right)}.
\]
This readily illustrates the difficulty with the original problem, that the expectation value of $n$ goes to infinity if the number of tickets is unconstrained ($n\rightarrow \infty$).

\exercise{Using the assumptions that $k$ and $N$ are large, show that
\[
    \ln\left( \frac{2N+1}{2k-1} \right) = \ln\left(\frac{N}{k}\right) + \frac{1}{2N} + \frac{1}{2k}.
\]
Then show that sensible answers are obtained from the above expressions for the posterior distribution and expectation value of $n$ in the limiting case $k=N$.}

\exercise{Using the techniques of this section for taking the limit of large $k$, investigate the prior $P(n)\propto n^{-\alpha}$, $\alpha > 1$. Notice that the normalization of $P(n)$ is not strictly needed to find the posterior distribution or the expectation value. It can be argued that, by replacing the sharp cutoff at $N$, this prior does not impose a preferred scale for $n$. Nevertheless, it is difficult to justify the choice of the free parameter $\alpha$.}

\section{Summary}

\begin{enumerate}
    \item Bayes' theorem (equation \ref{eq:bayestheorem}) provides a rigorous way of calculating a posterior distribution, $P(\M|\D)$, that can be used for statistical inference about a model $\M$ based on data $\D$.
    \item Bayes' theorem allows us to take prior information about the model, $P(\M)$, into account.
    \item When we calculate the posterior distribution, it is essential to derive $P(\D)$ and $P(\M)$ using compatible assumptions. That means we marginalize over the model parameter space according to equation \ref{eq:marginalizePD}.
    \item The interpretation of the posterior depends on the question being asked. In the case of
    parameter selection, for example, we might prefer the maximum likelihood, the expectation value,
    or some other descriptor of the posterior.
\end{enumerate}

\end{document}
