big list - What are some fundamental "sources" for the appearance of pi in mathematics?

As for the normal distribution, you can characterize it as the unique distribution with the following properties:

Let $X_1, X_2, cdots X_n$ be independent identically distributed normal random variables. Then the joint distribution of the vector $X=(X_1, X_2, cdots X_n)$ is the same as that of $AX$ where $A$ is any orthogonal matrix. So the normal distribution is intimately related to the geometry of real inner product spaces.

The $pi$ comes from the fact that you can integrate such a distribution by first integrating over a sphere and then integrating over $[0,infty]$. Because the distribution is orthogonally invariant, you pick up a constant corresponding to the area of the sphere. For $n=2$ you get the circle, and this is the usual calculation for computing the normalization constant for the normal distribution.

So then the mystery becomes: given that the normal distribution is so closely tied to inner product spaces, why does it show up all the time? The central limit theorem tells us that all that really matters in large scale limits are the first and second moments. The first moment can always be eliminated by re-centering. So all that matters is the second moment. But the second moment comes from the covariance, which is an inner product! (technically, only once you restrict to re-centered random variables, but we are doing that)

I'd venture a guess that most, if not all, appearances of $pi$ in statistics boil down to this fact that covariance is an inner product, and the fact that spheres, which are the norm-level sets for inner product spaces, have areas related to $pi$