Index of /www/kankel/ph567/examples/Octave/statistical_analysis/SpearmanKendall

	Name	Last modified	Size	Description
	Parent Directory		-
	Kendall 1.00conf-10000trials.pdf	2020-02-14 13:18	8.6K
	Kendall99.00conf-10000trials.pdf	2020-02-14 13:18	8.7K
	Spearman 1.00conf-10000trials.pdf	2020-02-14 13:18	8.6K
	Spearman99.00conf-10000trials.pdf	2020-02-14 13:18	8.8K
	helpfiles.txt	2016-04-20 01:27	2.0K
	pval.m	2016-04-20 01:27	2.2K
	pval_results/	2016-04-20 21:15	-
	pval_ties.m	2016-04-20 10:58	2.4K
	pval_ties1.m	2016-04-20 10:58	2.4K
	pval_ties1_results/	2016-04-20 10:58	-
	pval_ties_results/	2016-04-20 10:58	-

Robust Measures of Correlation: Spearman's ρ and Kendall's τ

Spearman's ρ and Kendall's τ statistics are the bread and butter of robust correlation testing (see the lecture notes). Both are nonparametric tests. Both begin by replacing the data (x,y) with ranks (R,S). The choice of which statistic to use depends on several factors. Here are the bare essentials, from my perspective:

• Ease of interpretation. Kendall has a natural interpretation: it is an estimate of the difference between the probabilities that any two pairs of data points are concordant or discordant. Spearman is not so easy to characterize but is a direct nonparametric analog of Pearson's r.
• Versatility. Kendall is a bit more nonparametric (hence, less sensitive to bizarre data distributions) than Spearman. Kendall also explicitly compensates for rank-order ties (repeated data values), in a way that is intended to remove any resulting bias from the statistic.
• Computational speed. Spearman runs in N logN time, limited only by sorting the data to obtain ranks. Kendall has terms for every possible pair of data points, so requires N². That's a major difference for big datasets.

Statistical Significance

The traditional way of expressing statistical significance is with a P-value, often denoted α, which is defined as the probability that a statistic would reach a certain value under the null hypothesis. We use 1-α as the confidence. In the case of correlations, which are distributed symmetrically about zero and thus might be either significantly positive or negative, we will use a two-sided confidence. In other words, α is defined as the probability, under the hypothesis of independent data, that the absolute value of the correlation statistic exceeds a certain value.

Side Sermon: The Prosecutor's Fallacy

Let's think for just a moment what this means. In traditional confidence testing, we start by articulating a null hypothesis H. We then ask, under H, what is the probability of obtaining data D? Perhaps that's a little hard to answer, since D is a complicated object containing lots of information. So instead we use D to calculate a statistic S that we suppose to be sensitive to the truth or falsehood of H. The usual P-value is then defined as

A small value of α is typically taken as grounds to reject H. We must not, however, equate α with the posterior probability, P(H | S). Consider Bayes' Theorem, which follows from the interchangeability of arguments in the joint probability distribution:

The posterior probability is therefore:

In view of this complication, how meaningful is the typical frequentist assertion that the null hypothesis, H, may be "rejected" with "confidence" equal to 1 - α? What we must realize is that P(H) and P(S) are "prior" probabilities. That is, they encode knowledge available to us before the experiment was performed. For example, if we have pre-existing evidence favoring the null hypothesis, then it would require proportionately stronger evidence to counter that. If instead we are beginning from a tabula raza, we have no prior knowledge. The best we can say is something like P(H) = P(S) = 1/2. We could apply this reasoning to the first or best study on a particular question. In summary, traditional confidence levels assess each experiment on its own terms.

For a detailed discussion, see the Wikipedia article on the Prosecutor's Fallacy.

Should I publish a novel result with low α that conflicts, at least apparently, with strong prior evidence? I think so, provided that I have carefully reviewed and documented my experimental procedure, the data, the analysis, and any assumptions made along the way. If possible, I should try to replicate the result or have it replicated elsewhere. What I cannot do is jump to the conclusion that a single experiment demolishes everything that has gone before. The result may, however, raise questions that prove productive in the long term. If I and my colleagues routinely refrain from reporting or discussing results that are contrary to received wisdom, yet significant on their own terms, then we run the risk of an ossified scientific enterprise that is immune to discovery.

P-values for ρ & τ

Look at GNU Octave's help files for functions spearman and kendall. These functions do not offer built-in significance testing. We could gripe and complain, but this is free software.

How may we calculate P-values for the Spearman and Kendall statistics? In particular:

• The asymptotic (large N) distributions of these statistics are normal, and the help pages provide the variances σ_τ² and σ_ρ².
• How large must N be to use the normal distribution?
• What if N is small?
• Do repeated data values matter?

MonteCarlo results:

A rude surprise. When repeated values are encountered, the confidence thresholds for τ deviate markedly from those implied by the σ_τ² formula, even for large N. Surprisingly, it performs even worse than spearman with ties. Looking back at the help files, I realize that Octave's primitive kendall does not correct for repeated values!

Conclusion

1. Significance testing for both spearman and kendall.
2. An implementation of kendall that tests and compensates for tie values.

Page maintained by Charles Kankelborg