Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The specification of permutations in anova.cca() has issues, and the number of permutations should be 999, not 1000. #17

Open
pedrohbraga opened this issue Apr 26, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@pedrohbraga
Copy link
Member

pedrohbraga commented Apr 26, 2023

The definition of the argument to change permutations in the presentations and in the books is done as in anova.cca(... , step = 1000).

step is not an argument included in or used by anova.cca(). Somehow, anova.cca() does not throw errors if one uses gibberish arguments, but the argument to change the permutations is permutations, as in anova.cca(... , permutations = how(nperms = 999).

How how() works here is less trivial than just changing the number attributed to nperms and needs to be reviewed in this workshop. See this document for help.

Moreover, in certain slides, the number of permutations used is 1,000 instead of 999. There is no explanation provided for this choice.

The p-value for permutations in the models there is calculated as p = (N + 1) / (n + 1), where n is the number of permutations requested, and N is the number of values of the test statistics (any type) that have been as extreme as the test statistic for the observed data.

That addition of +1 is what explains why we often see 999 or 199 permutations instead of 1000 or 200.

The reason that a division of 90/(99 +1) is often preferred over a 90/(100 + 1) is that it makes p-values "pretty" or easily interpretable in terms of the number of test statistics from permuted values that came as extreme as the observed test statistic.

@pedrohbraga pedrohbraga added the bug Something isn't working label Apr 26, 2023
@pedrohbraga pedrohbraga changed the title The number of permutations in the permutation tests could be 999, not 1000. The specification of permutations in anova.cca() has issues, and the number of permutations should be 999, not 1000. Apr 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant