Great questions! Some suggestions / comments below in bold:
- Although not as “pure” as extracting the raw data, we are fairly certain that calculating the effect size from F statistics is legitimate. Is that correct?
Yes, you can calculate effect sizes from F-statistics and other test statistics, BUT, you should to be very careful when extracting statistics from mixed models for a number of reasons. Shinichi and I have spent some time thinking about this problem, some of the issues we outline in the papers below. Actually, taking test statistics from models in general, come with some important caveats you need to consider the least of which is the fact the models themselves are likely constructed with non-overlapping covariates. So, best to be careful and if you can stick with the raw data whenever possible. Most equations with F stats are based on simple one-way ANOVA style analyses.
D.W.A. Noble, M. Lagisz, R. O’Dea, S. Nakagawa (2017) Non-independence and sensitivity analyses in ecological and evolutionary meta-analyses. Molecular Ecology, 26, 2410-2425.
Nakagawa, S., D.W.A. Noble, A.S. Senior, M. Lagisz (2017) Meta-evaluation of meta-analysis: 10 appraisal questions for biologists. BMC Biology, 15 (1), 18.
Nakagawa, S. and Cuthill, I. (2007). Effect size, confidence interval and statistical significance: a practical guide for biologists. Biological Reviews, 82, 591-605.
- How reliable are measures of effect size (Hedges d) based on F statistics vs. sample sizes? To that end, how important is it to obtain raw values if the F statistics are reported?
Not quite sure what you mean by “F-statistics” vs. “sample sizes” because even with an F-statistic you still require the sample size (Although some formulas may use degrees of freedom).
- It seems that the majority of studies doing the kind of things we’re trying to do use Hedges d rather than other effect size metrics (e.g., log response ratio). Are there particular reasons why one would use Hedges d rather than, say, the log response ratio?
This is a great question. We outline in our paper the tradeoffs between the two See Noble et al. 2017. Mol. Ecol. If you think you have lots of non-independence, lnRR maybe the way to go, but only certain measurement variables can be used as it’s a ratio (so need interval data) and it can’t be converted to other effects sizes
- We may want to expand our scope and consider different traits (e.g. space use, escape speed, etc.). These metrics may require different statistical procedures for calculating effect size. Is it acceptable to compare different effect size metrics? We are assuming not, but did want to check.
In some cases this is fine to do, or even all you can do. Borenstein et al. 2009 is a great book to consult with converting between effect sizes.
Borenstein M, Hedges L, Higgens J, Rothstein H (2009) Introduction to Meta-Analysis. John Wiley & Sons Ltd, West Sussex, UK.
- Can effect sizes across different types of traits be combined in some manner to calculate an overall effect of a given consumer (predator/parasite) on a resource?
Not sure about this one. I’m not aware of an overarching effect size that will do this off the top of my head, but let me think about this
- Lastly, a number of our studies report data for multiple days (see attached screenshot). Other meta-analyses that we have found usually take only the last time point. If the effect is expected to increase consistently over time, perhaps that is a reasonable procedure. However, for our analysis, that may overlook effects that occur earlier in the interaction if, for example, the effects are expected to be most pronounced early on in the interaction. Is it acceptable to use, say, a mean of the daily values instead of a single time point, then calculate the effect size based on the treatments’ means?
This will likely be a bit tricky to implement in practice because you would also need to calculate a sampling variance for this combined effect, which I suppose could be done. There are some papers that discuss composite effect sizes (see below). You maybe able to implement this approach (after making some assumptions). Alternatively, you could calculate an effect for all of these means across time and include them in a single analysis and account for the covariance induced by this (See Noble et al. 2017). Lisa Schwanz and I just implemented something similar in our meta-analysis on incubation temperature and I am happy to discuss this with you
Mar ın-Mart ınez F, S anchez-Meca J (1999) Averaging dependent
effect sizes in meta-analysis: a cautionary note about procedures.
The Spanish Journal of Psychology, 2, 32–38.
D.W.A Noble, V. Stenhouse, L.E. Schwanz. (2017) Developmental temperatures and phenotypic plasticity in reptiles: A systematic review and meta-analysis. Biological Reviews
I realise these are very specific questions, but if anyone has advice on any of them (even if not all), it would be a big help. Thanks in advance!
Sorry, I don’t think I answered all your questions well, but hopefully some of these responses and papers provide some guidance. This stuff is really quite hard as it involves making so many decisions that depend on what was done in the paper etc.
If you have any questions happy to chat over a coffee!