Introduction: Knowledge of how abstracts may be se-lected for medical conferences in an efficient and reliable manner is sparse. To improve abstract selection, the Danish Orthopaedic Society implemented the International Society of the Knee (ISK) quality-of-reporting system and visual analogue scale (VAS) scoring for abstract evaluation at its 2014 Annual Congress. We sought to find out if a simple VAS score was more reliable than a multiple-question system for assessment of over-all abstract quality.
Methods: A total of 214 abstracts were submitted for review. All abstracts were reviewed by 3 reviewers using a VAS score and the ISK score. Of the 214, 71 abstracts were reviewed again 6 months later to estimate intra-rater agreement.
Results: The VAS and the ISK score were poorly correlated (r = 0.64), and the ISK score demonstrated a better intra- and interrater agreement (p < 0.001). The VAS scores of all abstracts were more widely distributed than the ISK scores, which clustered around values in the 50-70 range. Chronbach's alpha for the ISK score was 0.66 (95% confidence interval: 0.62-0.68).
Conclusions: The VAS score has a poorer intra- and interrater agreement than the ISK score, and the two scores do not correlate well. VAS scores were more widely distributed, which is beneficial when selecting a scientific programme, but the score is unreliable. We continue to use the ISK score, although its reliability may still be improved.
Funding: none.
Trial registration: not relevant.
Articles published in the Danish Medical Journal are “open access”. This means that the articles are distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits any non-commercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.