Out of the Garden and Into the Multiverse: Open Science Practices, Multidimensional Degrees of Freedom, and the Speech Sciences

About the Presentation:

The early 2010’s saw the reproducibility crisis take hold of the psychological sciences. As a consequence, a push for increased transparency and reproducible methodology (e.g. open access publishing, preregistration, sharing study materials, data, and analyses, performing study replications, declaring conflicts of interest, ect.) to help mitigate the effects of questionable research practices (e.g., p-hacking, HARKing, researcher degrees of freedom) has followed. The resulting methodological framework and associated techniques, now referred to as open science, have reshaped research methods in psychology and have slowly but surely made their way into adjacent fields. In this talk I will consider challenges related to open science practices and researcher degrees of freedom in the field of linguistics.

Language-related data are complicated and typically subject to myriad operationalizations. Analytic flexibility in data analysis—commonly referred to as researcher degrees of freedom (Gelman & Loken, 2014; Simmons, Nelson, & Simonsohn, 2011)—can result in substantially different conclusions based on the same data set, a situation some refer to as the inference crisis (see Rotello, Heit, & Dubé, 2015; Starns et al., 2019). The so-called “garden of forking paths” associated with researcher degrees of freedom directly affects our ability to replicate findings and accumulate knowledge.

Moreover, the scientific study of language spans numerous disciplines and encompasses social, cultural, cognitive, technological, and biomedical studies of human nature and behavior. As linguistics becomes increasingly empirical and quantitative, unique challenges arise regarding the methodological practices aimed at improving reproducibility/replicability. One of the pro- posed solutions to the replicability crisis has been the implementation of open science practices. As a consequence, today’s speech researcher must be up to date on the current protocols of open science in order to produce research projects that meet these new methodological stan- dards. However, an important consideration often overlooked in the wake of the open science movement deals with how current linguists train the next generation of researchers. Few, if any, researchers have had explicit instruction on the practices of open science as part of their professional training.

In this talk I will present research on open science and transparency practices in the field of linguistics. I will show that, by and large, linguistics does not engage in open science and transparency practices, though there is reason to believe this is changing for the better. Furthermore, I will present data from the Many Speech Analyses Project showing that the problem of researcher degrees of freedom is likely much more serious than our current under- standing suggests. To conclude, I will discuss what we learn from methodological reform, many analyst-type studies, and big team science, as well as what current trends mean for linguistics, generally—and for some sub-fields of linguistics, specifically—. I will present solutions aimed at reducing researcher degrees of freedom and fostering open science practices in linguistics.

References

• Gelman, A., & Loken, E. (2014). The statistical crisis in science: Data-dependent analysis–a “garden of forking paths”–explains why many statistically significant comparisons don’t hold up. American Scientist, 102(6), 460–466.
• Rotello, C. M., Heit, E., & Dubé, C. (2015). When more data steer us wrong: Replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin & Review, 22(4), 944–954.
• Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undis-closed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.
• Starns, J. J., Cataldo, A. M., Rotello, C. M., Annis, J., Aschenbrenner, A., Bröder, A., et al.others. (2019). Assessing theoretical conclusions with blinded inference to investigate a potential inference crisis. Advances in Methods and Practices in Psychological Science, 2(4), 335–349.

About the Presenter:

Joseph V. Casillas is an Associate Professor of Spanish Linguistics in the Department of Spanish and Portuguese Department at Rutgers University, New Brunswick. His main areas of research are phonetics, laboratory phonology, and second language acquisition, with a specific interest in the role of native phonetic experience and its influence on L2 speech production, perception and lexical processing. Additionally, Joseph is dedicated to improving methodological practices in the language sciences through pedagogy, focusing specifically on open science and quantitative analysis.

Please note: To register, please visit https://open-linguistics.eventbrite.ca