Schult, J., & Sparfeldt, J. R. (2016). Reliability and validity of PIRLS and TIMSS: Does the response format matter? European Journal of Psychological Assessment. doi:10.1027/1015-5759/a000338
Johannes says: "The big one - lots of manual syntax, mainly IRT scaling and plots"
Version: V1
Data analysis finished: 2014-06-26
Article published: 2016-08-03
Replication files published: 2016-09-18
Software used: Stata 13.1, R 3.1.0, IBM SPSS 21, MS Excel 2007
File space: Below you find the do-files used for the analyses. It was rather complicated and - as often with such projects - there was never much time (or pressure) to tidy things up. I mean, I did streamline the analyses, but still: lots of files and due to the composition of the booklets there are thousands of lines of code (double-checked and triple-checked). The analyses were run on a separate file system named W:/ with two directories for the two Studies, IGLU2006/ and TIMSS2007/ (IGLU is the German abbreviation for PIRLS). Due to the huge number of files and dependent syntax I did not attach V1 to the file names (indicating the first, i.e., original/submitted/published version).
Data sources: The item parameter data were downloaded from timssandpirls.bc.edu/timss2011/downloads/T11_ItemParameters.zip and timssandpirls.bc.edu/timss2011/downloads/T11_ItemParameters.zip. (NB: Those are the 2011 files which also include the estimates from the previous round.). The raw pupil data for the validity analyses were kindly provided by and are available from the Research Data Centre (FDZ) at the IQB Berlin.
Data management: I did the unthinkable and cleaned the Excel files by hand. Sorry for that. The cleaned files are the two *.xlsx files below. The raw pupil data was provided in SPSS format. The two scientific use files were converted into Stata files using IGLU2006/crstatafiles.sps. Then came a barrage of additional data management steps to get the data into the proper format. The two Studies used the same approach using the files IGLU2006/cr1itemiglu2006.do, IGLU2006/cr2iglu2006.do, IGLU2006/cr3dataX.do, IGLU2006/cr4validity.R, and IGLU2006/cr5iglu2006X.do for PIRLS, and then TIMSS2007/cr1itemtimss2007.do, TIMSS2007/cr2timss2007.do, TIMSS2007/cr3dataX.do, TIMSS2007/cr4validity.R, and TIMSS2007/cr5timss2007X.do for TIMSS.
Auxiliary analyses: Information in the methods section was calculated using anitemhalf1half2.do and anitemnonresponse.do.
Reliability analyses: First, I created two slightly modified versions of PlotIRT (Hill & Langer, 2005) to obtain suitable IRT functions in R: PlotIRTA.R and PlotIRTI.R. Both are called upon in the following R analyses. The analyses sometime include both graphic commands and analysis commands. Sorry for that. The files are gr1reliability.R, gran2reliability.R, and gran3reliability.R. I looked long and hard for a simpler way to obtain the IRT results for the combination of different item/scoring types, but none of the available packages seemed to provide the features necessary for my analyses.
Validity analyses: Validity results were obtained in Stata with the two an1.do files, which repeatedly call on the ancompcorr.do files.
IGLU2006/P11_ItemParametersA.xlsx
IGLU2006/crstatafiles.sps
IGLU2006/cr1itemiglu2006.do
IGLU2006/cr2iglu2006.do
IGLU2006/cr3dataX.do
IGLU2006/cr4validity.R
IGLU2006/cr5iglu2006X.do
IGLU2006/an1.do
IGLU2006/ancompcorr.do
TIMSS2007/T11_G4_ItemParametersA.xlsx
TIMSS2007/cr1itemtimss2007.do
TIMSS2007/cr2timss2007.do
TIMSS2007/cr3dataX.do
TIMSS2007/cr4validity.R
TIMSS2007/cr5timss2007X.do
TIMSS2007/an1.do
TIMSS2007/ancompcorr.do
anitemhalf1half2.do
anitemnonresponse.do
PlotIRTA.R
PlotIRTI.R
gr1reliability.R
gran2reliability.R
gran3reliability.R
V1: 2016-09-18
Please visit http://www.jutze.com/research for more materials on publications and possibly newer versions.
Written by Johannes Schult (jutze@jutze.com)
Last updated: 2016-09-18