MRI Quality Control

Domain overview

Please scroll horizontally to view the number of variables and events of administration for the displayed tables.

Researchers analyzing neuroimaging data should ensure responsible and thoughtful use of demographic and participant-related variables. Analyses comparing groups based solely on demographic categories may lead to misinterpretations about neurobiological differences and unintended implications regarding participants’ abilities or outcomes. To avoid potential misinterpretation or misuse of research findings, data users must carefully consider all variables included in their analyses, and ensure results are presented with appropriate scientific context.

Any group comparisons should incorporate comprehensive, contextual factors beyond single demographic categories, such as environmental or behavioral variables, to enhance scientific rigor and interpretation accuracy. Researchers are encouraged to proactively mitigate the risk of results being misconstrued or used in ways that could negatively affect individuals or groups participating in or impacted by their research.

Missingness issues

Participants in the ABCD Study may not have usable data for a variety of reasons. They may have withdrawn from the study, or they may have missed assessments or sessions. In other cases, data may have been collected but it is inappropriate for inclusion in an analysis. For example, neuroimaging data may be inappropriate for analysis due to excessive head motion (see the discussion of head motion in Resting-State MRI Data Documentation).

Factors associated with missing or unusable data can correlate with participant characteristics or environmental variables. Researchers should carefully assess patterns of missing data to understand how these patterns may influence analytical outcomes and interpretations. Additionally, researchers are encouraged to implement appropriate statistical methods, such as imputation or population weighting (Gard et al. 2023), to reduce biases resulting from missing data, thus improving the representativeness and generalizability of their results.

Recommended imaging inclusion

mr_y_qc__incl

Users of the imaging data are advised to choose inclusion criteria that are appropriate for their analyses. We provide a set of recommended inclusion criteria for each modality that take into account factors such as imaging QC, task performance, etc., and for convenience, we provide an instrument mr_y_qc__incl with modality-specific imaging inclusion flags based on those criteria. Each modality (T1w, T2w, DTI/RSI, rsfMRI, SST, nBack, and MID tfMRI) is scored as 1=include, 0=exclude.

MRI clinical findings

mr_y_qc__clfind

T1w and T2w-weighted images, if available, were screened for incidental findings by a Board Certified Neuroradiologist. Any findings requiring clinical investigation were relayed to appropriate site personnel via the ABCD Coordinating Center (CC).

The most important measure is the Report Score (mrif_score):

  • 0 = Image artifacts prevent radiology read
  • 1 = No abnormal findings
  • 2 = Normal anatomical variant of no clinical significance
  • 3 = Consider clinical referral
  • 4 = Consider immediate clinical referral

Although not included in the recommended inclusion criteria, users may, depending on their research question and analytical design, opt to exclude participants with mrif_score != 3 OR mrif_score !=4.

MRI raw QC

mr_y_qc__raw

  • Protocol compliance checking
    • Performed by on-site FIONA workstations to provide feedback to scan operators
    • Out-of-compliance series reviewed by DAIRC staff
    • Criteria included whether key imaging parameters matched expected values for a given scanner, such as voxel size or repetition time
    • Presence or absence of B0 distortion field map series was checked for diffusion MRI (dMRI) and functional MRI (fMRI) series
    • Each imaging series checked for completeness (i.e., no missing files)
  • Automated quality control metrics
    • Structural MRI (sMRI): mean and standard deviation of brain values and spatial SNR
    • dMRI: mean motion (average framewise displacement), and the number of slices and frames affected by slice dropout caused by abrupt head motion
    • fMRI: mean motion (average framewise displacement), the number of seconds with framewise displacements less than 0.2, 0.3, or 0.4 mm (Power et al. 2012), temporal SNR (tSNR) (Triantafyllou et al. 2005)
  • Manual review of data quality
    • Reviewers assigned binary QC score
      • 0 = reject
        • Most severe artifacts or irregularities
        • Rejected series excluded from subsequent processing and analysis
      • 1 = accept
    • Types of images reviewed
      • T1w, T2w, dMRI, dMRI field map, fMRI, and fMRI field map
      • Raw and some derived images were reviewed
      • dMRI derived images included average b=0 image, FA, MD, tensor fit residual error, and direction encoded color image
      • fMRI derived images included the average across time and the temporal standard deviation
    • Inspected for signs of artifacts and poor image quality
      • Presence of wrap-around field of view artifacts
      • Brain cut-off due to participant movement outside prescribed slices
      • Magnetic susceptibility artifacts due to dental implants
      • T1w and T2w motion artifact (e.g., blurring and ghosting)
    • Artifact type and severity for series with QC = 0 is indicated in reviewer notes made with internally standard abbreviations

Manual post processing QC

FreeSurfer QC

mr_y_qc__post__man__fsurf

  • Manual review of FreeSurfer cortical surface reconstruction
  • Reviewers assigned binary QC score
    • 0 = reject
    • Most severe artifacts or irregularities
    • Results still included in shared tabulated data
    • Recommended exclusion from group analyses involving cortical surface ROIs
    • 1 = accept
  • Reviewers gauged the severity of five types of artifact or processing problem
    • QU_motion: motion artifact in T1w image
    • QU_inhomo: intensity inhomogeneity
    • QU_wmunder: white matter underestimation
    • QU_pialover: pial overestimation
    • QU_artifact: magnetic susceptibility artifact
  • Numeric values assigned on a scale of 0-3
    • Absent, mild, moderate, or severe, respectively
    • Overall QC score of 0 assigned if severity score of 3 for any artifact type

Note: Imaging-derived results are included in shared tabulated data regardless of post-processing QC. QC variables (derived from procedures described above) are included in shared tabulated data. The overall, binary QC score described above indicates whether inclusion or exclusion is recommended based on this criterion alone (see above Recommended Imaging Inclusion). FreeSurfer QC was performed for ~7% of participant-events with T1w imaging data.

sMRI T2w post processing QC

mr_y_qc__post__man__t2

  • Manual review of DTI reconstruction
    • Reviewers assigned binary QC score
      • 0 = reject
        • Most severe artifacts or irregularities
        • Results still included in shared tabulated data
        • Recommended exclusion from group analyses involving cortical, subcortical, and tract-based ROIs
      • 1 = accept
    • Reviewers gauged the severity of four types of artifact or processing problem
      • QU_motion: motion
      • QU_inhomo: intensity inhomogeneity
      • QU_artifact: magnetic susceptibility artifact
      • QU_regT1: registration with T1w image
    • Numeric values assigned on a scale of 0-3
      • Absent, mild, moderate, or severe, respectively
      • Overall QC score of 0 assigned if severity score of 3 for any artifact type

Note: The T2w Post Processing QC was performed for ~7% of participant-events with T2w imaging data.

dMRI post processing QC

mr_y_qc__post__man__dmri

  • Manual review of processed dMRI data
    • Reviewers assigned binary QC score
      • 0 = reject
        • Most severe artifacts or irregularities
        • Results still included in shared tabulated data
        • Recommended exclusion from group analyses involving cortical, subcortical, and tract-based ROIs
      • 1 = accept
    • Reviewers gauged the severity of five types of artifact or processing problem
      • QU_B0warp: residual (uncorrected) B0 warping
      • QU_ImagQual: image quality based on motion-related artifacts and magnetic susceptibility-induced drop-out artifact or other image quality issue
      • QU_FOV: full head coverage
      • QU_regT1: registration with T1w image
      • QU_fseg: accuracy of fiber tract segmentation
    • Numeric values assigned on a scale of 0-3
      • Absent, mild, moderate, or severe, respectively
      • Overall QC score of 0 assigned if severity score of 3 for any artifact type

Note: The dMRI Post Processing QC was performed for ~13% of participant-events with dMRI data.

fMRI post processing QC

mr_y_qc__post__man__fmri

  • Manual review of processed fMRI data
    • Reviewers assigned binary QC score
      • 0 = reject
        • Most severe artifacts or irregularities
        • Results still included in shared tabulated data
        • Recommended exclusion from group analyses involving cortical, subcortical, and tract-based ROIs
      • 1 = accept
    • Reviewers gauged the severity of five types of artifact or processing problem
      • QU_B0warp: residual (uncorected) B0 warping
      • QU_ImagQual: image quality based primarily on magnetic susceptibility-induced drop-out or other image quality issues artifact
      • QU_FOV: full head coverage
      • QU_regT1: registration with T1w image
    • Numeric values assigned on a scale of 0-3
      • Absent, mild, moderate, or severe, respectively
      • Overall QC score of 0 assigned if severity score of 3 for any artifact type

Note: The fMRI Post Processing QC was performed for ~18% of participant-events with fMRI data.

Automated post processing QC

mr_y_qc__post__aut

Automated QC measures were defined and calculated based on processed imaging data.

  • FreeSurfer
    • Number of topological defects
      • Calculated from Euler number
  • dMRI
    • Field of view (FOV) brain cutoff
      • Quantified by % intersection of brain mask with frame borders
    • Registration to T1w
      • Window-based estimation of geometric registration discrepancy with respect to the T1 scan, decomposed into rigid and warp components for calculation of registration error
  • fMRI
    • Field of view (FOV) brain cutoff
      • Quantified by % intersection of brain mask with frame borders
    • Registration to T1w
      • Window-based estimation of geometric registration discrepancy with respect to the T1 scan, decomposed into rigid and warp components for calculation of registration error
  • sMRI T2w
    • Registration to T1w
      • Window-based estimation of geometric registration discrepancy with respect to the T1 scan, decomposed into rigid and warp components for calculation of registration error

Automated selection for manual post-processing QC

Our manual quality control process involves manual examination of brain images from a subset of participant-events for each modality, with ratings for each dataset according to pre-defined quality criteria such as brain cutoff, residual distortion, or registration to T1. Such measures are described above (see FreeSurfer QC, sMRI T2w Post Processing QC, dMRI Post Processing QC, and fMRI Post Processing QC). Because an exhaustive manual review of every scan is not practical, we deploy statistical learning/AI guided sub-sampling methods where we generate automated metrics for an array of quality control issues (see Automated Post Processing QC) and sub-select participant-events to be sent for a given modality to manual review based on their quality measure scores. The automatic selection priority scores generally indicate how likely a dataset of a given type is to contain identifiable data quality issues such as a residual distortion or brain cutoff.

Overview of selection strategies

Because there were no significant changes in the MRI processing pipeline between ABCD Release 4.0 and Release 6.0, it was not necessary to reprocess most of the previously collected imaging data, except in rare cases, such as due to recovery of missing data or revision of manual raw QC. Because of this, manual post-processing QC scores from Release 4.0 were mostly carried forward into Release 6.0. Changes to the processing were made between Release 3.0 and 4.0, so the old QC scores were not carried forward for Release 4.0. Instead, participant-events were selected for manual post-processig QC based on failed post-processing QC from Release 3.0, random selection of participant-events, and statistical outliers (see below for details). The manual scoring and automated QC metrics for those three sets were then used for classifier guided selection (see below for details), and those additional visits were reviewed as well. For more recently collected imaging data not included in Release 4.0, additional subsets of participant-events were selected for manual post-processing QC using classifier guided selection and Mahalanobis distance outliers (see below for details). In addition, we undertook a limited retrospective review of past QC failures to help ensure that usable data were not inappropriately or mistakenly marked for exclusion (see below for details).

Selection strategies used for Release 4.0

Failed post-processing QC from Release 3.0

All participant-events that failed QC for a given modality in Release 3.0 were manually reviewed again after Release 4.0 processing, with the expectation that they would be likely candidates for failure, unless perhaps recovered due to improvements in the Release 4.0 processing pipeline. These made up about 25-50% of manually reviewed participant-events for Release 4.0.

Random selection

Roughly 5% of the manually reviewed participant-events for Release 4.0 were chosen at random to avoid possible biases and insensitivities of the automated measures that guided the initial sub-sampling selection process.

Statistical outliers

ROI summary vectors for all individual participant-events were used to deduce a statistical ensemble profile. Participant-events with the greatest “distance” to the ensemble were tagged as outliers and selected for manual review. The formal framework we used for defining the ensemble statistics and the “distance” measure between a scan and the ensemble is based on the Mahalanobis distance (see https://en.wikipedia.org/wiki/Mahalanobis_distance). Mahalanobis distance is a scalar measure of the distance between a point P, defined in a multi-dimensional vector space, and a distribution D. In our case, participant-events were sorted based on their Mahalanobis distances, and our manual review goal targeted the top 5% of that list. These made up ~20-25% of manually reviewed participant-events.

Classifier-guided selection

Using the manual scoring of the above sets, namely the random samples, Release 3.0 QC failures, and Mahalanobis distance outliers, we collected a subset of manually labeled scan data. For each participant-event we generated a set of automated measures described in Automated Post Processing QC. Using binarized (pass/fail) manual quality control (QC) labels associated with these measures, we constructed a Bayesian classifier (see https://en.wikipedia.org/wiki/Bayes_classifier) that calculates the probability of QC failure associated with different types of QC issues (e.g., bad registration or brain FOV cutoff) for any given dataset. Each combination of automated QC metric and manual quality (QU) score was tested. Combinations of manual QC scores and automated QC metrics with low correlation (r < 0.5) or low classification accuracy (ROC AUC < 0.6) were excluded for use in selection. Selected manual QU scores were binarized such that “QC fail” was defined as QU score > 2, and “QC pass” was defined as QU score <= 2. Relative QC failure probabilities given a particular automated QC metric value were calculated for each combination of manual scores and automated metics. Participant-events were sorted based on their QC failure probabilities, and roughly the top 3-5% were selected for manual review for each modality, making up ~30-50% of manually reviewed participant-events in Release 4.0.

Selection strategies used for Release 5.0 and 6.0

Multivariate classifier-guided selection

For imaging data collected more recently than that included in Release 4.0, the classifier-guided selection approach described above was extended to use a multivariate logistic regression that uses multiple auto QC metrics as well as the Mahalanobis distance metrics derived from ROI summary measures. The Mahalanobis distance value was calculated separately for each modality represented in the tabulated ROI summary measures (i.e., FreeSurfer, T1w, T2w, dMRI-DTI, dMRI-RSI, MID, nBack, SST, rsfMRI). These outlier distance metrics were included along with the other automated QC metrics used for prediction.

For each manual post-processing QU score, automated metrics (including automated QC metrics and Mahalanobis distances) with low correlation (r < 0.5) or low classification accuracy (ROC AUC < 0.6) to the manual QU score were excluded for use in selection. The remaining set of automated metrics were transformed with singular value decomposition to remove redundant information. The resulting reduced matrix was entered into a logistic regression with the binarized QU score. The values predicted by the logistic regression were then passed to the Bayesian classifier described above to calculate the probability of QC failure. Additional methodological details and an analysis of selection performance will be presented in a forthcoming paper (Hagler et al., in preparation).

Retrospective QC review

A retrospective review of prior, post-processing QC failures was undertaken to ensure that usable data were not inappropriately or mistakenly marked for exclusion. This was done on a limited basis for particular causes of QC failure when we found reason to doubt some of the reviews of a given type; e.g., white matter underestimation, T1 or T2 intensity inhomogeneity. In this re-review, scores were changed in some but not most cases selected for re-review.Methods

Image processing and analysis methods corresponding to ABCD Release 2.0.1 are described in Hagler et al. (2019).

Additional references:

  • Cardenas-Iniguez et al. (2024)
  • Laird (2021)
  • Power et al. (2012)
  • Saragosa-Harris et al. (2022)
  • Simmons et al. (2021)

References

Cardenas-Iniguez, Carlos, Jared N. Schachner, Ka I. Ip, Kathryn E. Schertz, Marybel R. Gonzalez, Shermaine Abad, and Megan M. Herting. 2024. Developmental Cognitive Neuroscience 65 (February): 101338. doi:10.1016/j.dcn.2023.101338.
Gard, Arianna M., Luke W. Hyde, Steven G. Heeringa, Brady T. West, and Colter Mitchell. 2023. Developmental Cognitive Neuroscience 59 (February): 101196. doi:10.1016/j.dcn.2023.101196.
Hagler, Donald J., SeanN. Hatton, M. Daniela Cornejo, Carolina Makowski, Damien A. Fair, Anthony Steven Dick, Matthew T. Sutherland, et al. 2019. NeuroImage 202 (November): 116091. doi:10.1016/j.neuroimage.2019.116091.
Laird, Angela R. 2021. NeuroImage 244 (December): 118579. doi:10.1016/j.neuroimage.2021.118579.
Power, Jonathan D., Kelly A. Barnes, Abraham Z. Snyder, Bradley L. Schlaggar, and Steven E. Petersen. 2012. NeuroImage 59 (3): 2142–54. doi:10.1016/j.neuroimage.2011.10.018.
Saragosa-Harris, Natalie M., Natasha Chaku, Niamh MacSweeney, Victoria Guazzelli Williamson, Maximilian Scheuplein, Brandee Feola, Carlos Cardenas-Iniguez, et al. 2022. Developmental Cognitive Neuroscience 55 (June): 101115. doi:10.1016/j.dcn.2022.101115.
Simmons, Cortney, May I. Conley, Dylan G. Gee, Arielle Baskin-Sommers, Deanna M. Barch, Elizabeth A. Hoffman, Rebekah S. Huber, et al. 2021. Psychological Science 32 (6): 866–70. doi:10.1177/09567976211003564.
Triantafyllou, C., R. D. Hoge, G. Krueger, C. J. Wiggins, A. Potthast, G. C. Wiggins, and L. L. Wald. 2005. NeuroImage 26 (1): 243–50. doi:10.1016/j.neuroimage.2005.01.007.