Supplementary Materials: Equity Analysis at a Large Scale (CRDC School Arrests)

This document collects the supplementary materials for the white paper Equity Analysis at a Large Scale: Using Small Area Estimation to Get the Most from the CRDC School Arrest Data. It has four parts: exploratory analysis of the combined three-year CRDC data, the data-construction and sample-restriction details, model performance and diagnostics, and additional applied prediction-interval examples. The headline tables and figures, and the national descriptive totals by race and sex, are reported in the paper itself (white_paper.html); this companion holds the supporting detail. Like the paper, every section reads the published artifacts via crdc_path() and renders standalone.

Part 1 — Exploratory Data Analysis

This part explores the combined three-year CRDC data (2015-16, 2017-18, 2021-22). Counts and rates here are pooled across all three waves unless stated otherwise, and so differ from the single-wave 2021-22 figures in the paper.

Basic Data Structure

Data dimensions: 6762624 rows, 23 columns

'data.frame':   6762624 obs. of  23 variables:
 $ LEA_STATE            : chr  "AL" "AL" "AL" "AL" ...
 $ LEAID                : chr  "0100005" "0100005" "0100005" "0100005" ...
 $ SCHID                : int  870 870 870 870 870 870 870 870 870 870 ...
 $ SCH_NAME             : chr  "Albertville Middle School" "Albertville Middle School" "Albertville Middle School" "Albertville Middle School" ...
 $ COMBOKEY             : chr  "010000500870" "010000500870" "010000500870" "010000500870" ...
 $ JJ                   : chr  "No" "No" "No" "No" ...
 $ RACE                 : chr  "AM" "AM" "AS" "AS" ...
 $ SEX                  : chr  "F" "M" "F" "M" ...
 $ ARRESTS              : num  0 0 0 0 0 0 0 0 0 0 ...
 $ REFERRALS            : num  0 0 0 0 0 0 0 0 0 0 ...
 $ total_arrests        : num  0 0 0 0 0 0 0 0 0 0 ...
 $ total_referrals      : num  0 0 0 0 0 0 0 0 0 0 ...
 $ stu_enroll           : num  1 3 2 1 19 17 247 238 1 0 ...
 $ arrest_rate          : num  0 0 0 0 0 0 0 0 0 0 ...
 $ referral_rate        : num  0 0 0 0 0 0 0 0 0 0 ...
 $ YEAR                 : chr  "21-22" "21-22" "21-22" "21-22" ...
 $ total_enroll         : num  901 901 901 901 901 901 901 901 901 901 ...
 $ highest_grade_offered: num  8 8 8 8 8 8 8 8 8 8 ...
 $ lowest_grade_offered : num  7 7 7 7 7 7 7 7 7 7 ...
 $ latitude             : num  34.3 34.3 34.3 34.3 34.3 ...
 $ longitude            : num  -86.2 -86.2 -86.2 -86.2 -86.2 ...
 $ enrollment           : num  920 920 920 920 920 920 920 920 920 920 ...
 $ LEA_NAME             : chr  "Albertville City" "Albertville City" "Albertville City" "Albertville City" ...

Summary Statistics by Year

Summary Plot of Trends

State Variation Summing over 3 years

Race and Sex Distribution

These counts and rates are pooled across all three CRDC waves (2015-16, 2017-18, 2021-22) and are shown here as number-needed-to-harm (NNH, students per arrested student). They therefore differ from the single-wave 2021-22 totals and per-1,000 rates reported in white_paper.qmd; they are not the same figures and should not be read as such.

Trends by group

State Trends

LEA Comparison

If we aggregate to the LEA level, what patterns do we see?

         1          2          3 
0.09864719 0.04783550 0.85351732 

[1] 18480

[1] 0.01894883 0.01585647 0.96519470

Rankings over time

Rankings of arrest rates

Part 2 — Data Construction & Sample Restrictions

This part documents how the analytic sample is built from the raw CRDC and the magnitude of each restriction the paper describes. The national descriptive totals and rates by race and sex are reported in the paper (white_paper.html).

Sample Continuity Analysis for CRDC Waves

Look at how many LEAs are in multiple years of data and how many LEAs drop out of the data when we add successive waves of the data.

**** Distinct Matches ****
**** Match Summary ****
X in Y
Of the 17604 X values, 16203 (92%) were matched.
********************************************
Y in X
Of the 17704 Y values, 16203 (92%) were matched.
******************************************

**** Distinct Matches ****
**** Match Summary ****
X in Y
Of the 17337 X values, 15877 (92%) were matched.
********************************************
Y in X
Of the 16203 Y values, 15877 (98%) were matched.
******************************************

District Concentration Analysis

For this analysis we look at the most recent CRDC wave.

District Size Categories

Expected vs Observed Arrests

Concentration Visualization

This visualization shows the relative concentration of arrests compared in LEAs compared to the concentration of population in LEAs.

Enrollment by Arrest Status

Sample Restriction Impact

Here we tabulate the figures about sample restrictions in the paper.

Section 504 Data

Missing Data Patterns

Law enforcement referral missingness data.

  FALSE    TRUE 
5303388  185172 


     FALSE       TRUE 
0.96626219 0.03373781 

Section 504 Enrollment

Grade Level Restrictions

This reports the impact of restricting the analysis to schools that offer a grade 7 or above.

CCD Matching

By matching to CCD data we further restrict the CRDC universe. This analysis shows the impact of that.

**** Distinct Matches ****
**** Match Summary ****
X in Y
Of the 98010 X values, 95254 (97%) were matched.
********************************************
Y in X
Of the 102130 Y values, 95254 (93%) were matched.
******************************************

            used   (Mb) gc trigger   (Mb)  max used   (Mb)
Ncells   4395634  234.8   15070707  804.9  29434972 1572.0
Vcells 219785437 1676.9  441836648 3371.0 454062917 3464.3

Enrollment Thresholds

We restrict the sample by enrollment size, and this reports the impact of that restriction.

Censored or truncated enrollment data

Here we report the impact of suppressed records on enrollment.

[1] 0.01887051

Part 3 — Model Performance & Diagnostics

Model Predictions and Performance

Now we calculate the performance results comparing the models to non-model methods for computing arrest rates.

Observed Data Preparation

Performance Metrics

This code calculates our performance metrics on the arrest rate scale, translating it from the arrest scale.

Full-sample model performance

The full-sample summary of coverage, precision, “% equal or better” intervals, and median % narrowed across all ten models is Table 3 in white_paper.qmd, computed there from this same observed-vs-modeled join. It is not duplicated here. The diagnostic cross-tabulations that are unique to this document follow.

Model Performance Diagnostics

Comparing models to non-modeled intervals

                 improved_precision
narrower_interval       0       1
                0   12475    4738
                1     936 1048881

                 improved_precision
narrower_interval       0       1
                0   12661    4552
                1 1049737      80

Looking at constant models

Some fitted values are constants with no variation. We want to look at these cases separately.

When we exclude constant prediction results and see how models perform to non-modeled intervals.

        higher_precsion
narrower      0      1
       0  12475   4738
       1    936 697450

Non-Constant Models

Now we look in more depth at results from observations without constant predicted values.

Non-zero-arrest model performance

Model performance restricted to district-student groups with at least one observed arrest is Table 4 in white_paper.qmd. It is not duplicated here.

Subgroup Analysis

Define Subgroups

Create largest 100 districts

The “most total arrests” and “largest-enrollment with zero arrests” district subsets are analysed canonically in white_paper.qmd (Tables 5 and 6), so only the largest-enrollment-by-any-arrest subset (big100) is built here.

Large Districts Performance

Look at how models perform on the 100 largest districts by student enrollment

High-arrest and zero-arrest district performance

Model performance for the 100 districts with the most total arrests is Table 5, and the expected-vs-observed arrests for the 100 largest-enrollment districts that reported zero arrests is Table 6 — both computed reproducibly in white_paper.qmd. They are not duplicated here.

Model Computation Statistics

Here we compute the model statistics.

Unified Models

Stratified Models

# A tibble: 0 × 5
# ℹ 5 variables: model_label <chr>, runtime <dbl>, parameters <dbl>, data_rows <int>, ndraws <dbl>

HMC Diagnostics

### Unified (m1) 

Divergences:

Tree depth:

Energy:

### Unified (m2) 

Divergences:

Tree depth:

Energy:

### Unified (m3) 

Divergences:

Tree depth:

Energy:

### Unified (m4) 

Divergences:

Tree depth:

Energy:

### Unified (m5) 

Divergences:

Tree depth:

Energy:

State-Level Analysis

State Predictions

            used   (Mb) gc trigger   (Mb)  max used   (Mb)
Ncells   4662848  249.1   15070707  804.9  29434972 1572.0
Vcells 142568456 1087.8  425250724 3244.5 531563027 4055.6

State Observed Data

State Performance

Part 4 — Applied Examples

These are additional applied prediction-interval examples beyond the case studies in the paper: a cross-district comparison, a single-district temporal comparison, a small-groups demographic disparity, and a national overview.

Cross-district, temporal, and small-group comparisons

Compare Baltimore City to Baltimore County Baltimore City to itself last year Broward White Female to Hispanic female

Here we apply the same approach as above but to compare arrest rates from two different school districts - Baltimore City Public Schools (orange) and Baltimore County Public Schools (blue). Here we see a curious pattern - if we look at our one year models, the frequentist interval and Bayesian intervals strongly agree that Baltimore City has a higher arrest rate than Baltimore County, with the most likely arrest rate in Baltimore County being 0. Covariates make little difference in these estimates.

The three year models have a very different picture - the Bayesian models see Baltimore County as having not just a higher arrest rate, but a much higher arrest rate than Baltimore City. The Baltimore City estimates are relatively unchanged, though covariates seem to shift the distribution slightly closer to 0 (unified and stratified models 4).

When we turn to our direct estimate of the differences, we see this pattern play out - the one year models are 100% certain in all specifications that Baltimore City has a higher arrest rate than Baltimore County, and that difference is most likely narrowly between just greater than 0 and 1. In contract, the 3 year models are equally sure that Baltimore City has no chance of a higher arrest rate than Baltimore County, and in fact, that it is most likely that Baltimore County’s arrest rate is 1-2 arrests per 1,000 greater than Baltimore City. When we pool all of our models together, we are left with a 50% chance one of the Baltimore’s has a greater arrest rate than the other (inset, bottom left).

To better understand what is going on we can use our models to make temporal comparisons in Baltimore County - what does the trend look like.

We can

State-level overview

            used   (Mb) gc trigger   (Mb)  max used   (Mb)
Ncells   4793619  256.1   15070707  804.9  29434972 1572.0
Vcells 144981148 1106.2  425250724 3244.5 531563027 4055.6

The recreation of the Table 1 state demographic-group differences is now part of the white paper’s “Comparing state-level demographic group differences” (wp_fig_state_differences); only the all-states overview above is kept here.