Evaluation metrics for spaced-repetition schedulers in R: the sample-weighted binned RMSE (RMSE(bins)) used to rank algorithms in the open spaced repetition benchmark, together with log loss, the area under the ROC curve, and calibration curves. The metrics take plain vectors of predicted recall probabilities and observed outcomes, so they can benchmark any scheduler’s predictions.
Installation:
# install.packages("remotes")remotes::install_github("chrislongros/srsbench")
Use:
library(srsbench)p <- c(0.95, 0.86, 0.72, 0.60, 0.91)y <- c(1, 1, 0, 1, 1)elapsed_days <- c(1, 6, 20, 4, 15)i <- c(2, 3, 4, 2, 3)lapse <- c(0, 0, 1, 0, 0)rmse_bins(p, y, elapsed_days, i, lapse) # calibration, lower is betterlog_loss(p, y) # cross-entropy, lower is bettersrs_auc(p, y) # discrimination, higher is bettercalibration_bins(p, y) # predicted vs observed by bin
More info here: https://github.com/chrislongros/srsbench
Available as an AUR package too: https://aur.archlinux.org/packages/r-srsbench
/root 