Projects / Comment Toxicity

Comment Toxicity

Four-class toxicity classifier with 66+ engineered features, out-of-fold target encoding, blend optimization, and tuned per-class thresholds.

Source on GitHubLive project siteImprovement journeyOpen in Colab

Demo

A recording of the live project site: the data exploration, the pipeline, the model comparison and the feature importance.

Overview

Comment Toxicity classifies comments into four levels, from non-toxic to extremely toxic. It turns a Kaggle notebook into a reproducible pipeline: feature engineering, out-of-fold target encoding, a class-weighted LightGBM model, a Nelder-Mead blend and per-class decision thresholds, on about 198,000 labelled comments.

The data is heavily imbalanced (57.7% of comments are non-toxic) and demographic fields are missing for roughly 73% of comments. So much of the work is in features that capture thread context, and in tuning decisions for the rare classes.

Out-of-fold results are 0.912 accuracy and 0.817 macro F1. The demo below is a recording of the project's live site, with its charts for the data, the model comparison and the feature importance.

What it does

Screenshots

Charts showing class imbalance and demographic presence by toxicity label
The data: strong class imbalance, and demographic fields filled far more often on toxic comments.
The four stages of the pipeline
The four pipeline stages: features, target encoding, LightGBM and threshold tuning.
Bar chart comparing accuracy, macro F1 and weighted F1 across four models
Model comparison from out-of-fold predictions: LightGBM well ahead of random forest and logistic regression.
Feature importance and the full classification report
Feature importance and the per-class report: post-level label rates matter most.

How it works

Interactive diagram · 9 steps
raw columns features encoded matrix probs OOF probs blended argmax About 198,000 labelled comments in four toxicity classes. Class 0 (non-toxic) is 57.7% of the data, and demographic fields are missing for roughly 73% of comments.Comments198K rows, 4 classes1 Text statistics, punctuation and style, datetime parts, engagement ratios and demographic flags, plus post-level features such as per-thread label rates, comment position and time between comments.ƒFeature engineering66+ features2 Smoothed mean encoding of the demographic columns, computed out-of-fold with a 5-fold stratified split so the encoding never sees its own row's label, which prevents leakage.ƒOOF target encoding5-fold, smoothed3 A scikit-learn ColumnTransformer with word TF-IDF (1-2 grams), character TF-IDF (3-5 grams), an SVD-reduced TF-IDF, one-hot encoding for categoricals and standard scaling for numeric columns.ƒTF-IDF pipelineword, char, SVD4 The primary model, trained with class weights (0:1, 1:4, 2:2, 3:8) to cope with the imbalance, and early stopping. It reaches 0.913 accuracy and 0.813 macro F1 out-of-fold.LightGBMclass-weighted5 Logistic regression and a subsampled random forest are trained for comparison. They score far lower (macro F1 0.494 and 0.587), which shows the gain comes from the features and the boosting.Baselineslogistic reg., forest6 Nelder-Mead searches for the best weights for blending the models' out-of-fold probabilities. Here it converged to 100% LightGBM.ƒBlend optimiserNelder-Mead weights7 Decision thresholds for each class are optimised with Nelder-Mead to maximise macro F1, which matters most for the rare classes. This lifted macro F1 from 0.813 to 0.817.Per-class thresholdstuned for macro F18 Final class predictions and a Kaggle-format submission file. Out-of-fold results: 0.912 accuracy, 0.817 macro F1 and 0.913 weighted F1.Predictionsmacro F1 0.8179
Pick a stepHover, focus or tap any shape to see what it does.
From raw comments to predictions. The dashed arrow carries the baselines' out-of-fold probabilities into the blend. Editable source (.excalidraw)
  1. About 198K comments in four classes are loaded, and 66+ features are engineered from text, style, time, engagement, demographics and thread context.
  2. Demographic columns are target-encoded out-of-fold, so no row's encoding is computed from its own label.
  3. A TF-IDF and scaling pipeline builds one feature matrix for a class-weighted LightGBM model.
  4. Logistic regression and random forest baselines are trained for comparison, and Nelder-Mead blends the out-of-fold probabilities.
  5. Per-class thresholds are tuned to maximise macro F1, which produces the final predictions.

Install and try it

You need Python 3 and the two competition CSVs (train.csv and test.csv), which are not included in the repository.

  1. Set it up
    git clone https://github.com/BriskAM/comment-toxicity.git
cd comment-toxicity
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
  2. Train

    Pass --train-csv, --test-csv and --out-dir to use your own paths.

    python scripts/train.py --config configs/default.yaml
  3. Predict
    python scripts/infer.py \
  --config configs/default.yaml \
  --ensemble runs/comment_toxicity_ensemble.npz \
  --train-csv data/train.csv \
  --test-csv data/test.csv \
  --output submission.csv
  4. Run the tests

    The original exploration and tuning notebook also runs in Colab; the link is below.

    pytest tests/ -v

Notes

All projects · Home