Causilo vs XGBoost and LightGBM on one dataset

One dataset, one split, three models, the numbers as they come out. This is not a benchmark; it is the comparison you would run first, on one dataset, to decide whether a proper evaluation is worth your time. A proper one runs several datasets and several splits, with the boosters tuned.

The dataset is German credit (OpenML id 31), the split is 80/20 stratified with random_state=42, and the boosters run with their library defaults. Causilo's time includes the network round trip to the API.

pip install -U causilo-client xgboost lightgbm scikit-learn
python causilo_vs_xgboost.py

import os
import time

import lightgbm as lgb
import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.datasets import fetch_openml
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

from causilo_client import Causilo

X, y = fetch_openml(data_id=31, as_frame=True, return_X_y=True)
y = (y == "good").astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
rows = []

Causilo

One call. A cold instance takes a few seconds to come up, and wake() waits for that, so the start-up is not counted against the call. classes gives the column order of the probability matrix, which is the one field you cannot read probas without.

cx = Causilo(os.environ["CAUSILO_ENDPOINT"], token=os.environ["CAUSILO_KEY"])
cx.wake()
t0 = time.perf_counter()
proba, meta = cx.predict_with_metadata(X_train.assign(good=y_train.values), X_test,
                                       target="good", model="causilo-clf", output_type="probas")
elapsed = time.perf_counter() - t0
p = np.array(proba)[:, meta["classes"].index(1)]
rows.append(("Causilo (API, one call)", roc_auc_score(y_test, p), elapsed))

The boosters

XGBoost at its defaults, with categorical columns passed through rather than encoded. enable_categorical=True is what lets it take the DataFrame as it comes, which is the closest thing to the API's own handling.

t0 = time.perf_counter()
booster = xgb.XGBClassifier(enable_categorical=True, tree_method="hist", random_state=42)
booster.fit(X_train, y_train)
p = booster.predict_proba(X_test)[:, 1]
rows.append(("XGBoost (defaults)", roc_auc_score(y_test, p), time.perf_counter() - t0))

LightGBM at its defaults too. Both are given random_state=42; neither is tuned, and tuning is where most of a booster's accuracy on this dataset lives.

t0 = time.perf_counter()
model = lgb.LGBMClassifier(random_state=42, verbose=-1)
model.fit(X_train, y_train)
p = model.predict_proba(X_test)[:, 1]
rows.append(("LightGBM (defaults)", roc_auc_score(y_test, p), time.perf_counter() - t0))

table = pd.DataFrame(rows, columns=["model", "ROC AUC", "seconds"]).round({"ROC AUC": 4, "seconds": 2})
print(table.to_string(index=False))

Reading the table

The boosters were given nothing but defaults, and Causilo was given nothing at all: no hyperparameters exist on the API side to set. On a table this size the round trip is most of Causilo's time. The honest comparison is the one you run on your own data, with the boosters tuned the way you would tune them in production, on more than one split; the seconds column then has to include the tuning.

Output

stdout

                  model  ROC AUC  seconds
Causilo (API, one call)   0.7771     0.64
     XGBoost (defaults)   0.7238     0.16
    LightGBM (defaults)   0.7229     0.44

← Quickstart Usage, limits and cancellation →

Recorded 2026-09-18 · image 0.12.16