The dataset is German credit (OpenML id 31), the split is 80/20 stratified with random_state=42, and the boosters run with their library defaults. Causilo's time includes the network round trip to the API.
pip install -U causilo-client xgboost lightgbm scikit-learn
python causilo_vs_xgboost.py
import os
import time
import lightgbm as lgb
import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.datasets import fetch_openml
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from causilo_client import Causilo
X, y = fetch_openml(data_id=31, as_frame=True, return_X_y=True)
y = (y == "good").astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
rows = []
Causilo
Causilo needs one call. A cold instance takes a few seconds to come up, and
wake() waits for that so the start-up is not included in the timing.
classes gives the column order of the probability matrix, which you need in
order to read probas.
cx = Causilo(os.environ.get("CAUSILO_ENDPOINT", "https://api.nums.world"), token=os.environ["CAUSILO_KEY"])
cx.wake()
t0 = time.perf_counter()
proba, meta = cx.predict_with_metadata(X_train.assign(good=y_train.values), X_test,
target="good", model="causilo-clf", output_type="probas")
elapsed = time.perf_counter() - t0
p = np.array(proba)[:, meta["classes"].index(1)]
rows.append(("Causilo (API, one call)", roc_auc_score(y_test, p), elapsed))
The boosters
XGBoost runs at its defaults with the categorical columns passed through
unencoded. enable_categorical=True lets it take the DataFrame as it comes,
which is the closest equivalent to the API's own handling.
t0 = time.perf_counter()
booster = xgb.XGBClassifier(enable_categorical=True, tree_method="hist", random_state=42)
booster.fit(X_train, y_train)
p = booster.predict_proba(X_test)[:, 1]
rows.append(("XGBoost (defaults)", roc_auc_score(y_test, p), time.perf_counter() - t0))
LightGBM also runs at its defaults. Both boosters are given
random_state=42 and neither is tuned, although tuning is where much of a
booster's accuracy on this dataset comes from.
t0 = time.perf_counter()
model = lgb.LGBMClassifier(random_state=42, verbose=-1)
model.fit(X_train, y_train)
p = model.predict_proba(X_test)[:, 1]
rows.append(("LightGBM (defaults)", roc_auc_score(y_test, p), time.perf_counter() - t0))
table = pd.DataFrame(rows, columns=["model", "ROC AUC", "seconds"]).round({"ROC AUC": 4, "seconds": 2})
print(table.to_string(index=False))
Reading the table
The boosters ran with their default settings, and Causilo has no hyperparameters on the API side to set at all. On a table of this size most of Causilo's time is the network round trip. On your own data, with the boosters tuned as they would be in production and over more than one split, the seconds column should include the time spent tuning.
Output
stdout
model ROC AUC seconds
Causilo (API, one call) 0.7771 0.57
XGBoost (defaults) 0.7589 0.16
LightGBM (defaults) 0.7229 0.36