predict_cached leaves the context table on the server and returns a handle.
The first call looks exactly like an ordinary one, with the context and the
query sent together, and every call after it sends only the query.
What the cache saves is the upload. The server reads the table back and encodes it again on each call, so a cached batch takes about as long as an uncached one and uses the same number of cells. At this example's size of 12,000 rows by 14 columns, or 122 KB stored, that is a convenience, and it becomes worthwhile when a context is tens or hundreds of megabytes and the questions keep coming.
pip install -U causilo-client scikit-learn
python repeated_queries.py
import os
import time
import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from causilo_client import Causilo
cx = Causilo(os.environ.get("CAUSILO_ENDPOINT", "https://api.nums.world"), token=os.environ["CAUSILO_KEY"])
The running total is read from the server before the first call, so the first difference measured is that call rather than the whole month.
_spent = [(cx.usage() or {}).get("used", 0)]
def counted(meta) -> str:
"""Cells this call added to the month, from the server's own running total.
`metadata.quota.used` is what the service has counted for this key, so the
difference between two calls is what the second one cost. Reading it this
way means the example cannot disagree with the ledger.
"""
used = (meta.get("quota") or {}).get("used")
if used is None:
return "not metered"
delta, _spent[0] = used - _spent[0], used
return f"{delta:,} cells"
print("ready:", cx.health())
The data is the adult census dataset from OpenML, which has 48,842 rows and 14 columns of mixed numeric and categorical features with missing values, and a label for income above 50K. The context uses 12,000 of those rows, because a context of a few hundred rows has very little upload to save.
X, y = fetch_openml(data_id=1590, as_frame=True, return_X_y=True)
X_train, X_rest, y_train, y_rest = train_test_split(
X, y, train_size=12_000, test_size=6_000, stratify=y, random_state=42
)
history = X_train.assign(income=y_train.values)
The query rows are split into three batches, as applicants might arrive on three days. The split uses iloc, because np.array_split turns a DataFrame into plain arrays and loses the column names the server matches on.
cuts = [len(X_rest) * i // 3 for i in range(4)]
batches = [X_rest.iloc[a:b] for a, b in zip(cuts, cuts[1:])]
labels = [y_rest.values[a:b] for a, b in zip(cuts, cuts[1:])]
print("history", history.shape, "batches", [b.shape for b in batches])
POSITIVE = ">50K"
def auc(proba, meta, truth):
proba = np.array(proba)
good = proba[:, meta["classes"].index(POSITIVE)]
return round(roc_auc_score(truth == POSITIVE, good), 4)
Without the cache
print("\nwithout the cache")
plain = []
for i, (batch, truth) in enumerate(zip(batches, labels)):
t = time.monotonic()
proba, meta = cx.predict_with_metadata(
history, batch, target="income", model="causilo-clf", output_type="probas"
)
plain.append(auc(proba, meta, truth))
# The cells the server counted, read from the response instead of worked
# out here, so that the example always shows the server's own rule even
# if that rule changes.
print(f" batch {i}: {time.monotonic() - t:.2f}s "
f"counted {counted(meta)} "
f"AUC {plain[-1]}")
With the cache
print("\nwith the cache")
t = time.monotonic()
proba, ctx = cx.predict_cached(
history, batches[0], target="income", model="causilo-clf", output_type="probas"
)
The first call is an ordinary one. It carries the history, is counted for it and answers the first batch, and it leaves the handle behind.
print(f" batch 0: {time.monotonic() - t:.2f}s "
f"counted {counted(ctx.metadata)} (the context travels with this one)")
print(" handle:", ctx)
# The metadata of the call that stored the context is kept on the handle.
cached = [auc(proba, ctx.metadata, labels[0])]
for i, (batch, truth) in enumerate(zip(batches[1:], labels[1:]), start=1):
t = time.monotonic()
proba, meta = ctx.predict_with_metadata(batch)
cached.append(auc(proba, meta, truth))
print(f" batch {i}: {time.monotonic() - t:.2f}s "
f"counted {counted(meta)} "
f"AUC {cached[-1]}")
# The server reports what it is holding and how large it is.
for held in cx.list_contexts():
print("\nheld:", held)
Release the context when the run is over. It would expire on its own after six hours, and releasing it removes your table from our storage sooner.
print("released:", ctx.release())
print("held after release:", cx.list_contexts())
What to expect
Every call above counts 196,000 cells whether or not the context is cached, because the context rows are counted on every query. The calls also take about as long as each other, because the stored table is read back and encoded again each time.
The scores with and without the cache are identical.
stored_bytes says what is held: this 12,000-row context is 122 KB as a
table. Held as the model's encoded form it was 4.75 GB, which is why it is
not held that way.
print("\nAUC without the cache:", plain)
print("AUC with the cache: ", cached)
Output
stdout
ready: True
history (12000, 15) batches [(2000, 14), (2000, 14), (2000, 14)]
without the cache
batch 0: 1.76s counted 196,000 cells AUC 0.9385
batch 1: 1.66s counted 196,000 cells AUC 0.9285
batch 2: 1.62s counted 196,000 cells AUC 0.9226
with the cache
batch 0: 2.14s counted 196,000 cells (the context travels with this one)
handle: CachedContext(model='causilo-clf', target='income', rows=12000, context_id='ctx_gfGJepVKj6cZpXWrNHOIot')
batch 1: 1.58s counted 196,000 cells AUC 0.9285
batch 2: 1.55s counted 196,000 cells AUC 0.9226
held: {'context_id': 'ctx_gfGJepVKj6cZpXWrNHOIot', 'model': 'causilo-clf', 'target': 'income', 'n_context': 12000, 'n_features': 14, 'stored_bytes': 122523, 'expires_in_s': 21596}
released: True
held after release: []
AUC without the cache: [0.9385, 0.9285, 0.9226]
AUC with the cache: [0.9385, 0.9285, 0.9226]