Invoke Capture
Production traffic is the best source of new training data — it is exactly the distribution of inputs your function needs to get right. Invoke capture is how Nyckel taps into it: every invoke is evaluated, and the most informative ones are automatically captured to your function’s Review queue for annotation.
Capture is selective by design. Annotating thousands of near-identical, high-confidence predictions costs review time without teaching the model anything new. So instead of saving everything, Nyckel captures the invokes most likely to improve the model when annotated.
What gets captured
Each invoke is checked against several capture strategies:
- Low-confidence predictions — inputs the model is uncertain about. These are the predictions most likely to be wrong, and corrections here improve the model the most.
- Random sampling — a small random slice of all traffic. Reviewing it tells you (and Nyckel) how the function performs on typical inputs, and guards against data drift — the real world changing out from under the model.
- Underrepresented classes — predictions for labels the function has seen few examples of.
A few additional rules keep the queue worth your time:
- New functions capture more. While a function has little data, early invokes are captured liberally — so during your first tests, expect essentially everything to show up in the queue.
- Duplicates are skipped. An input identical to an already-captured sample is not captured again.
- Each strategy has daily and lifetime caps. Once the queue holds plenty of unreviewed material, capture slows down rather than piling up work faster than you can review it.
The lifecycle
- 1Application invokes the function
- 2Informative invokes are captured
- 3You annotate them in the Review queue
- 4Annotated samples join the training data
- 5The model retrains and improves
- Your application invokes the function and gets back a prediction and confidence score.
- Nyckel evaluates the invoke against the capture strategies above. Informative invokes are captured; the rest are counted and answered, but not queued.
- Captured invokes appear in the Review tab of your function in the console, together with their predicted label and confidence.
- You confirm or correct each one. That annotation converts the captured invoke into a labeled sample in your training data.
- Nyckel retrains in the background, benchmarks the new model, and promotes it if it is better. No deploy step.
The result is a data flywheel: the function improves precisely on the traffic it actually serves, at a small fraction of the annotation effort of labeling everything.
The Review tab is the built-in way to work the queue, but it isn’t the only one — you can pull captured invokes via the API and route them through your own annotation tools. See Build a feedback loop.
Invokes vs. annotations
Invoke counts and annotation counts measure different things, and they are not expected to match:
- The invoke count goes up on every prediction the function serves.
- The annotation count goes up only when a captured sample is annotated (or when you add labeled samples directly).
Because only informative invokes are captured, and only annotated captures become samples, a healthy production function will always show far fewer annotations than invokes. A function with thousands of invokes and a few hundred annotations is working exactly as intended — it is not dropping data.
externalId you send. Invoke capture is a training-data mechanism, not an audit log.Opting traffic out
Capture is on by default for every function. To keep specific traffic out of the queue — load tests, synthetic traffic, inputs you never want stored — pass capture=false on the invoke:
POST https://www.nyckel.com/v1/functions/{functionId}/invoke?capture=false
Related
- The feedback loop — the improvement cycle invoke capture feeds into
- Review and improve predictions — how to spend review time well
- Build a feedback loop — send annotations from your own application instead of the console