Machine Learning Development
Predictive models trained on your data, evaluated honestly, shipped to production.
A model is only useful once it is making predictions on real data in real time. We design, train and evaluate ML models against the specific metric your business cares about — not accuracy on a benchmark that has nothing to do with your users — and we deploy them as reliable, versioned APIs that your product can call.
Our ML work is grounded in data you actually have, not a hypothetical dataset. We run an honest baseline first, so you know how much lift the model provides over a simple heuristic before committing to the full engineering investment. Then we build, instrument and hand it over with the documentation and retraining schedule that keeps it performing as your data evolves.
What this covers
Exploratory data analysis and feature engineering on your own dataset
Baseline modelling and honest lift measurement versus simple heuristics
Model development: classification, regression, time-series forecasting, ranking
Cross-validation, hyperparameter tuning and model selection with documented rationale
Evaluation report: held-out test metrics, confusion matrices, failure mode analysis
Model deployment as a versioned REST API with latency SLA
Retraining schedule, drift detection triggers and a clear owner for ongoing quality
Deliverables
- A trained, validated model deployed as a callable API on infrastructure you own
- An evaluation report covering accuracy, precision/recall, known failure modes and limitations
- Feature engineering pipeline and training code in source control, fully reproducible
- A retraining and monitoring playbook so the model stays accurate as data shifts
What we build with
Custom Predictive Model Development
Custom ML model development means training a statistical model on your specific data to automate or augment a specific decision — classification, regression, ranking, or forecasting. It is distinct from prompting a foundation model: the output is a versioned, deployable artifact whose behaviour is determined by the patterns in your training data, not a general-purpose model's parametric memory.
The critical decision that most teams face too late is whether the problem actually needs a custom model. A well-engineered gradient-boosted tree on a thousand labelled examples frequently outperforms a fine-tuned transformer on the same classification task at a fraction of the compute cost and with far better interpretability. The architecture choice should be driven by a measured baseline, not by which approach sounds more sophisticated.
Principles for Reliable Predictive Modelling
These principles describe where ML projects most commonly fail and what engineering discipline each failure mode demands.
Baseline before model
The first deliverable on any ML project is a baseline — a rule-based heuristic, a random forest trained on raw features, or the current manual process expressed as a deterministic function. The baseline sets the minimum bar the model must beat and reveals how much signal is actually in the available features. Teams that skip this step frequently discover late that a business rule would have been sufficient, or that no model can improve on the baseline because the signal is too weak.
Class imbalance is a design constraint, not a tuning parameter
On imbalanced classification problems — fraud detection, churn prediction, rare event classification — optimising for accuracy produces a model that predicts the majority class for every input and scores 97% accuracy while being operationally useless. The correct response is to choose an appropriate metric (precision-recall AUC, F-beta, Matthews correlation coefficient) before training, and to use class-weight adjustments, oversampling (SMOTE), or undersampling as a deliberate design choice rather than a post-hoc fix.
Feature leakage is silent and catastrophic
A feature that encodes information about the target label — a field populated after the event being predicted, a timestamp that correlates with the label assignment process, an aggregate computed on the full dataset rather than a training-time window — will produce an offline metric that makes the model appear significantly better than it is. The leak is invisible until the model is deployed and makes predictions on new data where the leaking feature is unavailable or uninformative. Leakage detection requires temporal audit of every feature's data lineage, not just a correlation check.
Slice evaluation, not just aggregate metrics
A model with 89% aggregate accuracy may perform at 61% on a specific demographic, product category, or geographic segment that is underrepresented in the training set. Aggregate metrics hide these failure modes. Every model evaluation report must include performance breakdown by the slice dimensions that matter for the business — if the model is a content recommender, break down by content age and category; if it is a credit model, break down by applicant segment. Slices that fall below the business SLO are not a statistical curiosity — they are a deployment blocker.
Quantify uncertainty where the decision is high-stakes
A point prediction from a classification model — a 0.73 probability of churn — is not the same as a calibrated probability. If the model is poorly calibrated, the 0.73 score does not mean 73% of similarly-scored customers churn; it could mean anything. For decisions where a miscalibrated confidence score changes an action — whether to call a customer, whether to flag a transaction for review — use Platt scaling or isotonic regression to calibrate the model's output probabilities, and validate calibration on a held-out set using an expected calibration error (ECE) measurement.
How we'd choose
There's rarely one right answer — these are the trade-offs we weigh before recommending an approach.
Classical ML vs deep learning vs pretrained + fine-tune
The architecture decision should follow from the data characteristics and the deployment constraints, not from the current state of the art on benchmark leaderboards. This guide covers the three regimes most common in custom ML engagements.
| Criterion | Classical ML (GBM, logistic regression, SVM) | Deep learning from scratch (MLP, CNN, LSTM) | Pretrained model + fine-tune (Hugging Face, timm) |
|---|---|---|---|
| Data volume required | Low to medium — competitive performance from hundreds to tens of thousands of examples on tabular and structured data | High — requires tens of thousands to millions of examples to learn useful representations from random initialisation; generalises poorly on small datasets | Low to medium — transfer learning means the model arrives with strong priors; fine-tuning on hundreds of domain examples can yield strong performance |
| Input modality | Tabular, structured, or hand-engineered features — the native domain of GBMs and linear models | Raw signals where feature engineering is the bottleneck: image pixels, raw audio waveforms, byte sequences | Text (language models), images (ViT, ResNet, EfficientNet), or audio (Wav2Vec2) — any modality with a strong public pretrained model |
| Interpretability | High — SHAP values, feature importances, and partial dependence plots are first-class outputs; regulatory environments that require explainability favour this regime | Low — learned representations are not directly interpretable; SHAP approximations are available but less reliable on deep architectures | Low to medium — attention weights provide weak interpretability signals; probe classifiers on intermediate representations are the best available tool |
| Training compute cost | Minimal — trains on CPU in minutes to hours; hyperparameter search via Optuna is inexpensive | High — GPU-hours to GPU-days depending on architecture and dataset; expensive to iterate | Low to medium — fine-tuning a pretrained checkpoint on a GPU takes minutes to hours; full fine-tune of a large model is expensive; LoRA or adapter methods reduce this significantly |
| Inference latency | Sub-millisecond on CPU — trivial to deploy as a lightweight FastAPI endpoint with no GPU requirement | Variable — small networks are fast on CPU; larger architectures require GPU for acceptable latency; ONNX export reduces serving overhead | Model-dependent — large pretrained models are slow without quantisation or distillation; exporting to ONNX and applying INT8 quantisation typically recovers 2-4x throughput |
| Recommended starting point | Yes — for any tabular or structured-data problem, start here; a well-tuned LightGBM is the baseline that deep learning must demonstrably beat before the additional complexity is justified | Rarely — only when the modality genuinely lacks a pretrained model and the dataset is large enough to support training from random initialisation | Yes — for text, image, and audio tasks; choose the smallest pretrained checkpoint that achieves acceptable accuracy on your eval set before scaling up |
Anti-Patterns in Custom ML Development
Validation set contamination
Preprocessing steps — imputation, normalisation, feature selection, or oversampling — are fitted on the full dataset before the train-validation split. The validation set has implicitly seen information from the test distribution, producing offline metrics that overstate real-world performance. The contamination is often introduced accidentally by scikit-learn code that calls fit_transform on a DataFrame before splitting rather than inside a Pipeline.
Wrap all preprocessing in a scikit-learn Pipeline object so that every fit step executes only on the training fold during cross-validation. For temporal data, enforce a time-based split and validate that no feature for row t is computed using data from time t+n. Run a leakage audit as a mandatory step before any model evaluation: for every feature, document its data source and the timestamp at which it would be available in production.
Optimising the wrong metric
The model is trained and evaluated against accuracy or AUC-ROC because these are the default metrics in the chosen library. The business problem is a recall-sensitive fraud detection task where missing a positive is ten times more costly than a false alarm. A model that achieves AUC 0.92 but operates at a threshold that yields 40% recall is not a useful fraud detector — it is a model that was never evaluated against the decision criterion it will actually be judged on.
Define the business decision criterion before model selection — not after. For recall-sensitive tasks, use precision-recall AUC and set the operating threshold by the acceptable false-negative rate, not by maximising F1. For cost-sensitive tasks, construct a cost matrix and evaluate expected cost per prediction as the primary selection criterion. The metric the model is optimised for must match the metric the business will evaluate it on in production.
Pickle-file deployment
The trained model is serialised to a pickle file and copied to a server by the engineer who trained it. There is no model registry, no version record linking the artifact to the training run and dataset snapshot, no containerised inference environment, and no rollback mechanism. When the model breaks — because the server Python version changes, because the scikit-learn minor version is incompatible, or because the preprocessing logic in the notebook diverges from the serving code — the only recovery path is asking the original engineer to reproduce the training run from memory.
Register every trained model artifact in MLflow or Weights and Biases with a pointer to the training run, the dataset version, and the exact dependency environment. Containerise the serving environment with pinned dependencies so the model runs identically in development, staging, and production. Define the rollback procedure — switching the model registry pointer to the previous version — before deploying any model, not after a production incident.
Common questions
How much data do we need before ML is worth the investment?
It varies by problem. A well-framed classification problem with a thousand labelled examples can produce a model that outperforms a manual process significantly. We run a data audit in the first week and tell you honestly whether you have enough, whether you need more, or whether a simpler rule-based system is the right answer.
Who retrains the model after you hand it over?
We document the retraining process so your team can run it, and we set up the MLflow or W&B experiment tracking so the history is preserved. If you prefer, we can stay on retainer to run quarterly retraining cycles and monitor drift — the choice is yours.
Related capabilities
Have something in mind?
Tell us what you're building or stuck on. The first consultation is free — no obligation, no hard sell.