Model Lifecycle Support

The Hub does not just hand a team a GPU and walk away. It supports the full model lifecycle – helping choose the right model, preparing it for training or fine-tuning, testing it on the node, and evaluating it against real quality, local-language, and safety bars before anything reaches production. On a shared national resource serving health, finance, and education, "it seems to work" is not a deployment standard. This page covers the four stages that make it one.

1. Model selection support

The first question for any use case is which model, and at what size. The Hub advises on:

• Model family & task fit – matching the workload to an appropriate open-source model (e.g. an 8B Nigerian-language LLM for advisory, Whisper for ASR, a TrOCR/PaddleOCR-class model for document OCR).
• Parameter scale – the identified use cases span lightweight forecasting models through sub-8B and 7–13B language models (cited up to ~33B for the most demanding). Bigger is not automatically better on a shared node.
• Precision & quantisation – FP16 versus INT8, and what each costs in VRAM and quality. Quantisation is often the difference between a comfortable fit and a tight one in the shared 1,128 GB of HBM3e.
• Build vs adopt – whether to fine-tune an open model, adopt one as-is, or replace a proprietary dependency (e.g. moving Self-CAIRE off a GPT-4 dependency onto an open, fine-tunable model).

This works hand-in-hand with feasibility assessment, which sizes the chosen model against the node's capacity.

2. Training & fine-tuning preparation

Once a model is chosen, the Hub helps prepare the training run:

• Data staging – loading curated datasets into the node's scratch for the training window, drawing on data readiness and corpora such as AfricanVoices.
• Efficient fine-tuning – LoRA / QLoRA and similar parameter-efficient methods that cut the compute needed to adapt large models.
• Scheduling – placing training and fine-tuning in off-peak workload windows via Volcano / Kueue, using the full-mesh NVLink fabric for multi-GPU jobs.
• Experiment setup – runs, hyperparameters, and checkpoints tracked in MLflow from the start, so results are reproducible.

3. Testing

Testing spans two layers: the node itself, and the model running on it.

Infrastructure-level testing

Before any model runs, the node must be proven fit. The hardware acceptance plan verifies this through:

• GPU benchmarks – ResNet50 plus transformer micro-benchmarks, required to reach ≥90% of the vendor baseline with no thermal throttling. Throttling under sustained load would silently halve real-world throughput, so this gate matters.
• CPU throughput – a 60-minute sustained test across the dual-Xeon platform.
• End-to-end pipeline – a full ingest → preprocess → train → infer run inside the agreed SLA, proving the whole path, not just isolated components.
• Telemetry validation – confirming Prometheus / DCGM streams are live so performance and health are observable in production.

Functional testing

Exercising the model through its real pipeline – audio in → transcript → score for ORF; page in → structured record out for digitisation – rather than isolated components, so integration problems surface before users hit them.

4. Evaluation

Finally, evaluation establishes the model is good enough – and safe enough – to deploy:

• Quality benchmarks – task-specific accuracy, such as word-error-rate for the ORF ASR pipeline or OCR accuracy for health-record digitisation. Each use case gets metrics that reflect its actual job, not a generic leaderboard.
• Local-language evaluation – performance on Nigerian languages and dialects, not only English benchmarks. A model that scores well in English but poorly in Hausa fails the use case it was built for; AfricanVoices data supports realistic evaluation here.
• Safety evaluation – especially for clinical use. Self-CAIRE requires safety evaluation as a non-negotiable pre-deployment gate, tightly linked to Responsible AI.
• Continuous tracking – every run, metric, and model version stays in MLflow, so regressions are caught and re-evaluation after retraining is routine.