← All experience

CACIB, Credit Agricole CIB · Corporate and investment banking

Most of the latency was not the model

ML engineer, workflow performance and deployment architecture

Nov 2025 - May 2026

A banking analyst assistant answered in 50 to 60 seconds. I profiled the complete request path before changing the model and found that roughly 70% of the delay came from prompt construction, orchestration, reasoning settings, and cold starts.

The constraint was the complete request path

The workflow combined external APIs, internal information, several agent steps, and model calls over classified C3 and C4 data. The initial brief focused on fine-tuning, but model quality alone could not explain the response time.

I separated inference time from the work surrounding it. Large prompts were rebuilt on every call, reasoning was enabled even when the task did not need it, orchestration repeated work, and serverless endpoints introduced cold starts. That attribution changed the mission from a model-only exercise into a serving-architecture problem.

The deployment redesign

The revised path removed redundant orchestration, reduced prompt construction, and applied reasoning only where the task required it. A fine-tuned Qwen model ran through vLLM on the client's own GPUs, keeping confidential data inside the client's perimeter.

The smaller model was selected for the latency and deployment envelope, then evaluated at equal task performance against the original workflow. Rebuilding the serving path and model configuration reduced latency by 50%.

Fine-tuning without labelled ground truth

The client had no labelled dataset for the target task. Training examples therefore came from generated candidates reviewed against the workflow's expected behaviour. Supervised fine-tuning established the task format, followed by Group Relative Policy Optimization experiments.

Fine-tuning was one part of the result, not the complete explanation. The latency audit showed that a faster model would have left most of the original delay untouched.

Why the model stayed small

The model needed strong tool-calling behaviour, low serving latency, and a deployment path onto the client's own GPUs. That narrowed the choice to the Qwen family in the 4B to 8B range. A larger general model could improve language quality, but it would work against the latency and infrastructure constraints that motivated the mission.

With no production ground truth, Claude Sonnet generated full trajectories from the user request to the final answer. Reviewed trajectories became the training set for the smaller student model. Evaluation then compared task behaviour before deployment and measured latency through the revised serving path.

One model did not improve every capability equally

Fine-tuning improved tool calling. It did not produce the same gain when the small model also had to manage every tool call and formulate the final response. Better tool use did not translate proportionally into stronger general language generation.

That result changed the recommended architecture. A specialised small model can handle tool selection while a stronger language model formulates the final response. The recommendation follows the measured capability boundary instead of asking one model to be optimal at incompatible tasks.

The serving choice was constrained by the client's environment

vLLM was the preferred runtime for the on-premises GPUs. A SageMaker endpoint remained an option on AWS, but SageMaker did not provide the required serverless GPU endpoint. The client also did not operate Kubernetes, so the strongest long-term serving recommendation could not become the delivered platform during this engagement.

Result and delivery boundary

The measured result was a 50% latency reduction at equal task performance. The application had not reached production by the end of the engagement, so the result does not establish production concurrency or scaling behaviour.

The recommended long-term serving platform was Kubernetes, but the bank did not run Kubernetes during the mission. Delivery used on-premises GPUs with AWS available in parallel. Kubernetes remains a recommendation, not implemented work.

• LangGraph
• Qwen 4B-8B
• SFT
• GRPO
• vLLM
• AWS SageMaker
• Bedrock
• Lambda
• EC2
• On-premises GPUs
• MLflow