Flows

For the past several years, financial institutions have focused on whether generative AI can perform useful work. As pilots move toward production, the more consequential question is architectural: what is the smallest, most economical and controllable model that can perform a defined task reliably enough for a regulated environment?

That reframing is quietly redrawing AI architecture across the sector. Institutions that spent 2023–24 wiring frontier models into every workflow are now doing the arithmetic on what those workflows cost at real volume, what they cost in latency, and what they cost in explainability. The answer, increasingly, is that the largest available model is the wrong default for most of what a financial institution actually does with language.

This is the case for small language models: models roughly under ten billion parameters, often fine-tuned to a single task, frequently deployed on infrastructure the institution owns. It is also, more usefully, a description of what an implementation actually looks like and where it goes wrong.

Quick Answer

Small language models can suit bounded banking tasks such as document extraction, classification, structured drafting and internal-data access when they meet defined quality and control thresholds. They may offer lower cost, predictable latency and greater deployment control than a frontier-model API. They are not a universal replacement: strong deployments use evaluation, output validation, human oversight and escalation to larger models when needed.

Finance was never a general-purpose problem

A frontier model is built to be a generalist. The same weights must be ready for a legal argument, a Python refactor, a poem, and a question about monsoon patterns. That breadth is a real achievement, and for most financial workflows it is dead weight.

Consider what language models are actually asked to do inside a lending or banking stack:

• Extract a fixed set of fields from documents that follow one of perhaps forty layouts
• Classify an inbound complaint into one of thirty categories
• Convert a business question into SQL against a schema that changes quarterly
• Summarise a repayment history into three sentences a relationship manager will skim
• Decide whether a transaction pattern warrants human review
• Turn a rule-based risk score into narrative text that satisfies an adverse-action requirement
• Validate a payment message against a rigid format before it hits the network

None of these is open-ended. Each has a bounded input space, a fixed output schema, and a correct answer a domain expert could adjudicate in under a minute. These are not conversations. They are functions, and the industry has been implementing them on the most expensive general purpose interpreter ever built.

A 2025 NVIDIA Research position paper argued that small language models may be sufficiently capable and more economical for many repetitive calls within agentic systems (arXiv:2506.02153). That is a design proposition rather than a universal benchmark result. Institutions should test it against their own traffic by instrumenting production or pilot workloads, clustering calls by task type, and measuring quality, latency, cost, and escalation rates.

What institutions are actually buying

Cost is the headline benefit, but the magnitude varies with model, hardware utilisation, traffic, engineering overhead, and API pricing. A well-utilised small model on controlled infrastructure can reduce unit inference cost and make task-specific adaptation more practical, but institutions should compare total cost of ownership rather than assume a fixed multiplier or timeline.

Three other properties are driving the actual decisions.

Latency that survives a synchronous path. Some financial workflows—such as payment validation, real-time assistance, or field operations with weak connectivity—need predictable response times. Moving suitable inference onto controlled infrastructure can give the institution greater influence over performance, capacity planning, and fallback behaviour, although it also transfers operational responsibility to internal teams.

A deployment locus the institution controls. On-premises, private-cloud, or edge deployment can reduce the amount of sensitive data sent to external model providers and give institutions more direct control over access, logging, retention, and processing location. It does not remove obligations under India’s data-protection framework or RBI requirements; the institution must still assess purpose, consent or other lawful basis, security, outsourcing, model risk, and the full data flow.

Model change velocity matched to product change velocity. Financial products and reporting formats change frequently. Hosted models may offer prompts, retrieval, fine-tuning, or vendor-managed updates, while self-hosted specialists offer different forms of control. Neither route makes change automatic: retraining or adaptation still requires curated data, evaluation, approval, deployment controls, and monitoring. The relevant question is which operating model gives the institution dependable change on an acceptable schedule.

Where each segment lands

The three groups converging on small models are not solving the same problem, and their implementations should not look alike.

Tier-1 and mid-sized banks may prioritise governance and unit economics across high, relatively stable volumes. Their natural entry points are high-volume, lower-ambiguity and format-constrained workflows such as document extraction, structured-message validation, alert triage, and controlled narrative drafting. The value case should be demonstrated through workload-specific evidence: quality, cost, latency, traceability, human-review effort, and supervisory defensibility.

NBFCs and cooperative institutions face a different binding constraint: capability access. Frontier-model licensing and the infrastructure teams to exploit it are not evenly distributed, and RBI’s own survey work shows adoption concentrated among larger entities while smaller ones remain at the interest stage. For these institutions, small models are not an optimisation. They are the difference between having domain AI and not. A fine-tuned model on a decade of the institution’s own lending history, running on modest hardware, delivers underwriting support, collections prioritisation, and vernacular document handling that was previously out of reach at any price they could pay. The realistic constraint here is not compute; it is labelled data discipline and the two or three people who can maintain the pipeline.

Fintechs and lending-infrastructure platforms are optimising for gross margin and defensibility. Per-transaction inference cost sits directly in cost of revenue, and a platform whose margin structure improves with scale beats one whose API bill scales linearly. There is a second, sharper motive: a specialist model trained on proprietary flow (underwriting outcomes, repayment behaviour, document corpora) is an asset a competitor cannot rent. Renting general intelligence produces a product anyone can replicate; owning narrow intelligence does not.

Across all three, the same use cases keep surfacing first: document intelligence, structured-output generation, classification and triage, and natural-language access to internal data. These are the tasks where the specification is tight enough that a small model closes the capability gap almost entirely.

The reference architecture: the router is the product

The most useful shift in framing is to stop asking which model and start designing traffic allocation.

A mature deployment looks less like a model and more like a switchboard:

• A cheap classifier inspects each request and estimates task type and confidence.
• Specialist small models handle high-confidence, in-distribution, well-characterised tasks, the bulk of volume.
• Output validators sit on every path, checking schema conformance and business invariants before anything reaches a downstream system, with automatic retry.
• A frontier model serves as escalation for novel formats, low-confidence outputs, and genuinely open-ended reasoning.
• A human sits above that, on anything touching a high-stakes decision.

This architecture has a property that matters enormously in a regulated context: it degrades in the right direction. When the specialist is uncertain, cost rises and correctness is preserved. That is the opposite of what happens when a single model is tuned down until it is cheap enough.

It also makes migration incremental. Nothing is replaced. A percentage of traffic moves, gets watched, and moves further.

What the benefit lists leave out

Most writing on this topic stops at the advantages. The failure modes are more instructive.

Small specialists fail confidently. A large model encountering an out-of-distribution input usually degrades gracefully. It hedges, qualifies, or signals uncertainty. A tightly fine-tuned model given a document format it has never seen will not say so. It will emit a well-formed JSON object full of wrong values. In a credit decisioning path, a confidently wrong answer is considerably more dangerous than a refusal. This is why output validation and confidence-based escalation are not optional components.

Evaluation debt comes due immediately. A frontier model is good enough that spot-checking hides gaps for an embarrassingly long time. A specialist requires a labelled evaluation set, refreshed as data shifts, and a regression suite gating every model promotion. Most programmes underestimate this by an order of magnitude. The model is a weekend. The evaluation harness is a quarter.

Model sprawl is a real operational risk. One specialist per task is architecturally correct and operationally treacherous. Eleven models is eleven training pipelines, eleven evaluation sets, eleven drift monitors, and eleven items for a model risk committee. Institutions that adopt small models without simultaneously investing in MLOps discipline end up with worse governance than they started with: inconsistent outputs, unclear ownership, and audit findings.

Some workloads should stay large. Genuinely open-ended analysis, multi-step reasoning without fixed shape, and first-pass handling of uncharacterised document classes belong on a frontier model. Forcing these onto specialists produces a worse product and a heavier maintenance burden. The endpoint is a hybrid system, not a replacement.

The regulatory tailwind

In August 2025 the RBI released the FREE-AI committee report, a framework of seven guiding principles and twenty-six recommendations spanning infrastructure, policy, capacity, governance, protection, and assurance for AI use by regulated entities. Two elements bear directly on model architecture choice.

First, the framework’s insistence that final decision-making rests with humans, supported by explainability and independent assurance expectations, favours architectures where a specific model, trained on a specific dataset, evaluated against a specific test set, can be pointed at. A narrow model whose entire behavioural surface can be characterised is a fundamentally easier artefact to govern than a general model whose behaviour can only be sampled.

Second, the report explicitly contemplates indigenous financial AI models as sector infrastructure. Building an Indian financial model at frontier scale is a national-budget undertaking. Building a 3B model fine-tuned on Indian lending documents, regulatory formats, and language variation is something a single institution can staff.

Then there is the requirement no benchmark captures: field operations conducted across Assamese, Hindi, and English in the same conversation, on a device with intermittent connectivity. There is no cloud-API answer to that. There is a model-on-the-device answer.

RBI’s own survey found roughly a fifth of supervised entities running AI in production, with around two-thirds interested but not yet deployed. That gap is not made of enthusiasm. It is made of cost, compliance uncertainty, and scarce infrastructure talent. Small models attack the first two directly.

An implementation path that does not blow up a quarter

Instrument before optimising. Log every model call: prompt, tool arguments, output, latency, downstream outcome. Strip PII. Run it for a month. Intuitions about which calls dominate spend are reliably wrong.

Cluster and rank. Group calls by task shape; rank by volume × unit cost. Two or three task types will typically account for most of the bill, and they will be the most mechanical things the system does.

Build the evaluation set before the model. Take a few hundred real examples from the highest-volume cluster, have a domain expert label them properly, and freeze the set. This is the gate everything else passes through. Skipping it means there is no way to know whether the migration worked.

Distil or fine-tune rather than start from scratch where appropriate. Logged outputs can support dataset creation only when their use complies with privacy, contractual, intellectual-property, and quality requirements. The sub-10B class includes options across the Qwen, Phi, Gemma, Granite, and Llama families, but licences and use restrictions differ by model and version. Complete legal and security review before selection, then evaluate against the frozen test set.

Route a slice, not the workload. Send 5% of traffic to the specialist behind a validator with automatic fallback. Watch the escalation rate, not just accuracy. Then 20%. Then more.

Keep the frontier model permanently. It is not a phase to graduate from. It is the escalation path, the cold-start path for new task types, and the teacher for the next specialist.

Staff the governance layer from day one. Model inventory, ownership, drift monitoring, and promotion gates are not a phase-two concern. They are what separates a portfolio of specialists from a liability.

The real constraint is accountability, not capability

Capability is no longer concentrated only in the largest hosted models. For many bounded financial workflows, smaller open-weight or commercially licensed models may now provide a credible starting point. Availability, cost, licence terms, deployment requirements, and task performance still vary materially, so each workload requires its own evaluation.

What is scarce now is accountability: being able to state precisely what a system does, prove it to a supervisor, and change it on your own schedule without asking a vendor’s permission. And accountability is not a property of a model. It is a property of an architecture: a narrow model whose behavioural surface can be fully characterised, a schema that constrains what it is able to emit, an evaluation set that catches regression before promotion, and a router that escalates rather than guesses.

The useful inversion is that model choice should follow the task and control requirements. A smaller, purpose-built model can make evaluation boundaries and operational ownership clearer, but size alone does not guarantee explainability or safety. Those properties come from constrained outputs, representative testing, monitoring, documentation, human oversight, and escalation design.

None of this argues for abandoning frontier models. It argues for stopping the practice of treating them as the default. The default should be the smallest component that clears the bar, with everything larger reserved for the cases that genuinely need it, which is how every other piece of critical financial infrastructure has always been engineered.

The institutions that lead the next few years will not be the ones that licensed the largest model. They will be the ones that worked out, task by unglamorous task, the smallest model they could fully account for.

---

Frequently Asked Questions