The reason to run a self hosted LLM against your own source code is short. The code never leaves the building.
The harder question comes next, and most teams reach it a week later. Which model, at what size, on what hardware, and how does anybody prove it is good enough before the purchase order goes out?
This is an evaluation guide for that decision. It covers what the job actually demands, how model size follows from the job, and how to build a test set out of programs your team already understands.
What self hosted has to mean before your security team signs
Two definitions circulate under the same phrase, and they carry different risk.
The first means the model weights sit on hardware your organization controls, and every inference call runs on that hardware. Nothing about your source reaches an outside network. This is the definition your security team has in mind.
The second means a vendor runs a dedicated instance for you inside their cloud. That arrangement has real advantages and it is a different control story, because your source still travels to infrastructure somebody else administers.
Settle which definition applies before the technical evaluation starts. Our page on air-gapped AI deployment for enterprise covers the network side of the first definition in detail.
Size the model against the job it will actually do
Teams evaluate models by asking them to chat. The job here is narrower and the narrowness is an advantage.
A model reading your IBM i library does four things, and none of them looks like a chatbot. It explains what a program does. It answers questions about structure, such as which programs a change touches. It drafts converted code in a target language your team picked. It flags places where the source does something the model cannot account for.
None of those tasks needs general world knowledge. None of them needs the model to hold a conversation across an afternoon. All four benefit from a model that follows instructions precisely and admits uncertainty, and those two traits do not track parameter count as closely as the marketing suggests.
Write the four tasks down before you look at a single model. The team that starts from a leaderboard buys capability it will never call.
Context window is the first hard constraint on AS/400 source
This is where most RPG evaluations go wrong, and the failure is quiet.
A fixed-format RPG member on an AS400 system can run thousands of lines. The program pulls in copy members from other libraries, and the copy members pull in more. Understanding one program means holding the program and every copy member it pulls in, plus the record formats of the files it opens.
Measure your own library before you choose. Find the longest source member in production and count its lines. Follow its copy members and count those too. That combined number, converted to tokens, sets the floor for the context window your team needs.
Converting lines to tokens takes a few minutes and no guesswork. A token is roughly a short word or a piece of one, and every model ships with the tokenizer that produced its training data. Run your longest member through that tokenizer and you have the real number for that model. Do it again for each candidate, because the count moves between tokenizers.
A model whose window sits under that floor still runs. It reads a truncated program and produces a description that sounds right and describes half the logic. Nobody catches it, because checking requires reading the program, which is the work the tool was supposed to do.
Two ways out exist when the floor is high. Pick a model with a larger window, or reduce what the model has to hold. The second route is worth understanding, because it changes the hardware bill.
Structure lowers the model size you need
Handing a model raw fixed-format RPG asks it to do two jobs at once. It has to work out the structure of the code, then reason about the behavior. The first job is mechanical and a model is an expensive way to do it.
AS/Forward splits the two. It reads six RPG dialects through a proprietary ingestion layer and builds an interconnection graph across every program, file, copy member and display file in the library. From that structure it generates a plain-English description of each program. That first pass involves no model at all.
The model then receives two inputs for one program: the source and the description the ingestion layer generated from that same source. The ingestion layer finishes the structural work before inference starts, and the model spends its context on behavior.
The practical effect lands on your budget. A smaller model does useful work on a structured prompt, which means the hardware you buy can be the hardware your data center already accommodates. AS/Forward carries one US patent pending on the approach.
There is a verification benefit too. Because the description comes from the ingested source, a developer can open the member and check any sentence in it. A description that came only from a model is a claim your team has to trust.
Sizing hardware without a benchmark you cannot verify
Vendor throughput numbers come from somebody else's prompt on somebody else's hardware. Derive your own from four quantities you already control.
Memory comes first. The memory a model needs scales with parameter count and with the precision of the weights. Quantization reduces precision to fit a larger model into less memory, and it costs some accuracy. How much accuracy it costs on RPG description work is a question only your test set answers.
Library volume comes second. Four thousand programs at an average member length gives you total tokens for a full pass, and total tokens divided by throughput gives you the length of the run. Count the members before anybody sizes a box.
Concurrency comes third. One developer asking questions interactively is a different load from a batch pass running overnight across the whole library. Most teams need both, and the batch pass usually sets the hardware.
Latency tolerance comes fourth. A developer waiting on an answer notices seconds. An overnight pass notices nothing until the window closes at six in the morning. Decide which one governs before anybody quotes a GPU.
Run the arithmetic in that order and the hardware conversation becomes concrete. Skip it and you will buy for the demo you watched.
Build the evaluation set from your own IBM i library
A public benchmark tells you how a model handles somebody else's code. Your test set has to come from AS/400 programs your team already understands cold.
Pick the programs deliberately. Take one your senior developer wrote and can describe from memory. Take one nobody wants to touch, the four-thousand-line program that runs the nightly close. Take one with a copy member that stopped resolving years ago. Take one that calls another program by a name it builds at runtime.
Then score each answer against four questions.
- Does the description match the program, when the developer who knows that program checks it line by line?
- Does it catch the exception path, meaning the branch that fires twice a year for one customer?
- Does it name what it could not resolve, or does it fill the gap with a confident guess?
- Does the same program produce the same description on a second run?
That third question separates candidates faster than any other. A model that invents a plausible explanation for an unresolved copy member is worse than one that stops and says the member is missing. The invention survives into a design document, and nobody questions it.
Run the same set against every candidate model. Keep the answers. Six months later, when somebody proposes a swap, the test set is still there and the comparison takes an afternoon.
Keep the decision reversible
Model choice is the fastest-moving variable in this whole project. A tool that welds itself to one model makes your team redo the evaluation every time the field moves.
AS/Forward runs against Claude, OpenAI, Gemini and local models on customer hardware. That matters in two directions. A team can evaluate the whole workflow against a hosted model on public sample code, then run production entirely on local models inside an air-gapped install. A team that starts local can test a new local model without changing anything else in the pipeline.
The evaluation set makes the swap safe. Your team reruns the same programs and decides on the same four answers.
What a self hosted LLM does not fix
Four limits are worth stating plainly, because a demo hides every one of them.
A model cannot read a source member your team never gave it. Members sitting in a library the list never reaches stay invisible to the model and to everyone reading its output. The ingestion pass reports those as unresolved, and that report is the honest inventory.
A model cannot tell you that the source no longer matches the running object. Somebody edited a member in place in 2011 and recompiled from a different copy. The description will correctly describe the source it read, and the production behavior will differ.
A model cannot see the parts of the system that live outside source. Job scheduler entries and exit program registrations shape what happens at two in the morning, and neither one sits in a member. Your team documents those separately and attaches them to the map.
A model cannot decide what the business wants. It can tell your team that eleven programs read the tax identifier. Whether that is acceptable is a decision a person makes. Our page on RPG code analysis covers what the source-level inventory does and does not settle.
Who owns the model after it lands
A hosted model has an operator. A self hosted model has your team, and the ownership shows up in four places.
Somebody patches the host and the serving software on your normal cycle. Somebody decides when new weights come in and signs off on the swap after running the evaluation set against them. Somebody watches capacity, because a batch pass that finished by six in the morning last quarter takes longer as the library grows. Somebody answers the auditor who asks which model version produced the descriptions in the evidence pack.
Name those people at install time. The most common failure in a self hosted deployment has nothing to do with the model. The host still runs the weights somebody carried in eighteen months ago, and the evaluation set sits in a folder nobody opens. Nobody can say whether a newer model would read the nightly close more accurately.
Questions to ask before you buy hardware
Take these into the vendor call and write the answers down.
- On our longest source member with its copy members attached, what is the largest single prompt this workflow sends?
- Which stages call a model, and which stages run without one?
- Which models has your team run against real RPG, and what changed between them?
- How does our team swap the model later, and what breaks when we do?
- What does the product do when it cannot resolve a dependency?
A vendor who answers the last question with a mechanism has thought about the failure mode. A vendor who answers it with reassurance has not.
Start with the programs, then the model
The order matters more than the model does. Scope the source libraries you want read. Measure the longest member and its copy members. Build the test set from programs your developers know. Then evaluate models against that set on hardware you can actually buy.
Golden Path Digital tested the ingestion layer behind AS/Forward against roughly 5,000 programs drawn from PUB400 and public code repositories. The AS/Forward page covers the product, and a call gets you a scoped answer for your own library. Reach us at 501-232-7188 and tell us how long your longest program runs.