Modernization

How Well Does IBM Bob Really Understand IBM i? Six Tests, Some Surprises, and a Few Hallucinations

A hands-on benchmark of IBM Bob across RPG, Db2 for i, SQLRPGLE, REST APIs and DevOps — including where it impressed, where it struggled, and where it hallucinated.

Saurabh Arora

Saurabh Arora

IBM i Solution Architect | Technical Lead | Modernization & Integration Specialist

Published September 16, 202616 min read
IBM iIBM BobRPGDb2 for iSQLRPGLEDevOpsAIModernization

How Well Does IBM Bob Really Understand IBM i? Six Tests, Some Surprises, and a Few Hallucinations

AI coding assistants are becoming increasingly capable.

But IBM i presents an interesting test.

It is not difficult for an AI assistant to explain an IF statement, generate a SQL query, or refactor a generic piece of application code.

The harder questions are different.

Can it understand a program that mixes fixed-format RPG, free-format RPG, native record-level access, embedded SQL, external program calls and IBM i-specific lifecycle behaviour?

Can it modernize old RPG without accidentally changing its runtime behaviour?

Can it distinguish a good SQL optimization from one that changes locking or commitment-control semantics?

Can it design an outbound REST integration using actual IBM i capabilities rather than APIs that merely sound plausible?

And can it understand that an IBM i deployment pipeline should promote the artifact that was tested rather than quietly compiling it again?

Those were the questions I wanted to explore with IBM Bob.

So instead of giving it a few simple prompts and writing a first-impressions review, I built a small reproducible IBM i benchmark.

The results were more interesting than I expected.

Bob did some things remarkably well.

It also made a few mistakes that would be easy to miss if you did not already understand the IBM i platform.

And that, in my opinion, is where this gets interesting.

The Benchmark

I created six synthetic tests representing work that an IBM i developer or architect could realistically encounter.

Test Area What I wanted to know
1 RPG comprehension Can Bob understand mixed legacy/modern RPG?
2 RPG modernization Can it modernize RPG without changing behaviour?
3 SQLRPGLE Can it identify SQL, transaction and error-handling problems?
4 Db2 for i performance Can it reason about indexes and query access paths?
5 REST + JSON Can it design a safe IBM i integration?
6 Git / CI-CD Can it review an IBM i deployment pipeline and release model?

The source was purpose-built for the benchmark. No production or proprietary code was used.

That mattered for two reasons.

First, I wanted the tests to be publishable so somebody else could eventually repeat them.

Second, I wanted to deliberately insert subtle problems where an answer could look convincing while still being technically wrong.

For every test I had a hidden answer key containing the important behaviours, intentional defects and several hallucination traps.

Bob did not see those answer keys.

The goal was not to see whether Bob could reproduce an answer I had already given it. The goal was to see what it would discover independently.

How I Scored It

Each response was evaluated across five areas:

That last category became increasingly important as the testing progressed.

An answer can be beautifully formatted, highly detailed and completely confident — and still contain one IBM i-specific statement that would send a developer in the wrong direction.

I preserved Bob’s first response before correcting anything.

For the baseline runs, each test was intended to be isolated from the others. Tests 1 and 2 were run before I fully standardized that reset, so those two results have a small methodology caveat. From Test 3 onward, I used a fresh context so earlier results could not influence the next test.

Here is the scorecard from the first-pass evaluation:

Test Area Score
1 RPG comprehension 8.6 / 10
2 RPG modernization 8.4 / 10
3 SQLRPGLE 7.8 / 10
4 Db2 for i performance 9.0 / 10
5 REST + JSON 6.9 / 10
6 IBM i DevOps / CI-CD 8.5 / 10
Average 8.2 / 10

I would not focus too much on the average, though.

The individual mistakes tell a much more useful story.

Test 1: Can Bob Actually Understand RPG?

The first program deliberately looked like something many IBM i developers eventually encounter.

It contained:

The instruction to Bob was simple:

Understand the program first. Do not rewrite it yet.

I wanted to see whether Bob could explain the existing implementation and separate what it could determine directly from the code from what it was merely inferring.

What Bob Did Well

This was probably the moment when I realized the benchmark was going to be more interesting than I expected.

Bob reconstructed the program flow accurately.

It understood the interaction between native RPG I/O and Db2 for i SQL.

It recognized CHAIN, %FOUND, EXTPGM, *INLR, the internal procedure and the hybrid fixed/free source.

More importantly, it found most of the deliberate defects.

For example, the customer lookup contains a SQL error path that builds an error message but then simply continues processing.

Bob noticed that.

It also noticed that:

It even found something deeper in the business logic.

A held order returned a non-zero code from the internal procedure, causing the mainline logic to treat a valid H business state almost as though processing had failed.

That was a good observation.

Bob was reasoning about the program rather than merely translating RPG syntax into English.

Where It Went Wrong

Then came a very confident explanation about the RPG data-structure parameter.

Bob effectively treated the absence of CONST as though it implied pass-by-value semantics.

That is not the right interpretation.

The better observation is that the parameter is modifiable by the called procedure even though the procedure does not modify it, and CONST could therefore better communicate the intended contract.

It was a small part of an otherwise strong answer.

But it was also important.

A developer unfamiliar with RPG parameter semantics could read the explanation and reasonably assume it was correct.

That became the first clue to a pattern I would see repeatedly.

Bob was very good at understanding the application.

It was less reliable when explaining some of the deeper IBM i runtime semantics.

Test 1 score: 8.6 / 10

Test 2: Modernize the RPG — But Don’t Change What It Does

For Test 2 I gave Bob a more traditional RPG program containing result indicators, GOTO, TAG, fixed-format calculations and an *ENTRY PLIST.

The most important instruction was:

Preserve behaviour.

That matters because modernization does not always mean replacing every old technique with whatever looks newest.

A Good Modernization Decision

Bob converted the source into readable full-free RPG.

It replaced:

But the best decision was something it didn’t do.

It did not automatically replace native CHAIN and UPDATE operations with SQL.

Instead, it noted that changing the access method could change locking and update semantics, and kept the first modernization focused on syntax and maintainability.

That is a good modernization principle.

Modernize intentionally.

Do not change three architectural dimensions simply because you were asked to remove fixed-format RPG.

Again, Runtime Semantics Were the Weak Point

The weaker part of the answer involved activation groups.

Bob changed the program to ACTGRP(*CALLER) and tried to justify it as the safest equivalent of the original runtime model.

That is exactly the kind of change that deserves more caution.

Activation-group choice affects resource lifetime and program behaviour, and “modernization” by itself is not enough justification to change it.

The code modernization was strong.

The explanation of IBM i lifecycle semantics was less dependable.

Test 2 score: 8.4 / 10

Test 3: SQLRPGLE, Transactions and the Danger of Specific-Sounding Advice

The third test contained a cursor-driven SQLRPGLE batch process with deliberately poor error handling.

Bob found almost everything.

It noticed that:

Then I added one follow-up condition:

Assume this can process 100,000 rows. Re-evaluate the design for performance and lock duration.

This produced one of Bob’s best recommendations in the entire benchmark.

It realized that the best optimization was not to optimize the cursor.

It was to remove it.

The per-row business rule could be expressed directly as set-based SQL:

UPDATE ORDHDR
   SET OHSTS =
       CASE
         WHEN OHAMT > 10000 THEN 'H'
         ELSE 'R'
       END,
       OHPROC_TS = CURRENT_TIMESTAMP
 WHERE OHCUST = :customer
   AND OHSTS = 'N';

followed by:

GET DIAGNOSTICS :updated = ROW_COUNT;

That is exactly the type of modernization I want an AI assistant to suggest.

Do not make the loop 15% faster if SQL can eliminate the loop.

Then the Answer Became Too Specific

The same response later moved into highly specific IBM i locking guidance and introduced system-value-style details and thresholds that were not supported.

This is where the benchmark became particularly useful.

The high-level recommendation was excellent:

A very large unit of work can hold resources for too long and reduce concurrency.

That is a useful engineering consideration.

The problem was not that Bob discussed locks.

The problem was that it turned a generally correct concern into highly specific platform guidance that sounded authoritative without being reliable.

That may be one of the most important lessons from the entire benchmark:

Specificity is not the same thing as correctness.

An AI assistant saying:

“Large transactions can increase lock duration and reduce concurrency”

is useful.

An AI assistant giving you a precise IBM i system value, default threshold and operational limit should trigger verification.

Test 3 score: 7.8 / 10

Test 4: Db2 for i Performance

This was Bob’s strongest result.

The test used a synthetic product cross-reference table rather than any production schema.

The query looked like this:

SELECT
    COALESCE(
        LISTAGG(DISTINCT PRODUCT_CODE, '^')
        WITHIN GROUP (ORDER BY PRODUCT_CODE),
        ''
    ),
    COALESCE(
        LISTAGG(DISTINCT CATALOG_CODE, '^')
        WITHIN GROUP (ORDER BY CATALOG_CODE),
        ''
    )
FROM PRODUCT_XREF
WHERE CANONICAL_ID = ?;

The synthetic table had a primary key and an existing index built around:

CATALOG_CODE
PRODUCT_CODE
SOURCE_SYSTEM

while the query actually filtered by:

CANONICAL_ID

The existing index therefore did not line up well with the predicate.

What Bob Did Well

Bob recognized the problem immediately.

It proposed an index beginning with:

CREATE INDEX PRODUCT_XREF_IX2
ON PRODUCT_XREF (
    CANONICAL_ID,
    PRODUCT_CODE,
    CATALOG_CODE
);

The important part was not the exact suffix order.

The important part was recognizing that CANONICAL_ID should lead the candidate index for this query shape.

Bob then went further and considered whether PRODUCT_CODE and CATALOG_CODE could help make the access path covering.

Just as importantly, it did not guarantee that Db2 for i would use the new index.

It recommended validating the decision with Visual Explain and IBM i index-advisor information.

Its discussion around LISTAGG DISTINCT was also thoughtful.

An index ordered by:

CANONICAL_ID, PRODUCT_CODE, CATALOG_CODE

might help one aggregation’s ordering more than the other.

Bob correctly treated possible sort elimination as something to verify rather than promise.

There were still a few overly absolute statements about leading-key behaviour, but overall this was the cleanest combination of IBM i specificity, performance reasoning and restraint.

Test 4 score: 9.0 / 10

Test 5: REST + JSON — Great Architecture, Dangerous Details

Test 5 produced perhaps the biggest gap between high-level quality and implementation accuracy.

The requirement was realistic:

An interactive RPG application needs to send JSON to an external HTTPS endpoint.

The requirements included:

The Architecture Was Very Good

Bob proposed a clean asynchronous pattern:

Interactive RPG
    |
    | immediate POST
    |
    +-- 2xx --------> success
    |
    +-- 429 / 5xx --> retry queue
    |
    +-- other 4xx --> permanent failure

                     |
                     v

                 Batch worker
                     |
                     +-- exponential backoff
                     +-- retry
                     +-- dead letter

It also separated credentials from source, avoided logging Authorization values, considered TLS trust and created an audit trail.

Architecturally, I would happily use that as the basis of a design discussion.

Then I looked closely at the IBM i code.

The HTTP Function Was Used Incorrectly

The generated sample treated the chosen SYSTOOLS HTTP function as though it exposed a verbose table-function interface with response columns that were not actually part of that function.

It also mixed conventions from the older SYSTOOLS HTTP functions with the newer QSYS2 HTTP function family.

The generated code looked exactly like something an IBM i developer might expect to work.

But it would not work as written.

That distinction matters.

This was no longer a vague architecture discussion.

It was concrete code with concrete IBM i service names and signatures.

The benchmark specifically asked Bob not to invent those details if it was uncertain.

The Timeout Handling Was Also Too Loose

The requirement explicitly asked for a five-second timeout.

Bob proposed multiple possible mechanisms and configuration approaches, but some of them did not match the interface it was actually using.

Again, the overall design intent was good.

The exact IBM i implementation was not reliable enough to use without verification.

Another Platform-Level Miss

Bob also suggested CGI-oriented IBM i APIs as possible outbound HTTP client alternatives.

That is a good example of how an answer can contain real IBM i names and still be conceptually wrong.

The architecture remained strong.

The low-level platform implementation did not.

This was the weakest test overall, not because Bob could not design the integration, but because it became too confident about APIs and signatures that needed much tighter validation.

Test 5 score: 6.9 / 10

Test 6: IBM i DevOps and CI/CD

The final test was close to real work many IBM i teams are doing today.

The deployment hook:

Bob immediately found the central problem.

If you build into a temporary library, test those objects, delete them, and then compile everything again into the target library, you are not deploying the artifact you tested.

You are testing Build A and deploying Build B.

That is a release-integrity problem.

The Architecture Was Strong

Bob proposed a much better conceptual pipeline:

Git control
     |
     v
Build on DEV IBM i
     |
     v
Test temporary library
     |
     v
Package tested objects + SQL + source version
     |
     v
Immutable release
     |
     v
Approval
     |
     v
Restore/install on production

It also correctly recognized that:

Those are strong DevOps conclusions.

But the IBM i Implementation Needed Review

The response proposed a library-renaming command that does not exist as written.

The core idea — promote the already-tested artifact rather than rebuild it — was right.

The IBM i implementation was wrong.

It also missed a subtler Git integrity problem.

The hook receives the immutable commit SHA as $newrev, but if the build checks out the branch name rather than that immutable SHA, another push can move the branch before the checkout happens.

That creates a dangerous possibility:

release metadata says commit A
compiled objects actually came from commit B

A safer build should be tied directly to $newrev, ideally in an isolated worktree.

That was another example where Bob got the architecture right while experienced engineering review was still necessary.

Test 6 score: 8.5 / 10

The Pattern That Emerged

After six tests, I stopped thinking of the results as six independent scores.

A very consistent pattern had emerged.

Bob was strongest when the task involved:

Its reliability dropped when answers depended on:

That distinction matters.

Because those are two very different ways to use an AI development assistant.

Where I Would Use Bob

Based on these tests, I would be comfortable using Bob as an accelerator for several kinds of work.

Understanding Unfamiliar RPG

Give it a program and ask it to explain the flow, dependencies and suspicious areas before starting a change.

That can reduce the time it takes to orient yourself in an unfamiliar code path.

Modernization Planning

Ask it to convert legacy constructs, simplify control flow, identify dead code and suggest incremental modernization.

The key word is incremental.

Bob was generally better when it modernized structure than when it tried to reinterpret IBM i runtime behaviour.

SQL Review

It was good at recognizing inefficient row-at-a-time patterns and at thinking in terms of set-based operations.

The SQLRPGLE test demonstrated that clearly.

Db2 Query Analysis

Especially as a starting point for index discussions that are then validated using Visual Explain, actual statistics and the real workload.

Architecture Brainstorming

The REST retry architecture and DevOps release model were both strong despite implementation-level mistakes.

This may actually be one of the best uses of an AI coding assistant on IBM i: exploring several architecture options quickly before validating the platform-specific details.

Code Review Assistance

Bob found issues that would genuinely be useful during peer review.

It should not replace the reviewer.

But it can give the reviewer another set of eyes.

Where I Would Verify Before Acting

I would independently verify anything involving:

In other words:

Use AI to accelerate the investigation.
Do not outsource platform knowledge to it.

Is IBM Bob Useful for IBM i?

After these six tests, my answer is yes.

But perhaps not for the reason people expect.

The most useful thing Bob did was not generate RPG faster.

It helped reason about existing systems.

That is particularly relevant on IBM i, where the difficult work is often not creating a new 50-line program.

It is understanding a 20-year-old process, figuring out why it behaves the way it does, identifying where the real risk is, and deciding how far you can modernize without breaking the business.

Bob showed meaningful capability there.

At the same time, the benchmark also reinforced something I think is important as AI becomes more embedded in development workflows:

A confident explanation is not documentation.

A generated API name is not necessarily an API.

And a beautifully formatted IBM i answer can still contain one sentence that should never make it into production.

The experienced developer’s role does not disappear.

It changes.

Instead of spending all of our time producing the first draft of an explanation, query, modernization plan or architecture, we can increasingly spend that time validating assumptions, testing behaviour and making the platform-specific decisions that require context.

For me, that may be the more interesting future of AI on IBM i.

Not AI replacing IBM i expertise.

AI making that expertise more productive.

A Note on Reproducibility

This benchmark used synthetic source specifically created for testing.

No employer, customer or production source code was provided to IBM Bob.

The benchmark was designed so that the prompts, source, scorecard and hidden answer keys can be published separately and repeated by other IBM i professionals.

That part matters to me.

This article is not meant to be a claim that every version of Bob will always produce the same answers.

Models change. Tooling changes. Context changes.

What I wanted was a transparent snapshot:

Here is what I tested.
Here is what Bob got right.
Here is where it was wrong.
And here is where I would still rely on IBM i expertise.

That is a much more useful conversation than simply asking whether AI is “good” or “bad” for IBM i.

What I Want to Test Next

The baseline benchmark answered one question:

What does Bob do on the first attempt?

The next question may be even more interesting:

What happens when Bob is challenged on one of its own incorrect IBM i claims?

For example:

That would separate two very different limitations:

  1. Bob does not know the platform detail.
  2. Bob initially gets it wrong, but can recover when challenged.

That is probably the next experiment.

And if the results are interesting, I will share those too.

Comments

Share your thoughts, questions, or real-world IBM i experiences related to this article.