· ARJUN · WRITING
Map the Workflow Before Choosing an Agent Framework
A support assistant may need to ask a follow-up question, wait for a reply, and pick up where it left off. Use that workflow to shortlist Agent frameworks, then compare their results, latency, and cost on the same tasks.
Suppose you are building an assistant for a technical support team. A ticket arrives with one sentence: “I can’t log in after the upgrade.”
The first version might retrieve product documentation, put together a few troubleshooting steps, and pass them to a support specialist. Soon, the requirements grow: check the account status, confirm the versions before and after the upgrade, ask for missing information, and continue when the user replies several hours later.
The choice now involves conversation state, tool execution, and task recovery. Knowing that a framework can call a model tells you little about how it will handle that work.
Start with the ticket’s execution flow
For this hypothetical assistant, the process could look like this:
- Extract the known facts and identify what remains unconfirmed about the product, account, and version.
- Ask a specific question when information needed for the lookups is missing, save progress, and wait for a reply.
- Once the required information is available, query the account status and retrieve documentation for the relevant version.
- Use the returned evidence to produce troubleshooting advice, or hand the known facts and unresolved questions to a support specialist.
These steps involve different kinds of work. Application code can query an account by ID. A model can help decide what to search for in a user’s description. Waiting for a reply, tracking progress, and resuming the task require the runtime and storage to work together.
For a single retrieval-and-response request, a model SDK plus application code makes a useful baseline. When the model needs to choose tools over several turns, compare existing agent loops. Explicit branches, waits across requests, and recovery requirements justify a closer look at orchestration and persistence.
“Supports state storage” needs a more specific question: is that state held in memory, a database, or a managed service? Can the application retrieve it after a process restart? LangGraph checkpointers, for example, save graph state within a thread, but an in-memory implementation does not retain data across restarts. Storage must be configured for the deployment. See the official persistence documentation.
Let the requirements narrow the shortlist
Write down the team’s languages, deployment model, existing data interfaces, and maintenance responsibilities. These constraints often narrow the options before a feature comparison does.
LangChain has both Python and JavaScript implementations. A Java team can also run an agent written in another language as a separate service, though that adds deployment, communication, and debugging work. Whether that boundary is worthwhile depends on the existing system and the team’s ability to maintain it.
The following starting points can help organize a shortlist. Their capabilities overlap.
| Immediate requirement | Candidate to examine | What to check in the POC |
|---|---|---|
| Connect tools to a model and run a standard agent loop | LangChain | Fit of tool interfaces, context management, and middleware |
| Define branches, waits, and state transitions explicitly | LangGraph | How state is saved and tasks are paused and resumed |
| Connect processing steps around existing LlamaIndex data and retrieval components | LlamaIndex Workflows | Whether events and steps express the existing process clearly |
| Organize multiple agents around roles and tasks | CrewAI | Maintainability of the combination of Crews and Flows |
| Call models and business tools within a Spring application | Spring AI | Reuse of existing service interfaces and operations practices |
| Provide visual configuration and application management | Dify | Fit of custom logic, version management, and deployment options |
LangChain agents are built on LangGraph. Using a standard agent loop and writing a graph directly are choices at different levels of abstraction; production use does not automatically require switching from one to the other. CrewAI Flows also supports conditional routing and state persistence, beyond fixed role-based pipelines.
Check maintenance status as well. As of October 16, 2026, the official AutoGen repository identifies AutoGen as being in maintenance mode and recommends Microsoft Agent Framework for new users. Existing AutoGen projects need a separate assessment of migration work.
You do not need an implementation in every framework on the list. Compare one or two candidates that meet the key requirements with a simple baseline.
Take the POC far enough to expose the differences
For this assistant, retrieving documentation and generating an answer does not cover enough of the workflow to distinguish candidates. Include cases with complete information, a missing version, a lookup timeout, and a process restart while waiting for a reply.
The questions become concrete: after the restart, can the application recover the ticket, its completed lookups, and the person it is waiting for? If a tool fails, can a support specialist locate the stopping point in the execution record?
For a controlled comparison, keep the model, prompt, tool definitions, document snapshot, concurrency, and timeout settings as consistent as possible. If one implementation also changes the model, you cannot attribute the difference solely to the framework. When comparing the best separately tuned implementation of each candidate, record those model and configuration differences too.
The same task may produce different outcomes across runs. Report the sample size, repetitions, and failures. A single successful run establishes a working path; it says much less about sustained performance.
Define a passing result before scoring it
A ticket such as “I can’t log in after the upgrade” rarely has one exact reference answer. Evaluation can instead specify what must be established and what should happen when information is missing.
For example:
Case: Login failure with an unknown product version
Input: The user says, “I can't log in after the upgrade”
Known facts: Account identified; product version not provided
Expected: Ask for the version and set the task to waiting_for_user
Must not: Present a version-specific fix as a confirmed conclusionAsking for clarification passes this case. It does not mean the ticket has been resolved. A case with complete information would instead check whether the advice applies, the citations are valid, and the conclusion agrees with the account lookup.
Specify outcome criteria separately from execution paths. Where several paths are valid, exact tool-call order need not match a reference. Check ordering when the business process requires it—for example, an account lookup before a particular judgment. Traces help explain results and locate failures.
Read quality, latency, and cost together
The following definitions suit this example. They are evaluation design suggestions to adapt to the application.
| Metric | Definition for the support assistant |
|---|---|
| Task pass rate | Runs meeting every acceptance criterion for their case divided by all evaluation runs. Keep timeouts, errors, and incomplete runs in the count. Define in advance whether a correct clarification passes a case |
| Tool-use quality | Track tool selection, the business meaning of arguments, argument formatting, and execution results separately |
| Grounding and correctness | Check that claims have supporting evidence, then check whether that evidence applies to the product and version |
| Latency | Report the median (P50) and 95th percentile (P95) alongside timeout and incomplete-run counts. Distinguish first response, final result, and time waiting for the user. State whether failed runs are included in the percentiles |
| Cost | Record cost per run and calculate total cost across all evaluation runs divided by the number of passing runs, so failed runs contribute to cost per passing run |
A supported claim can still lead to a wrong recommendation. The assistant may faithfully repeat documentation for an older release that does not apply to the current version. Ragas Faithfulness measures whether claims in a response are supported by retrieved context. It cannot, on its own, establish correctness for the application.
Total cost also needs more detail than total tokens multiplied by one price. Input, output, and cached tokens may be billed differently, while tools and infrastructure can add charges. Record these separately before adding them up, and report evaluation costs separately. If no run passes, cost per passing run has no valid denominator; it should not be reported as zero.
Model calls, tool calls, and retries help explain latency, but fewer steps do not automatically mean better performance. Skipping an account lookup may save time and miss decisive information. Observable call events support this analysis; generated thinking text is not a reliable record of the model’s internal reasoning.
Make the scores traceable to evidence
Deterministic conditions, such as account IDs, output fields, and task status, are good candidates for code-based checks. An LLM can help assess whether advice is complete or an explanation is clear, using an explicit rubric and the relevant documentation and tool results.
A stronger judge model does not make its scores inherently reliable. Compare its decisions with a set of human-reviewed cases. Check whether fluent but incorrect answers pass, then adjust the criteria and examples. Retain human review for important conclusions. See LangSmith’s evaluation concepts and Arize’s guidance on calibrating judges.
There is no need to assemble hundreds of cases at the outset. Cover the main tasks and known failure branches first, then expand the dataset. Keep cases used repeatedly for prompt tuning separate from those reserved for the final comparison, reducing the risk of favoring familiar examples.
For this assistant, a simple implementation remains a useful comparison while the job is retrieval followed by advice. If waiting for replies and continuing across process restarts are required, the POC should demonstrate that path. The decision record should identify candidate versions, execution conditions, tasks that passed, and unresolved limitations. When requirements change, the team can revisit the choice using that record.