Formalizing Foundational Agent/Flow Testing

TL;DR

• Process starts with defining a new foundational agent or flow, then local testing to check it performs as expected
• Once initial testing works, outcomes get formalized and boundaries get fine-tuned
• Example: MR review flow needs token limits per step to control context retention
• May require a mandatory human-in-the-loop step (e.g., always requiring a human step at step two depending on output)
• Tools are used to write tests that measure the output against these formalized requirements

Feeling the Pressure

I feel like there’s quite a bit of pressure. But here we go.

Defining and Testing New Flows

Essentially, when we have a new foundational agent or flow, what we do is define it. We give it a definition, then do local testing to make sure it actually performs as we think it should. This is all fairly loosey-goosey at first, but once we get there, we start to formalize the outcomes we require and fine-tune the boundaries.

Example: MR Review Flow

For instance, we have a flow whose purpose is to review MRs. Things we may want to look out for: for any given step, the agent at that step should only use or retain context up to a certain number of terms. So we have a token limit we can measure. We may also have a requirement that the next step must be a human step. Depending on the output, we may always require a human step at step two—essentially a human-in-the-loop decision that must have happened at that point.

Writing Tests

So we have these tools, and what we do is write tests to measure the output.