Engineering a Semi-Deterministic AI Dark Factory

Most discussions about AI-assisted development start too late. They start with a coding agent and a prompt.

The difficult part is not producing code. It is converting an imprecise business request into a requirement that is consistent with the existing system, bounded enough to implement, and precise enough to test. If that input is weak, adding more agents only automates ambiguity.

We split the problem into two systems:

1. The Requirements Agent turns business intent into a repository-grounded, testable specification.
2. The Dark Factory turns the approved specification into reviewed code, validates it in a real environment, and hands the final decision back to engineers.

The Requirements Agent lives in TARS, our engineering workbench for conversational agents. TARS is the intake and refinement layer. The Dark Factory is the execution layer.

The important property is not autonomy. It is controlled progression through explicit gates.

This is no longer only an architecture diagram. In its first complete autonomous execution, the platform spent five hours implementing a feature for itself, from an approved requirement to a deployed cloud test environment. It recovered from transient network errors without human intervention and completed the entire automated SDLC section without an escalation.

The End-to-End Flow

#

flowchart LR
    B[Business requirement] --> R[Requirements Agent]
    R --> Q{Gaps or blockers?}
    Q -->|Yes| A[Closed Q&A]
    A --> R
    Q -->|No| S[Executable specification]
    S --> H1[Engineer review]
    H1 --> P[Planner Agent]
    P --> C[Serial task loop]
    C --> E[Deploy to UAT AWS account]
    E --> U[Independent UAT Agent]
    U -->|Fail| C
    U -->|Pass| D[Documentation Agent]
    D --> H2[Final MR for humans]

Humans approve the specification before implementation and retain the final merge. Everything between those gates is automated, observable, and recoverable.

The First Five-Hour Run

#

The first successful run implemented a redesign of the platform’s own agent experience:

• A reusable drawer for starting agent runs from the sidebar.
• A read-only catalog containing every available agent.
• A panel exposing the selected agent’s system prompt.
• A full-width run detail page with the old prompt split-view removed.
• Backend contracts, rate limiting, and the tests required by the repository guardrails.

The Planner produced five ordered implementation tasks. The factory processed one task at a time, opened and reviewed five task MRs, merged them into one integration branch, and produced the final handoff. The assembled change touched 19 Go and React files, with approximately 2,500 additions, including backend, middleware, UI, contract, and test changes.

The result was then built and deployed into the AWS test account. Application tests, static checks, image builds, infrastructure planning, rollout, and health verification all completed as part of the delivery path.

The feature is interesting, but the five-hour control loop is the more important result. Every agent response had to remain compatible with the next state transition. One malformed plan, missing MR reference, stale review verdict, duplicate dispatch, or incorrectly classified network failure could have stopped the run.

That is why the engineering objective is semi-determinism. We cannot make an LLM deterministic, but we can make the set of valid outputs, transitions, side effects, and recovery paths deterministic around it.

The Requirements Agent Is Not a Document Generator

#

The Requirements Agent is a chatbot, but its job is not to rewrite a ticket in better English. It has read access to the target repository, or to the relevant repositories when the target is a GitLab group.

Before asking questions, it reconstructs the current state:

• Reads the root and nested CLAUDE.md files.
• Inspects source code, configuration, API contracts, tests, migrations, and delivery files.
• Expands groups and subgroups, then ranks repositories by relevance to the objective.
• Identifies behavior that is already implemented, partially implemented, missing, or unclear.
• Prefers the smallest change that satisfies the objective and preserves unrelated behavior.

This produces an As-Is Baseline. Questions are then grounded in the application that exists, not in a hypothetical greenfield design.

For example, if pagination already uses offset and limit, the agent does not ask which pagination style to use. It records that convention as a guardrail. If a retry endpoint already exists but the UI does not expose it, the requirement describes the missing UI and authorization behavior instead of proposing a second API.

flowchart TD
    I[Business intent + repository] --> X[Collect repository context]
    X --> ASIS[Build As-Is Baseline]
    ASIS --> GAP[Classify gaps and blockers]
    GAP --> DEC{Material ambiguity remains?}
    DEC -->|Yes| QA[Emit closed questions]
    QA --> ANS[User selects answers]
    ANS --> ASIS
    DEC -->|No| DOC[Emit complete specification]
    DOC --> READY[Readiness: open items = 0]

Gaps and Blockers Are Different

#

A gap is a difference between the current system and the requested outcome. The agent can document it and propose the minimum-impact change.

A blocker is a decision that cannot safely be inferred from code or policy. Examples include which roles may retry a failed run, whether retry must be idempotent, or whether a data migration may be destructive.

The agent must never convert a blocker into an assumption just to finish the document.

A Protocol, Not Just a Prompt

#

The refinement loop is enforced by tools and application state. The agent emits structured events such as:

emit_commentary          explain the current analysis
emit_document_section    update one or more requirement sections
emit_questions           present closed questions and possible answers
emit_readiness           declare the number of unresolved items
retrieve_context         recover older persisted context when needed

The critical invariant is simple:

open_items_count > 0  =>  emit_questions is mandatory
open_items_count = 0  =>  the specification may become ready

If the model emits readiness while open items remain, the application rejects that transition and asks the agent to continue. The conversation cannot complete merely because the model sounds confident.

Closed questions also reduce interpretation drift. Instead of asking, “How should retries work?”, the agent presents concrete choices such as:

• Retry creates a new run linked to the failed run.
• Retry reuses the original run identifier.
• Retry behavior already defined by an existing API remains unchanged.

The user may still provide a custom answer, but each open decision is explicit, persisted, and traceable to the resulting requirement.

Keeping Long Sessions Lossless

#

Repository analysis and requirement documents can become large. Sending the entire transcript back to the model on every turn eventually exhausts the context window.

The full conversation and document therefore remain persisted as the source of truth. Only the model-facing copy is compacted against a token budget. The agent can retrieve an older message or document section with retrieve_context when it needs exact details.

This separates two concerns:

• Persistence must be lossless. No requirement detail is silently truncated.
• Inference context must be bounded. Old tool exchanges do not need to be replayed forever.

Document updates are batched where possible, and emit_readiness is terminal once the readiness invariant is satisfied. This avoids a chain of unnecessary model calls after the document is complete.

Application Architecture

#

The user sees the TARS React chat application. Django owns authentication, conversations, documents, questions, permissions, and job state. Long-running turns are queued rather than held inside an HTTP request. The agent itself runs in Amazon Bedrock AgentCore with repository-scoped credentials.

flowchart LR
    UI[React chat and document UI] --> API[Django API]
    API --> DB[(PostgreSQL)]
    API --> QUEUE[Worker queue]
    QUEUE --> AC[Bedrock AgentCore runtime]
    AC --> GIT[Git repository, read only]
    AC --> LLM[Foundation model]
    AC --> TOOLS[Structured event tools]
    TOOLS --> API
    API --> STREAM[Live event stream]
    STREAM --> UI

The separation matters. AgentCore performs analysis, but Django owns durable workflow state. A model response is input to validate, not authority to mutate arbitrary application state.

What a Ready Requirement Looks Like

#

A complete requirement can be long, but its shape should be predictable. Here is a shortened example:

# Retry failed runs from the UI

## Objective
Allow an operator to retry a failed run without leaving the run detail page.

## As-Is Baseline
- The backend already exposes POST /api/runs/{id}/retry.
- The endpoint creates a new run linked through retried_from_id.
- The React detail page displays failures but has no retry action.

## Scope
- Add a Retry action for failed runs.
- Show the new run and navigate to its detail page.
- Preserve the existing backend retry semantics.

## Non-Goals
- No retry for queued, running, or successful runs.
- No change to the scheduler or retention policy.

## Functional Requirements
- FR-1: Only operators may see and invoke Retry.
- FR-2: The UI must prevent duplicate submission while the request is active.
- FR-3: API errors must leave the original run visible and show a recoverable error.

## Acceptance Criteria
- AC-1: Given a failed run and an operator, when Retry is selected,
  then a new linked run is created and its detail page is opened.
- AC-2: Given a running run, when its page is opened,
  then no Retry action is rendered.

## Guardrails
- Reuse the existing retry API and authorization policy.
- List APIs remain limit-offset paginated.
- Do not change unrelated run states.

## UAT
- Verify AC-1 and AC-2 through the browser.
- Verify keyboard operation and the duplicate-click guard.

## Open Items
None.

The most important line is the last one. “None” is not prose generated optimistically. It is the result of closing every question in the refinement loop.

Entering the Dark Factory

#

An engineer reviews the requirement before it enters the factory. This gate checks product intent, risk, feasibility, and whether the declared constraints match team policy.

After approval, the specification becomes immutable input for the run. Agents may report that it is inconsistent or impossible, but they may not quietly reinterpret it.

1. Planning Into Ordered Atomic Tasks

#

The Planner Agent reads the complete specification and emits a strictly ordered list of bounded tasks. Each task includes:

• A single objective and explicit non-goals.
• The files or interfaces likely to change.
• The acceptance criteria it covers.
• Expected tests and verification commands.
• Applicable engineering guardrails.
• A size small enough for one meaningful review.

The factory deliberately does not execute a dynamic task DAG in parallel. Parallel coders look attractive in a demo, but overlapping assumptions, branch conflicts, and non-deterministic completion order make recovery harder.

The task array is the execution order. Exactly one implementation task is active at a time.

2. Coding, Unit Tests, and a Task MR

#

For every task, the Coder Agent:

1. Creates a task branch from the current integration branch.
2. Reads the task, specification, repository instructions, and relevant code.
3. Implements only the declared scope.
4. Adds or updates unit tests.
5. Runs the repository’s static checks and tests.
6. Pushes the branch and opens a merge request into the integration branch.

The MR is the coder’s product. The coder cannot approve or merge it.

3. Two Independent Reviews

#

The same task MR is inspected by two agents with read-only credentials:

• Code Review Agent: correctness, contracts, regressions, maintainability, tests, and repository conventions.
• Security Agent: authentication, authorization, injection, secrets, SSRF, unsafe deserialization, dependencies, and infrastructure permissions.

Both verdicts are bound to the current commit SHA. If the coder pushes another commit, previous approvals are invalidated.

flowchart TD
    T[Next ordered task] --> C[Coder implements + unit tests]
    C --> MR[Task MR to integration branch]
    MR --> CR[Code Review Agent]
    MR --> SR[Security Agent]
    CR --> G{Both pass at current SHA?}
    SR --> G
    G -->|No, blocking or major findings| RW[Coder rework mode]
    RW --> C
    G -->|Yes| M[Control plane merges task MR]
    M --> N{Tasks remaining?}
    N -->|Yes| T
    N -->|No| ENV[Promote integration branch to UAT]

In rework mode, the coder receives the original task plus the blocking and major findings. It fixes the same task branch, reruns tests, and updates the existing MR. Both reviewers then start again. Minor observations may remain for humans, but unresolved blocking or major findings cannot pass the gate.

The control plane, not an agent, validates canonical MR metadata and performs the merge. This prevents an agent from claiming that an unrelated MR passed review.

UAT Without Looking at the Code

#

Once all task MRs are merged, the integration result is promoted to a branch that deploys to a dedicated AWS acceptance account.

The UAT Agent has already derived its scenarios from the approved requirement. It did so without seeing the implementation. This independence is intentional: a test author that reads the code is likely to reproduce the implementation’s assumptions.

The UAT Agent uses Playwright against the deployed application and verifies externally observable behavior:

• User journeys and role boundaries.
• Given/When/Then acceptance criteria.
• Error and recovery paths.
• Browser state, navigation, and accessibility-relevant interactions.
• Regression scenarios identified in the requirement.

If UAT fails, the control plane creates a bounded rework task containing the failed scenario, evidence, and expected behavior. That task enters the same coder, review, merge, deploy, and UAT loop. The UAT Agent does not patch code itself.

If UAT passes, the software is functionally ready, but it is still not automatically released. The factory prepares a final definitive merge request for humans.

Documentation Is Part of the Product

#

The last agent updates operational and developer documentation, including CLAUDE.md. These files are not generic AI instructions stored somewhere outside the project. They are versioned repository knowledge used by engineers and by the next Requirements, Planner, Coder, and Review agents.

For a large repository, the documentation is layered by directory, following the Claude Code large codebase guidance:

repository/
  CLAUDE.md                         # repository-wide architecture and rules
  .claude/
    settings.json                   # permissions and excluded generated paths
    rules/                          # path-scoped cross-cutting rules
  backend/
    CLAUDE.md                       # Django commands, API and migration conventions
    .claude/skills/
      api-testing/SKILL.md
    src/
  frontend/
    CLAUDE.md                       # React commands, routing and component conventions
    .claude/skills/
      component-testing/SKILL.md
    src/
  infrastructure/
    CLAUDE.md                       # Terraform, environments and deployment constraints

The root file contains only rules that apply everywhere. A nested file adds stack-specific commands, module maps, and non-obvious constraints. It does not duplicate the root.

The Documentation Agent updates these files in the same change as the behavior they describe. It verifies commands and paths before writing them. This closes the loop:

flowchart LR
    CODE[Changed code] --> DOC[Updated repository knowledge]
    DOC --> NEXT[Next Requirements analysis]
    NEXT --> PLAN[Better grounded plan]
    PLAN --> CODE

Without this step, every future agent starts from stale assumptions and pays the discovery cost again.

Engineering Semi-Determinism

#

An agentic workflow is not deterministic in the traditional sense. The same prompt can produce different reasoning, different tool sequences, and differently worded results. A five-hour workflow also multiplies the probability that at least one response violates its expected contract.

The solution is a deterministic shell around stochastic workers:

flowchart LR
    A[Agent output] --> P[Parse strict contract]
    P -->|Invalid| R[Bounded retry or escalation]
    P -->|Valid| V[Validate invariants]
    V -->|Invalid| R
    V -->|Valid| DB[(Persisted state)]
    DB --> RED[Pure workflow reducer]
    RED --> RES[Reserve action by dedup key]
    RES --> FX[Execute one side effect]
    FX --> OBS[Observe canonical external state]
    OBS --> DB

Structured Output Is an Untrusted Proposal

#

Each workflow agent has a narrow output contract. A Planner returns an ordered tasks array. A Coder returns an MR reference. Reviewers return verdicts and findings.

The control plane parses and validates those results before persisting them. Missing and empty fields are deliberately different states. An absent tasks field is malformed output. An empty task list with a concrete explanation is a valid planning-blocked result. Duplicate task keys are rejected.

For a Coder result, the only useful claim is the MR IID. The control plane does not trust the agent-supplied branch, URL, target, or SHA. It fetches canonical MR metadata from GitLab using the run-scoped client and verifies that the MR belongs to the expected project and branch topology.

This distinction is fundamental:

agent output = proposal
validated database state = workflow truth
external API observation = side-effect truth

A Pure, Level-Triggered Reducer

#

The next action is computed from a database snapshot by a pure reducer. Given the same specification, ordered tasks, open MRs, and reviews at the current SHA, it produces the same action list.

It does not encode progress as a fragile chain of callbacks. On every reconciliation tick it asks what action is warranted by current state:

• If no task is active, release the first non-merged ready task.
• If a coder MR is open, run the fixed reviewer set.
• If both reviewers passed at the current SHA, allow the deterministic merge gate.
• If a new SHA appears, run both reviewers again.
• If all tasks are merged, create or recover the final handoff MR.

Actions use a deduplication key and are reserved before an agent starts. Reconciliation can therefore repeat after a process restart or temporary outage without starting the same logical action twice.

Serial execution is part of this design. It sacrifices theoretical throughput to eliminate branch races, overlapping task assumptions, and non-deterministic dependency scheduling.

Reviews Are Facts About a Commit, Not an MR

#

A review verdict is stored against the MR’s current head SHA. A pass is not a sticky property of the merge request.

If rework pushes a commit, both previous verdicts become irrelevant. Until Code Review and Security Review pass the new SHA, the merge gate returns hold. A fail or unparseable verdict returns blocked. Only two passes at the live head return approve.

The reviewer list is defined once and used by both reviewer dispatch and merge evaluation. This prevents configuration drift where the system starts one set of reviews but waits for another.

Transient Failure Must Not Become Business Failure

#

Long agent runs cross several unreliable boundaries: AgentCore invocations, network proxies, Git hosting, model APIs, and cloud control planes. A transient transport error must not flip a live workflow into a terminal state.

The poller therefore distinguishes:

• Transport error: keep the run active and retry on the next reconciliation tick.
• Running: persist progress only when it advances.
• Unknown job: tolerate several consecutive session-affinity misses before declaring the runtime lost.
• Lost runtime: classify it separately from agent failure so the control plane can recover it.
• Agent error: persist a genuine terminal failure and apply the stage’s bounded recovery policy.

Git operations follow the same rule. Branch and final-MR creation are idempotent. A transient merge or API failure is retried from observed GitLab state, not assumed to have failed before or after the side effect.

Coder review failures enter a bounded rework loop on the same MR. Merge conflicts enter a bounded conflict-resolution loop. Only after those budgets are exhausted does the platform create an escalation for a human.

The first five-hour run encountered transient network errors, but none changed the logical workflow state. Reconciliation observed the system again and continued. That recovery behavior, combined with five hours of contract-valid agent output, is the result that matters most.

The Control Plane Is the Real Product

#

The architecture works because agents do not own the workflow. A deterministic control plane constrains them, observes their effects, and decides what may happen next.

It is responsible for:

• Persisting specifications, tasks, attempts, findings, and evidence.
• Allowing only valid state transitions.
• Releasing exactly one task at a time.
• Binding review verdicts to commit SHAs.
• Encrypting run-scoped credentials and routing least-privilege identities.
• Retrying bounded transient failures and escalating terminal ones.
• Merging task MRs only after both review gates pass.
• Keeping the final merge a human action.

This is the main lesson from building a Dark Factory: reliability does not come from a larger prompt or a more capable model. It comes from making model outputs untrusted proposals inside a state machine with explicit invariants. The agents remain probabilistic; the production line becomes semi-deterministic.

What Humans Still Own

#

The process automates analysis and execution, not accountability.

Humans still own:

• Product intent and business priority.
• Approval of the final requirement.
• Exceptions to architecture and security policy.
• Risk acceptance for non-blocking findings.
• The definitive merge and production release.

That is the boundary I want. Agents do the repetitive reading, decomposition, implementation, testing, and evidence collection. Engineers make the decisions that carry organizational and production responsibility.

Final Thoughts

#

A Dark Factory should not be a room full of agents improvising against the same repository. It should be a narrow production line:

intent
  -> repository-grounded requirement
  -> closed decisions
  -> engineer approval
  -> ordered tasks
  -> code + unit tests
  -> code and security gates
  -> deployed black-box UAT
  -> current documentation
  -> human merge

The Requirements Agent gives the factory high-quality input. The control plane gives probabilistic agents deterministic boundaries. Independent review and UAT provide evidence. Layered CLAUDE.md files feed what was learned back into the next cycle.

The result is not autonomous software delivery. It is software delivery where ambiguity, authority, and risk are made explicit before code reaches production.

Want to go deeper on agentic software delivery, platform engineering, or AWS architecture? I offer 1:1 coaching sessions tailored to your background and goals. Check out the coaching page.