A research and implementation guide to autonomous external web application and API security testing

Autonomous penetration testing is evolving from conventional vulnerability scanning toward systems that can discover an application’s attack surface, form hypotheses about potential weaknesses, select appropriate tests, interpret results, adapt their next actions, and validate suspected vulnerabilities with reproducible evidence.

Yet a critical distinction is often overlooked: an AI-powered security tool is not necessarily an agentic penetration tester. A scanner may use a large language model (LLM) to explain findings or generate payloads while still following a fixed workflow. A genuinely agentic system uses observations to decide what to do next, changes its plan when evidence contradicts its assumptions, and recognizes when it should stop.

This article develops an implementation-oriented architecture for an external black-box web application penetration-testing platform. It covers system architecture, reconnaissance, browser and API discovery, authentication, authorization testing, stateful protocols, playbook design, independent validation, evidence storage, prompt-injection resistance, execution isolation, resource governance, benchmarking, and phased implementation.

It also examines the trade-offs between building a platform from scratch and adapting existing security tools. The objective is not merely to assemble scanners behind an AI interface. It is to build a system whose planning is adaptive, execution is constrained, findings are reproducible, and performance is measurable against known ground truth.

1. Introduction: From vulnerability scanners to autonomous pentesters

1.1 How traditional automation works

Traditional security automation executes a predefined sequence of operations.

For example:

1. Discover accessible web endpoints.
2. Crawl the application.
3. Apply known vulnerability checks.
4. Compare responses against signatures or expected behavior.
5. Produce a report.

This approach is valuable because it is repeatable, relatively predictable, and easier to benchmark. However, predefined workflows have difficulty reasoning about application-specific behavior.

A scanner may discover an invoice endpoint and test its parameters, yet fail to understand that the invoice belongs to a particular user and should not be accessible to another user.

The problem is not necessarily a missing payload. It may be a missing understanding of the application’s authorization model.

1.2 AI-assisted testing

An AI-assisted scanner adds model capabilities to an existing workflow. An LLM might explain a vulnerability, classify a response, generate a test variation, or recommend the next test.

The workflow can still remain fundamentally static.

For example, the system may run the same collection of checks regardless of what it discovers. AI improves individual tasks without controlling the overall investigative process.

1.3 Agentic penetration testing

An agentic system maintains a working model of the target, selects actions based on observations, evaluates outcomes, and updates its plan.

Consider a hypothetical authorized test:

• The crawler discovers a customer invoice endpoint.
• The planner observes that the response contains an object identifier.
• It hypothesizes that access may be based on possession of the identifier rather than ownership.
• It checks the engagement’s authorized test identities and expected permissions.
• It requests the same resource in the context of a second test account.
• A deterministic validator compares the observed behavior with the permission model.
• If unauthorized disclosure is demonstrated, the system records evidence and stops expanding that test beyond the authorized boundary.

If the second account is legitimately entitled to the resource, the hypothesis should be rejected or refined. If authentication expires, the system should repair or re-establish its authorized session rather than misclassify the result.

The agent’s behavior changes because of evidence.

1.4 A practical definition of genuine agency

For this project, a system should be considered agentic when it demonstrates the following capabilities:

An LLM is one possible component of this design, not a substitute for the entire system.

2. Define the problem before choosing the technology

The initial product should focus on external black-box assessments of web applications and APIs.

Assume the system receives:

• An authorized hostname, URL, or asset inventory.
• No source code or private infrastructure documentation by default.
• Optional test credentials representing different roles or tenants.
• Optional OpenAPI specifications, sitemaps, HAR files, or other supplied evidence.
• A controlled environment for browsers, scanners, and custom test modules.

The platform should not assume that discovering a hostname authorizes testing it, that a login page implies valid credentials are available, or that an HTTP response reveals the application’s complete internal architecture.

2.1 Distinguish evidence from assumptions

A mature framework must maintain explicit epistemic states.

A finding’s severity must be independent of its confidence. A potentially critical issue with weak evidence should not be represented as a confirmed critical vulnerability.

Similarly, the absence of detected vulnerabilities does not prove that an application is secure.

3. Reference architecture

The proposed architecture separates reasoning, authorization, execution, evidence, and validation.

3.1 Logical system diagram

The arrows represent logical relationships, not permission to bypass the enforcement boundary. Actual network and process controls must ensure that workers cannot circumvent the gateway.

3.2 The main components

The architecture’s most important principle is that the planner proposes actions, but does not grant itself permission to execute them.

4. The engagement control plane and scope model

Scope should be a machine-enforced policy, not merely text in a prompt.

4.1 Engagement manifest

An engagement manifest should identify:

• Engagement ID and version.
• Authorized domains, hostnames, IP addresses, ports, and protocols.
• Explicit exclusions and third-party dependencies.
• Permitted test classes and impact levels.
• Testing windows and rate limits.
• Maximum request counts, concurrency, runtime, and cost.
• Authorized test identities and credential references.
• Human approval requirements.
• Data retention and evidence-handling policies.
• Emergency-stop and recovery behavior.

A conceptual manifest might look like this:

engagement_id: engagement-2026-001
scope_version: 3

targets:
  - hostname: app.example.test
    protocols: [https]
    ports: [443]

excluded:
  - hostname: payments.example-provider.test

testing:
  mode: authorized_black_box
  permitted_classes:
    - passive_discovery
    - bounded_http_testing
    - authenticated_authorization_testing

limits:
  max_concurrent_requests: 4
  max_runtime_minutes: 60
  require_approval_for_state_changes: true

This is an illustrative schema, not a universal safe configuration. Real limits must be adapted to the application, authorization agreement, and environment.

4.2 Scope enforcement at execution time

Every action must be checked twice conceptually:

1. When the planner proposes it.
2. Immediately before the execution system performs it.

The second check is essential because scope, DNS resolution, session state, and policy may change between planning and execution.

The enforcement layer should evaluate the engagement, destination, protocol, port, operation, test class, resource budget, and approval requirements.

It should reject unregistered tools, invalid parameters, expired authorizations, and ambiguous destinations.

4.3 DNS, redirects, and network boundaries

A hostname may resolve to different addresses over time. A target may also redirect to an identity provider, payment processor, CDN, or unrelated third-party service.

The system must not assume that these destinations inherit the original authorization.

Required controls include:

• Validating resolved IPv4 and IPv6 destinations.
• Handling CNAME chains and DNS changes.
• Checking each redirect destination before following it.
• Restricting protocols and ports.
• Preventing unauthorized direct network connections from workers.
• Handling DNS rebinding and time-of-check/time-of-use risks.
• Defining how proxies, browser networking, and tool-specific connection behavior are constrained.

A proxy is not sufficient if a worker can open its own unrestricted socket. Where practical, enforce egress through network policy, firewall rules, an allowlisting proxy, or an equivalent independent mechanism.

Negative tests must attempt out-of-scope redirects, unauthorized connections, alternate ports, and direct egress. The expected result is rejection by the enforcement layer.

5. The agent planner: reasoning without unrestricted authority

5.1 Maintain explicit state

The planner needs a structured representation of what it knows.

At minimum, maintain:

• Discovered assets and endpoints.
• Authentication and authorization contexts.
• Application states and transitions.
• Observed parameters and response characteristics.
• Existing hypotheses and supporting evidence.
• Executed tests and their outcomes.
• Coverage gaps and blocked prerequisites.
• Remaining budgets and applicable policy constraints.

Without persistent state, the agent may repeat tests, lose track of authentication context, or incorrectly treat a previously tested endpoint as unexplored.

5.2 Planning as a feedback loop

A useful planning cycle is:

1. Read the current engagement state.
2. Identify an unresolved security question.
3. Generate one or more test hypotheses.
4. Estimate prerequisites, expected information value, cost, and impact.
5. Select an action from the approved capability registry.
6. Submit the action to the policy gateway.
7. Execute only if authorized.
8. Capture evidence and update state.
9. Validate or refine the hypothesis.
10. Continue, request approval, or stop.

This cycle should be bounded by execution limits and explicit stopping conditions.

5.3 How should the planner select tests?

A conceptual prioritization score could be:

This is a conceptual heuristic, not a validated universal scoring formula. It should never override hard authorization rules or resource limits.

For example, if two endpoints implement similar functionality but one exposes a more sensitive object type, the planner may prioritize the higher-risk endpoint. However, it must still satisfy the test prerequisites and impact restrictions.

5.4 One agent or several?

Multiple agents are not automatically better.

A single orchestrator with specialized modules is usually easier to debug, benchmark, and secure. A multi-agent architecture can be justified when tasks require different models, independent research, or parallel execution.

Start with a single planner and deterministic modules. Add specialized agents only when measurable improvements justify the additional coordination complexity.

The policy gateway should remain an independently enforced service regardless of how many agents are used.

6. Dynamic reconnaissance and attack-surface mapping

A web application is not just a collection of URLs. It is a collection of reachable states, actions, identities, objects, and transitions.

6.1 External asset discovery

A discovery pipeline may combine:

• Passive DNS and certificate-transparency data.
• Authorized subdomain and DNS discovery.
• HTTP and HTTPS endpoint checks.
• TLS and service configuration observations.
• Public API and version discovery.
• Publicly accessible documentation and configuration artifacts.
• Supplied asset inventories and API specifications.

Candidate assets must be validated against the engagement manifest before active probing.

The discovery engine should record provenance. For example, an asset may have been observed in a certificate, supplied by the customer, discovered through a link, or inferred from JavaScript.

These sources do not all provide equal confidence that the asset is part of the intended application.

6.2 Browser-driven discovery

Modern applications often hide important functionality behind client-side routing and asynchronous requests.

A browser automation framework such as Playwright can observe HTTP traffic, Fetch/XHR requests, redirects, and WebSocket activity. Its network APIs also support controlled request interception and response handling.

A browser discovery adapter should capture:

• Client-side route transitions.
• Forms and form submissions.
• Fetch and XHR requests.
• API paths, methods, parameters, and content types.
• Authentication changes.
• Relevant DOM snapshots.
• WebSocket connection and message metadata.
• Errors and failed transitions.
• Links and resources discovered in JavaScript.

Do not treat a discovered route as proof that the route has been fully tested.

6.3 Model the application as a graph

Represent the application as a graph:

• Nodes: routes, API endpoints, identities, objects, sessions, and workflow states.
• Edges: requests, redirects, form submissions, authentication transitions, and business actions.
• Attributes: permissions, prerequisites, evidence, and confidence.

For example, an order-management workflow may contain:

Each transition has prerequisites and an expected result. Testing the details endpoint without the relevant authenticated state may provide little useful information.

6.4 Coverage must describe more than discovery

Track separate milestones:

• Discovered.
• In scope.
• Reachable.
• Authenticated.
• Exercised.
• Security-tested.
• Validated.
• Blocked or inconclusive.

A route may be discovered but never reached. It may be reached anonymously but not as an administrator. It may be tested for injection but not for authorization.

A coverage report must preserve these distinctions.

6.5 Architecture inference

The system can infer aspects of the external architecture from HTTP behavior, DNS, TLS, JavaScript, API traffic, and supplied documents.

However, it should not claim to know internal databases, private IAM policies, cloud permissions, or internal network topology without evidence.

The architecture model must separate observed facts from hypotheses.

7. Authentication, sessions, and authorization testing

Authentication and authorization are separate security problems.

Authentication establishes who or what is interacting with the application. Authorization determines which objects, properties, and actions that identity may access.

A useful testing framework must model both.

7.1 Identity and permission matrix

Represent authorized test identities and their expected permissions.

These are examples, not universal authorization rules. Some resources are public, some roles share access, and some applications implement delegated permissions.

The framework must establish expected behavior before declaring a failure.

7.2 Session lifecycle

The system must understand:

• Login and logout.
• Cookie rotation.
• Session expiration and revocation.
• CSRF tokens.
• Refresh tokens.
• MFA and step-up authentication.
• External identity providers.
• Re-authentication flows.
• Concurrent sessions.
• Differences between browser sessions and API tokens.

If a session expires, the test should enter a known failure state and recover only through an authorized procedure. A redirect to a login page is not proof of a vulnerability.

7.3 BOLA, IDOR, BOPLA, and BFLA

Authorization testing is one of the most important requirements for an agentic web/API pentester.

The OWASP API Security Top 10:2023 identifies Broken Object Level Authorization, Broken Object Property Level Authorization, and Broken Function Level Authorization as distinct risks. See the OWASP API Security Top 10 guidance.

• BOLA/IDOR: A user can access an object they should not be permitted to access.
• BOPLA: A user can read or modify object properties outside their permissions.
• BFLA: A user can invoke functionality reserved for another role or privilege level.
• Tenant isolation failure: One tenant can access another tenant’s protected resources.

A strong validation process should:

1. Identify the object and its authorized owner or access policy.
2. Establish the expected permission model.
3. Execute a baseline request under an authorized context.
4. Compare behavior under another permitted test identity or role.
5. Evaluate whether the response exposes protected data or performs an unauthorized action.
6. Check alternative explanations, including shared access, public resources, and caching.
7. Preserve the evidence required to reproduce the issue.

Changing an identifier and observing a different response is not, by itself, proof of a vulnerability.

8. Stateful protocols and business workflows

Some security failures only become visible after a sequence of valid application interactions.

Examples include:

• A WebSocket message sent after an authenticated handshake.
• A transaction that moves through multiple approval stages.
• A workflow where a temporary token is issued in one step and used in another.
• An asynchronous job that completes after the initiating HTTP request.
• A GraphQL mutation whose effect depends on previously established application state.

A framework that treats every request independently will miss some of these behaviors.

8.1 Protocol adapters

Use protocol-specific adapters for WebSockets, Server-Sent Events, GraphQL, and other stateful interfaces.

Each adapter should track:

• Connection and handshake state.
• Authentication context.
• Message sequence.
• Correlation identifiers.
• Relevant workflow prerequisites.
• Partial responses and timeouts.
• Session expiration.
• Retry and reconnection behavior.
• Cancellation and cleanup.

8.2 Asynchronous workflows

A successful request may return a job identifier before the actual operation completes.

The framework must distinguish request acceptance from operation completion. It should use bounded polling or approved callbacks, preserve correlation IDs, and account for eventual consistency.

Retries require particular care: repeating a request can create duplicate orders, send messages, or trigger other side effects.

8.3 Business-logic testing

Business-logic flaws often require a model of intended behavior rather than a generic payload.

Examples include:

• Performing a workflow transition without the required prerequisite.
• Reusing an operation that should be single-use.
• Accessing an object after permissions change.
• Violating an expected relationship between ownership and action.
• Repeating a sensitive business action beyond an authorized limit.

The framework should encode business expectations as testable invariants. Destructive operations and financial transactions must be excluded unless explicitly authorized, and should be tested using synthetic data or dedicated test environments whenever possible.

9. Vulnerability coverage: OWASP web applications and APIs

A useful framework should use versioned security taxonomies instead of maintaining an unstructured list of tests.

9.1 OWASP Top 10:2025

The OWASP Top 10:2025 contains these categories:

The taxonomy is a classification framework, not a promise that all categories can be conclusively tested from the public interface.

For example, an external black-box test cannot generally establish whether internal security alerts were generated unless the engagement provides a way to observe the logging or monitoring system.

9.2 OWASP API Security Top 10:2023

The API-specific taxonomy adds useful focus:

The OWASP API Security project provides the relevant guidance.

9.3 Maintain a testability classification

For each category, document whether it is:

• Externally testable.
• Conditionally testable.
• Dependent on additional access or telemetry.
• Not assessable under the current engagement.

This prevents the presence of a playbook from being mistaken for complete security coverage.

The OWASP Web Security Testing Guide is another important reference for building the test catalogue. Its methodology covers information gathering, identity, authentication, authorization, session management, input validation, business logic, client-side testing, and reporting. The stable release is version 4.2, while the project is developing version 5.0.

10. Tool registry and execution adapters

Do not make the LLM responsible for inventing tool commands and executing them directly.

Instead, expose a registry of approved capabilities with typed inputs, explicit permissions, bounded outputs, and known limitations.

10.1 A practical initial toolchain

These are candidate components, not a recommendation to install everything at once. Select tools according to the target scope, operational environment, licensing, maintenance burden, and evidence quality.

10.2 Adapter interface

Every tool adapter should define:

• Input schema.
• Output schema.
• Allowed operations.
• Target and protocol restrictions.
• Impact classification.
• Resource limits.
• Timeout and cancellation behavior.
• Error states.
• Evidence format.
• Tool version and configuration.
• Known limitations.

An illustrative interface:

class TestAdapter:
    def validate_input(self, request, engagement):
        """Validate schema and required prerequisites."""

    def classify_impact(self, request):
        """Return the registered impact classification."""

    def execute(self, request, execution_context):
        """Execute an authorized, bounded operation."""

    def normalize_result(self, raw_result):
        """Return structured results and evidence references."""

This is a conceptual interface; the implementation must ensure that the adapter cannot bypass the external policy gateway.

10.3 Controlled script generation

Generated code can help with application-specific checks, but it creates substantial risk.

A generated script could contain an unintended network destination, leak credentials, consume excessive resources, or perform a destructive action.

Use this order of preference:

1. Reuse an approved playbook.
2. Invoke an existing tool through a controlled adapter.
3. Extend a reusable, reviewed test module.
4. Generate a narrowly scoped script only when existing capabilities are insufficient.

Generated scripts should undergo schema checks, static analysis, dependency restrictions, review proportional to risk, and isolated execution.

They should not receive unrestricted shell access, arbitrary network access, unrestricted package installation, or broad filesystem permissions.

11. Playbook engineering

A playbook should describe a complete testing method, not merely a payload.

Each playbook should specify:

11.1 Why this matters

Suppose an application returns a different response when an identifier changes. That observation may indicate an authorization issue, but it may also reflect a nonexistent object, cache behavior, or an intended public resource.

A well-designed playbook defines the baseline and expected authorization model before execution. The validation system then evaluates whether the observed difference meets the criteria for a finding.

This is more reliable than asking an LLM to decide whether a response “looks vulnerable.”

12. Independent validation and evidence quality

The planner should not be the sole judge of its own hypotheses.

A second LLM call does not automatically provide independent verification. Both calls may share the same mistaken assumptions.

12.1 A separate validation engine

The validation subsystem should use reviewed, deterministic checks wherever practical.

Examples include:

• Authorization findings evaluated against an expected permission model.
• Injection findings evaluated using evidence specific to the vulnerability class and appropriate baseline comparisons.
• SSRF findings evaluated using authorized, correlated evidence rather than a generic response difference.
• Dependency findings distinguishing a version fingerprint from evidence of vulnerable, relevant code.
• Security-header findings distinguishing a missing header from demonstrated impact.
• Error-handling findings distinguishing an expected failure from a security weakness.

Validation logic will vary by vulnerability class. Not every class can be fully validated using a simple deterministic predicate, and some findings will require expert review.

12.2 Evidence schema

A finding should contain enough information for another engineer to reproduce and assess it.

An illustrative JSON structure:

{
  "finding_id": "F-00042",
  "engagement_id": "engagement-2026-001",
  "title": "Possible cross-user object access",
  "status": "suspected",
  "severity": "unassigned",
  "confidence": "low",
  "asset": "app.example.test",
  "endpoint_ref": "endpoint-018",
  "identity_context": {
    "actor_ref": "test-user-b",
    "expected_policy_ref": "policy-007"
  },
  "evidence_refs": [
    "evidence-102",
    "evidence-103"
  ],
  "validation": {
    "validator_version": "authz-validator-1.0",
    "result": "pending"
  },
  "limitations": [
    "Expected object-sharing policy requires confirmation"
  ]
}

The example intentionally leaves the finding unvalidated. A schema should not turn a hypothesis into a confirmed vulnerability simply because a report was generated.

12.3 Evidence provenance

Capture:

• Request and response references.
• Relevant headers and status codes.
• Redacted bodies or bounded excerpts.
• Identity and session references.
• Application state before and after execution.
• Tool and playbook versions.
• Timestamps and execution IDs.
• Policy decision and validation outcome.

Sensitive raw artifacts should be stored separately from model context. The model should receive only the minimum relevant excerpts or structured summaries needed for the task.

12.4 Severity and confidence

Severity describes potential impact and exploitability. Confidence describes how strongly the available evidence supports the finding.

These are different dimensions. A suspected high-impact weakness may require urgent human investigation without being reported as a confirmed critical vulnerability.

13. Prompt injection and the security of the agent itself

An agentic pentester introduces a second attack surface: the testing platform itself.

Target-controlled content may include malicious instructions in HTML, JavaScript, API responses, error messages, uploaded files, or documentation.

A model might interpret that content as instructions to expand scope, disclose credentials, or invoke tools.

13.1 Treat target content as untrusted

The architecture should:

• Preserve provenance for all observations.
• Distinguish trusted policy from untrusted target data.
• Use structured input/output schemas.
• Keep secrets out of model context unless strictly necessary.
• Reject unauthorized tool calls outside the model.
• Restrict generated code.
• Validate evidence before it influences important decisions.
• Keep policy enforcement independent of planner output.

Delimiters and prompt wording can help the model interpret data correctly, but they are not sufficient security controls.

13.2 Test the agent platform adversarially

Create test cases where target content attempts to:

• Expand the authorized scope.
• Request credentials or secret material.
• Trigger an unregistered tool.
• Cause arbitrary code execution.
• Suppress or alter a finding.
• Exhaust context or execution budgets.
• Mislead the agent with fabricated instructions or false evidence.

The required result is not simply that the model recognizes the attack. The enforcement boundary must prevent prohibited actions even if the model proposes them.

For broader AI risk-management principles, the NIST Generative AI Profile provides a useful framework for identifying and managing risks associated with generative AI systems.

14. Execution isolation, credentials, and emergency stop

14.1 Isolated workers

Browsers, scanners, and custom modules should run in dedicated, least-privilege execution environments.

Controls should cover:

• Network egress.
• Filesystem access.
• Process execution.
• CPU, memory, and runtime.
• Output size.
• Credentials.
• Dependency installation.
• Inter-worker communication.
• Evidence retention.

A container is a useful isolation mechanism, but containerization alone does not establish a complete security boundary. Network policy, runtime configuration, host security, and permissions must also be considered.

14.2 Secrets lifecycle

Use a secrets manager and short-lived credentials where practical.

Persist references to credentials rather than their raw values in workflow state. Redact tokens, cookies, passwords, and session identifiers before general-purpose logging and model-context construction.

Define credential rotation, revocation, access auditing, and deletion.

Test for secret leakage through traces, errors, screenshots, browser artifacts, generated code, and final reports.

14.3 Cross-engagement isolation

Assign each engagement an immutable identifier and enforce isolation across:

• Database queries.
• Object storage.
• Caches.
• Queues.
• Vector retrieval.
• Credentials.
• Worker environments.
• Logs and reports.

Isolation must be enforced in the data-access and execution layers, not merely by asking the model to remember which customer it is testing.

14.4 Independent emergency stop

The emergency-stop service must function even if the planner is looping or compromised.

It should be capable of:

1. Revoking engagement-specific network egress.
2. Preventing new jobs from starting.
3. Cancelling queued tasks.
4. Terminating active workers.
5. Closing or invalidating sessions where practical.
6. Recording affected execution IDs and the stop reason.
7. Requiring explicit authorization before resumption.

Some already-issued requests or asynchronous operations may not be cancellable. The system must document this limitation and provide recovery procedures.

15. Resource budgets and failure handling

Autonomous systems can waste resources through repeated tests, unproductive retries, planning loops, or excessive model calls.

Define budgets at engagement, worker, tool, endpoint, and playbook levels.

Relevant limits include:

• Request and connection counts.
• Concurrency.
• Execution time.
• Retry ceilings.
• Browser and script resources.
• Model-call and execution cost.
• Error and timeout thresholds.
• Rate-limit responses.
• Output and evidence size.

Use conservative settings in the initial release and measure their effect in controlled environments.

15.1 Explicit terminal states

Each execution should end in a defined state, such as:

• Completed.
• Validated finding.
• No issue observed under the tested conditions.
• Inconclusive.
• Blocked.
• Cancelled.
• Failed.
• Out of scope.

A tool failure must never be interpreted as proof that a vulnerability is absent.

15.2 Recovery design

Define behavior for:

• Expired sessions.
• MFA and CAPTCHA gates.
• Browser crashes.
• DNS changes.
• WAF blocks and rate limits.
• WebSocket disconnections.
• Partial tool output.
• LLM timeouts.
• Context exhaustion.
• Worker crashes.
• Concurrent state conflicts.
• Database failures.
• Emergency cancellation.

Retries should be bounded, and state-changing operations should not be repeated blindly.

16. Data architecture and persistence

A relational database is generally sufficient for the first release.

Use it to store engagements, assets, endpoints, identities, sessions, tests, findings, budgets, and policy decisions.

Store large artifacts in object storage. Use immutable or tamper-evident evidence records where appropriate.

16.1 Do you need a graph database?

Not initially.

A relational schema can represent endpoints, state transitions, and relationships. Add a dedicated graph database when relationship queries become difficult to maintain or materially affect planning performance.

16.2 Do you need a vector database?

Not necessarily.

The MVP can retrieve approved playbooks and documentation using indexed metadata, text search, or ordinary database queries. A vector store becomes useful when semantic retrieval across a large, curated methodology library demonstrates measurable benefits.

Retrieval results should retain source, version, and provenance. Similarity is not evidence that a playbook applies to a specific target.

16.3 Suggested initial data entities

Use migrations and versioned schemas so old engagements remain interpretable after the software evolves.

17. Reporting and auditability

A report must explain not only what the system found, but also what it tested and what it could not test.

A complete report should contain:

1. Executive summary.
2. Scope and exclusions.
3. Methodology and tool versions.
4. Asset and endpoint inventory.
5. Coverage by route, identity, workflow, and vulnerability category.
6. Validated findings with evidence and severity.
7. Reproduction methodology.
8. Business and technical impact.
9. Remediation recommendations.
10. Untested, blocked, and inconclusive areas.
11. Resource limits and relevant execution failures.
12. Evidence provenance and timestamps.

Keep an audit record of each decision: the hypothesis, supporting evidence, proposed action, policy decision, execution result, validation outcome, and stopping reason.

Do not depend on private LLM reasoning traces as the audit record. A structured decision log is more useful for debugging, reproducibility, and independent review.

18. Benchmarking: how to prove the platform works

A framework that produces many findings is not necessarily a good pentester. It may produce duplicates, false positives, or findings that cannot be reproduced.

Benchmark the system against applications with known vulnerabilities and expected outcomes.

18.1 Controlled test environment

OWASP Juice Shop is a deliberately vulnerable web application that includes a broad range of security weaknesses and is designed for security training and tool evaluation. It is a useful starting point because it exercises JavaScript-heavy frontends and REST APIs.

Supplement it with other authorized lab applications and purpose-built test cases for authorization, authentication, stateful workflows, and tenant isolation.

Maintain a ground-truth catalogue that records:

• Known vulnerability.
• Affected endpoint or workflow.
• Required identity and state.
• Expected test prerequisites.
• Safe reproduction method.
• Expected evidence.
• Known limitations.

Ground truth should be independently verified and versioned.

18.2 Quantitative metrics

For a defined set of known vulnerabilities:

A missing or untested endpoint should not automatically be counted as a confirmed false negative unless the benchmark establishes that the test was applicable and the ground truth is reliable.

18.3 Safety acceptance tests

Before enabling substantial autonomy, verify that:

• Out-of-scope destinations are rejected.
• Unauthorized redirects cannot bypass scope checks.
• Unregistered tools cannot be invoked.
• Resource budgets are enforced.
• Target-controlled prompt injection cannot authorize prohibited actions.
• Credentials do not leak into general logs or reports.
• Cross-engagement data cannot be retrieved.
• Emergency stop revokes execution capability.
• Interrupted workflows recover without unsafe duplicate operations.
• Findings remain traceable to evidence.

Greater autonomy should only be enabled after the system satisfies both quality and safety gates.

19. Architecture decision records

The following decisions are recommended for an initial implementation.

Decision 1: Deterministic workflow or LLM-driven planning?

Recommendation: Start with deterministic orchestration.

It is easier to test, reproduce, and constrain. Add bounded LLM planning after the evidence model, adapters, policy enforcement, and validators are reliable.

Change this decision when benchmarks demonstrate that adaptive planning discovers meaningful additional vulnerabilities without unacceptable cost, false positives, or safety regressions.

Decision 2: Single agent or multi-agent system?

Recommendation: Begin with a single planner and modular specialist components.

Multiple agents introduce coordination overhead, conflicting state, duplicate work, and more complex debugging. Introduce separate agents only when experiments demonstrate a measurable benefit.

Decision 3: Existing tools or custom scripts?

Recommendation: Prefer existing tools for mature, repeatable capabilities.

Custom modules are justified for application-specific validation, missing protocol support, or workflows not covered by existing tooling. Generated scripts should remain a controlled exception.

Decision 4: Browser-first or protocol-first discovery?

Recommendation: Use a hybrid approach.

Browser-driven discovery reveals interactive states and real API traffic. Direct protocol testing provides finer control and is more efficient for repeatable checks. Neither approach alone is sufficient for all applications.

Decision 5: Automatic or human-approved testing?

Recommendation: Automate low-impact, reversible tests first.

Require additional authorization for tests with meaningful side effects, destructive behavior, financial consequences, or uncertain operational impact. Do not equate a user’s general authorization to test an application with permission to perform every possible action against it.

Decision 6: Model-based or deterministic validation?

Recommendation: Use deterministic validation wherever practical, supported by independent evidence and human review for ambiguous cases.

An LLM can help interpret context and identify alternative hypotheses, but its confidence should not be treated as proof.

Decision 7: In-process checks or independent network enforcement?

Recommendation: Use independent enforcement.

Checks inside a tool or planner can be bypassed by bugs, compromised processes, or incorrect model output. Network and execution controls should constrain what a worker can actually do.

20. A phased implementation roadmap

The safest and most efficient development strategy is incremental.

Phase 1: Deterministic MVP

Objective: Establish a trustworthy execution and evidence foundation.

Build:

• Engagement manifest and scope validation.
• Basic asset and endpoint inventory.
• Controlled HTTP client.
• Browser-driven discovery.
• A small set of reviewed test modules.
• Structured execution logs.
• Evidence storage.
• Finding and coverage schemas.
• Basic reporting.
• Worker isolation and emergency stop.

Do not require autonomous planning or generated scripts in this phase.

Exit criteria:

• Every active action is checked against scope.
• Out-of-scope requests are rejected in negative tests.
• Evidence is traceable to execution IDs.
• Findings can be reproduced manually.
• The system distinguishes confirmed, suspected, blocked, and untested cases.
• Resource limits and cancellation work as designed.

Phase 2: Authenticated and state-aware testing

Objective: Model application behavior beyond anonymous URLs.

Add:

• Authorized test identities and role matrices.
• Session lifecycle management.
• Browser and API state tracking.
• Workflow graphs.
• Authorization validators.
• GraphQL and WebSocket adapters where required.
• Safe recovery from expired sessions and interrupted workflows.

Exit criteria:

• The framework can exercise documented workflows under multiple authorized identities.
• Authorization tests compare observations with an expected policy.
• Stateful failures are recorded correctly.
• Coverage distinguishes discovered, reached, exercised, and validated behavior.

Phase 3: Bounded agentic planning

Objective: Introduce adaptive decision-making.

Add:

• A planner that proposes actions through the registered tool interface.
• Hypothesis and uncertainty tracking.
• Evidence-driven replanning.
• Deduplication and stopping logic.
• Cost-aware prioritization.
• Human approval for higher-impact actions.

The planner must not execute arbitrary shell commands or override policy decisions.

Exit criteria:

• The agent adapts to changed observations.
• It avoids repeated unproductive actions.
• It produces structured decision records.
• It improves benchmark results over the deterministic baseline.
• It does not regress safety or evidence quality.

Phase 4: Expanded autonomy

Objective: Improve depth and efficiency without losing control.

Consider:

• Specialized agents for genuinely distinct tasks.
• Controlled script generation.
• More protocol adapters.
• Larger playbook libraries.
• Advanced graph-based planning.
• More sophisticated evidence correlation.
• Additional model providers.

Each capability should be introduced behind its own tests, limits, and release criteria.

Exit criteria:

• The new capability produces measurable benefits.
• Its failure modes are documented.
• Its execution is bounded and auditable.
• Its findings remain reproducible.
• Emergency-stop and scope-enforcement tests continue to pass.

21. Recommended MVP technical specification

A practical first release can use the following components.

These are starting recommendations, not mandatory technology choices. The architecture matters more than the specific framework or database.

22.1 Minimal execution lifecycle

1. Receive engagement configuration
2. Validate authorization and scope
3. Create engagement state
4. Discover approved assets
5. Validate every candidate before active probing
6. Discover routes and application states
7. Select an approved test module
8. Submit the action to policy enforcement
9. Execute within the authorized boundary
10. Capture and store evidence
11. Validate the result
12. Update finding and coverage states
13. Replan or stop according to policy and budgets
14. Generate the report

The first release should be able to execute this lifecycle without an LLM.

Once it works reliably, an LLM can be introduced to improve prioritization and select among existing actions. This makes it possible to measure whether the model contributes real value instead of merely adding complexity.

22. High-priority engineering improvements

Five improvements deserve priority over adding more agents.

1. Non-bypassable scope enforcement

Why: A reasoning error must not become an unauthorized network operation.

Implementation: Enforce destinations, protocols, ports, redirects, resource budgets, and tool permissions outside the planner.

Acceptance criterion: Adversarial tests attempting direct egress, unauthorized redirects, and unregistered tool execution are rejected.

2. Independent evidence validation

Why: A large number of findings has little value if they cannot be reproduced.

Implementation: Create reviewed validators for priority vulnerability classes, beginning with authorization failures and common configuration or injection findings.

Acceptance criterion: Benchmark findings are evaluated against ground truth, with precision, recall, and reproduction rates reported separately.

3. State-aware authenticated discovery

Why: Important security failures often depend on roles, sessions, and workflow state.

Implementation: Track identities, sessions, routes, objects, and transitions in a persistent state model.

Acceptance criterion: The framework can document which authorized workflows and identity contexts were actually tested.

4. Secret and engagement isolation

Why: A penetration-testing platform handles sensitive credentials and evidence across potentially unrelated customers.

Implementation: Use secret references, redaction, access controls, immutable engagement identifiers, and isolated execution environments.

Acceptance criterion: Negative tests cannot retrieve another engagement’s artifacts or expose secrets through model context, logs, or reports.

5. Benchmark-driven autonomy

Why: More adaptive behavior does not necessarily produce better testing.

Implementation: Compare deterministic and agentic modes against the same versioned ground-truth suite.

Acceptance criterion: Adaptive planning improves relevant detection or coverage metrics without violating safety gates or materially degrading reproducibility.

23. Unresolved engineering questions

Some design decisions depend on the deployment environment and cannot be settled from general architecture principles alone.

Before implementation, determine:

1. Will the framework be single-tenant, internally hosted, or a commercial multi-tenant service?
2. Which model providers and deployment modes are permitted?
3. Will customers supply credentials, identity matrices, or application specifications?
4. Which testing operations require explicit human approval?
5. What level of production testing is authorized?
6. Which protocols must be supported in the first release?
7. What are the expected concurrency, cost, and runtime targets?
8. How long must evidence be retained, and who may access it?
9. Which jurisdictions and contractual requirements apply?
10. Which vulnerability classes must be independently validated before reporting?
11. What performance improvement would justify introducing agentic planning?
12. What evidence is required before the platform can claim production readiness?

External black-box testing cannot conclusively assess every internal control. Software composition, source-code logic, internal IAM configuration, logging quality, and private network segmentation may require source access, instrumentation, or separate assessments.

The reporting system must state these limitations clearly.

24. Final recommendations

A genuinely agentic penetration-testing platform should not be designed as a free-running LLM with access to a terminal and a collection of offensive tools.

It should be designed as a constrained security-testing system in which adaptive reasoning operates inside independently enforced boundaries.

The recommended development sequence is:

1. Build reliable deterministic discovery and execution.
2. Establish state-aware application mapping.
3. Implement reviewed evidence validators.
4. Enforce scope, isolation, budgets, secrets management, and emergency stop.
5. Benchmark against known vulnerabilities.
6. Introduce bounded adaptive planning.
7. Expand autonomy only when the measured results justify it.

The most important architectural distinction is between the component that decides what to investigate and the component that determines what is allowed to happen.

The planner may be uncertain, manipulated, or wrong. The authorization boundary must still work. A tool may fail. The system must not interpret that failure as evidence of security. A model may propose a convincing vulnerability. The validation system must still demand reproducible evidence.

If these principles are implemented correctly, the resulting platform can progress from deterministic scanning to adaptive penetration testing without sacrificing control, reproducibility, or auditability.

The objective is not to create the most autonomous agent possible. It is to create the most capable agent whose actions, findings, and limitations can be independently verified.