From "Walking the Store" to Agentic Auditing

At a global e-commerce platform, I ran a trust-badge certification program across 2M+ products in 14 countries. The review process was manual: weekly "walking the store" sessions where teams audited pages from different shopper perspectives. Even with about 65 people, we could only review about 500 products per week, so issues were found late and risk stayed high. As catalog growth accelerated, it became clear this model would not scale.

I led the build of an agentic auditing system that mirrored the manual workflow and made it operational at scale. A decision-tree with ReAct (Reasoning + Acting) checked page images, descriptions, and claims against policy and product rules. High-confidence cases passed automatically, while lower-confidence cases were routed to human reviewers with evidence and reasoning. I used a lightweight open source model plus prompt caching to cut audit cost per product from about $0.20 to below $0.10. Compliance and legal co-designed the rule tree and reviewed early error patterns, which improved trust and adoption.

Results were strong. Coverage moved from about 500 products per week to about 2,000 per day, roughly a 30x increase. The system also exposed hidden concentration risk, including one segment where a shared source caused defect rates near 45%, which triggered a larger policy update. Alongside being an auditing tools, the initiative delivered about $250K+ annual savings and capacity unlock equivalent to 2 FTEs.

Architecture

flowchart TD
    A[Page URL] --> B[Crawler Agent]
    B --> C[Policy and Rule Engine]
    C --> D[Reasoning Layer - LLM]
    D --> E{Confidence}

    E -->|High| F[Auto-pass]
    E -->|Medium| G[Human Review Queue]
    E -->|Low| H[Log for Learning]

    F --> I[Audit Trail]
    G --> J[Reviewer Decision]
    J --> K[Approve or Correct]
    H --> L[Pattern Analysis]

    K --> M[Feedback Loop]
    L --> M
    M --> C

    I --> N[Weekly Metrics Dashboard]
    K --> N

    classDef det fill:#DBEAFE,stroke:#1D4ED8,color:#1E3A8A,stroke-width:1.5px
    classDef guard fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:1.5px
    classDef prob fill:#FEF9C3,stroke:#CA8A04,color:#713F12,stroke-width:1.5px
    classDef fb fill:#F3E8FF,stroke:#9333EA,color:#581C87,stroke-width:1.5px

    class A,B,F,H,I,L,N det
    class C,E,G,J,K guard
    class D prob
    class M fb