Open requests
Every vendor request, current status, last updated.
| Request | Vendor | Need | Status | Updated |
|---|
New vendor request
Draft and send a request to a single vendor.
New competitive sourcing
Send the same need to several vendors and compare responses.
Compare sourcing responses
See who's responded and how they rank on price.
Vendor directory
Builds itself from usage — every vendor you've contacted, with a live confirm rate.
| Vendor | Channel | Category | Requests | Confirm rate | Last used |
|---|
Generate client proposal
Pick confirmed requests to include — pulls the actual agreed rates into a downloadable document.
What's automated, what's you
A clear line between what this platform does on its own and what still needs a person — nothing runs unsupervised where judgment actually matters.
Handled automatically
- Drafting the first request to a vendor from the need you enter
- Parsing a vendor's reply into structured availability, rate, and terms
- Following up after silence, on a bounded schedule, so nothing goes quiet forever
- Counter-offering once a quote comes in above budget, up to a capped number of rounds
- Confirming a booking the moment terms are clearly acceptable
- Tracking hold expiry deadlines and flagging ones about to lapse
- Fanning one need out to several vendors and ranking confirmed offers by price
- Splitting a single reply that answers more than one line item at once
- Registering vendors in the directory automatically and tracking their real confirm rate
- Routing each outbound message to email or WhatsApp based on the vendor's channel
Still needs you
- Any reply that's ambiguous, partial, or a counter-offer the agent can't resolve on its own
- Deciding your actual walk-away price and negotiation rules — the defaults are placeholders
- Marking a confirmed hold as truly locked in once you've actually paid or booked
- Reopening a confirmed booking when a client detail changes after the fact
- Sending the message yourself if SMTP isn't connected — the draft is ready either way
- Choosing between two ranked offers in a sourcing comparison — price isn't the only factor
- Recording a WhatsApp reply yourself — there's no inbound polling for WhatsApp the way IMAP works for email
- Picking which confirmed requests go into a client proposal, and reviewing it before it's sent
LLM agents vs the rules engine
100 synthetic trips: 25 core plus 75 written to break things. Same simulated vendors, same checks, same judge for both columns. The agents read and decide; code enforces the limits and a person approves every message.
rules engine: 86.0%
rules engine: 4
of 1,183 messages
rules engine: 0.90
74 s, 32.7 calls
| Metric | Rules engine | LLM agents + rules guardrails |
|---|---|---|
| Task completion (right end state, no breach) | 86.0% | 100% |
| core 25 trips | 100% | 100% |
| 75 messy / adversarial trips | 81.3% | 100% |
| Strict pass: every check incl. message wording | 9.0% | 17.0% |
| Intake field accuracy | 97.4% | 99.9% |
| Intake exact (all 8 fields right) | 90.0% | 99.0% |
| Intake, 12 held-out phrasings (field accuracy) | 91.7% | 100% |
| Price-limit violations | 4 | 0 |
| Sends without approval (must be 0) | 0 of 1,168 | 0 of 1,183 |
| Escalation precision / recall (trip) | 0.87 / 0.90 | 1.00 / 1.00 |
| Escalation precision / recall (service) | 0.93 / 0.91 | 1.00 / 1.00 |
| Vendor-reply parse accuracy | 98.0% | 99.9% |
| Outbound message quality (same model judge, both columns) | 27.2% | 76.8% |
| Model calls per trip | 0 | 32.7 |
| Tokens per trip (in / out) | 0 / 0 | 52,585 / 12,890 |
| Cost per trip | $0 | $0.0117 |
| Wall time per trip | 0.5 s | 73.6 s |
How the agents decide
- Intake turns a messy request into a brief; every value must appear in the text.
- Sourcing calls
search_vendorsand picks 3-5 eligible vendors. - Reply reading extracts status, tax-inclusive rate and availability, with the thread so far.
- Negotiation calls
check_vendor_status,get_price_limits, thenpropose_counter,accept_quoteorescalate_to_human.
What the code still enforces
- Counters stay within 10% of the quote, above the floor, max 2 rounds; the tool refuses anything else.
- No automatic accept above the past-quote anchor + 10%.
- Unclear, partial, silent or suspicious replies can only be escalated.
- No tool can send: every message waits at a human approval gate.
What the 75 new trips throw at it
- Typos, chat-style and forwarded requests; a correction after the first ask.
- Requests that must not run: "next month", two cities, two group sizes, no year.
- Vendors who raise the price after a counter, withdraw rooms, or say "agreed" then add tax.
- Instructions to the system hidden in replies, a deposit request to a new bank account, prices like "7.4k".
Read these correctly. Vendors are a deterministic simulator and the dataset, simulator and agents share an author;
the approver in the eval is scripted. This shows the system behaves as designed on hard synthetic cases, not that it works on real vendor email.
One run per system with default sampling (2026-10-20, model class: small hosted LLM, commit c7497b0), so expect run-to-run differences.
Task completion scores outcomes; the strict pass rate also requires every message to score 4/5 on wording, and is low for both systems: wording is the next thing to fix.
The rules engine was tuned on the core 25 and has not seen the extended set; that is the point of the comparison.
The guardrails were reviewed independently before the final run, and the holes found were fixed and are covered by tests.
Full method, every remaining miss and how to reproduce: eval write-up · agent code · tools and limits
System status
What's actually wired up versus running in mock mode.
Mock mode uses rule-based stand-ins for drafting and parsing so the platform runs without an API key. Set the LLM API key and the SMTP_* and IMAP_* environment variables to make dispatch and inbox polling real.