The problem
Confident, plausible, uncheckable
Ask an LLM what people complain about in a market and you get polished prose you can't verify. The failure mode isn't being wrong. It's that you can't tell. I wanted output where any claim clicks through to an actual Reddit thread, posted by an actual person, on an actual date.
An evening per topic
The honest alternative, reading niche subreddits for hours, works. It also costs 3–4 hours per idea, and I'd still worry I'd missed the one subreddit where the real conversation lived.
GummySearch left a window
Its decline left a choice between expensive social-listening suites built for brands and nothing at all. The window is why this got built now; speed-to-useful was a design constraint from day one.
Who it's for
Built for one user: me, validating product ideas. The pattern generalizes to anyone who has to justify a build-or-don't decision with evidence rather than vibes: indie hackers, PMs writing opportunity docs, founders doing customer discovery before the customers exist. Turn a vague market question into cited evidence for less than the price of a coffee.
The approach
One command:
deeplurk research "food logging and calorie tracking for Indian food"
Twenty-ish minutes later, two artifacts: a markdown report and a self-contained shareable HTML page. In between, the pipeline:
- Discovers which subreddits actually host the conversation. A search pass plus an LLM relevance judgment, with rejection reasons logged. For this topic it rejected r/indianfood: "recipes and cuisine appreciation, not calorie-tracking needs."
- Collects posts and comments under a hard item budget, split across the selected subs.
- Classifies every item (pain point, feature request, tool complaint, workaround, recommendation request, or none) with a small cheap model.
- Synthesizes themes with a large model, then verifies its own citations. A second, independent LLM pass re-checks every quote the synthesizer cited against the theme it supports, and drops the ones that don't hold up.
- Reports with every quote carrying author, subreddit, upvotes, post age, and permalink. Themes that survive verification with fewer than 3 citations get demoted to a "weak signals — read skeptically" section instead of being padded to look strong.
A real finding from a real run, to make it concrete: researching calorie tracking for Indian food (713 items), the pipeline surfaced that mainstream apps list the same dal tadka anywhere from 80 to 600 kcal, that users' dominant workaround is weighing the oil container before and after cooking, and that multiple indie devs are already recruiting beta testers for Indian-food-specific trackers. Each claim clickable to its thread.
Architecture
Local-first by conviction: no server, no external DB, no queue. A Typer CLI orchestrates five stages, shown in the diagram above. SQLite holds every scraped item with full provenance (which actor, which run, when) plus its classification, so nothing is ever re-scraped or re-classified. Reddit access goes through a pinned Apify actor behind a SourceAdapter protocol; the actor's output schema is treated as non-contractual and defensively parsed, with a named fallback (PRAW) if the actor breaks or reprices.
The stack, in full: Python 3.11, Typer, httpx, the Anthropic API (Haiku 4.5 for classification and verification, Sonnet 5 for synthesis, structured outputs via JSON schema), Apify, SQLite, Jinja2, and pytest with a fully offline suite. No framework, five runtime dependencies.
Architecture decisions & trade-offs
Each of these bought something and cost something. The costs are listed too.
Two models, deliberately split
Haiku classifies hundreds of items per run and verifies citations; Sonnet gets exactly one call, the synthesis. A run's LLM bill stays around a dollar, and the expensive model is spent only where judgment compounds. The trade: per-item classification is only as sharp as the small model, so the prompt tells it to prefer 'none' when unsure. Precision over recall.
Structured outputs everywhere
Every LLM call is JSON-schema-constrained; downstream code never parses free text. Failures are loud schema errors instead of silent misreads. The trade surfaced in production: a truncated response crashed a run, and the wrapper now detects truncation and retries once with a doubled cap.
The verifier is adversarial to the synthesizer
The synthesis model's citations are treated as claims, not facts. A separate pass re-reads every cited item against its theme and drops the ones that don't support it, with drop counts printed in the report. The drop rate itself became a quality metric that caught a real prompt bug.
Budget enforced in code, not in prompts
A BudgetTracker gates every scrape dispatch; the LLM never controls spend. Even Phase 2's planned agentic loop can only reallocate the budget Phase 1 would have spent, never raise it.
Cache with provenance columns
Every item records its source actor, Apify run ID, and scrape timestamp. When engagement data silently vanished, provenance made the blast radius knowable: exactly 168 items, scraped before the fix, permanently affected.
An independent reviewer with a journal
A pipeline-critic agent with separate context audits every report against the guardrails and keeps a dated journal of verdicts and resolutions. No phase is done until its review is logged. It has caught bugs the tests didn't: a count mismatch between report header and sources table, and a theme rendering one 5-word quote despite 6 verified citations.
What broke, and what it taught me
The most portfolio-worthy part of this build is what broke. Three production incidents and one measured quality win, each now a regression test or a code guard.
The dead signal that looked alive
Every quote in an early report showed 0 upvotes. Ranking, the under-the-radar detector, and top-N selection were all silently running on a dead engagement signal while looking functional. Root cause: the Apify actor omits upvote fields unless an undocumented flag is set. Fix: the flag, plus a tripwire that detects an all-zero corpus, degrades ranking, and stamps the report 'engagement data unavailable.' Lesson: validate the semantics of your data, not just its schema.
The orphaned runs that kept billing
When a scrape exceeded the client's timeout, the client gave up but the server-side run kept going, and kept charging. The 'timeout' was often a false failure: a search that takes 5–13 seconds on a good day took 804 seconds on a throttled one, and succeeded after the client stopped listening. Fix: configurable timeout, and the client now aborts the remote run whenever it gives up. Lesson: your timeout handler must clean up the remote side, and third-party latency variance can be 60x.
The truncation crash
The largest corpus yet (789 items) pushed the synthesis output past its token cap; the JSON was cut mid-string and crashed the run after the full scrape-and-classify spend. Fix: detect the truncation stop-reason, retry once with a doubled cap, else fail with an error that names the knob. Lesson: structured outputs remove parse ambiguity, not length limits.
The prompt number that incentivized padding
The synthesis prompt said to cite '3+' supporting items per theme. That number quietly rewarded padding: the verifier was dropping ~36% of proposed citations. Removing it, and stating that 1–2 solid citations is acceptable (the theme just gets demoted, honestly), cut the drop rate to ~17% on the next run. A one-sentence prompt change, quantified by an adversarial pass.
The plan itself got the same treatment before a line of code: three specialist critique agents (product, business, GTM) reviewed the draft and filed findings on platform dependency, cost ceilings, and evidence integrity, each resolved and logged before building. On top of the machine review sits a human acceptance bar: for each of the first 5 real runs, I grade every quote-to-theme attribution by hand, with at least 80% required to genuinely support their theme. The critic and I disagree sometimes. The journal records who was right.
Cost engineering
Cost per run is an argument; a monthly bill is an apology. Same philosophy as the job-finder build, applied here from the plan stage.
- Per run (~700 items): roughly $2.50–3 of pay-per-result scraping plus about $1 of LLM tokens. Under $4 all-in, against 3–4 hours of manual lurking.
- The item budget is the cost model. The actor charges per result, so
max_total_items(default 800) is the spend ceiling, enforced before every dispatch. - Caching is a cost feature. Re-running a topic reuses every cached item and classification; a crashed run's retry cost pennies because 789 items were already in SQLite.
- A shared API account has no per-project budget. Another project's scraper quietly consumed the month's Apify quota and DeepLurk's runs started failing with 403s. Diagnosis went through the usage API line by line. The standing rule now: check account limits before diagnosing "DeepLurk" cost problems.
Business impact
The number that matters most is the middle one. The citation-padding fix was measured by an independent verification pass, before and after, on real runs. Evidence quality here isn't a vibe; it has a metric, and the metric moved.
How it compares
++ Where DeepLurk wins
- Auditable evidenceEvery claim is a permalink, not an aggregate sentiment dashboard.
- Unit economicsAbout $4 a run, no subscription, and the data sits in a SQLite file you can query.
- Topic-first discoveryFinds the subreddits for you instead of making you name them.
− Where GummySearch-style SaaS wins
- Always-on monitoringNo alerting or scheduled watches yet; Phase 2's monitor command is the planned answer.
- Breadth and polishSingle data source (Reddit), single user, CLI-only.
++ Where it beats asking an LLM directly
- Grounded outputScraped, dated, attributed posts. Zero unlinked assertions.
- Reproducible corpusDeterministic budget, cached items, provenance stamps on everything.
− Where the LLM wins
- Speed and priceTwenty minutes and real dollars per question, against instant and effectively free.
Not strictly better than commercial tools. It optimizes for a different question: "should I build this?" needs auditable evidence more than it needs dashboards.
Guardrails
What it deliberately doesn't do:
- Never a silently thin report. Scraped items are schema-validated, and a run that collects fewer than 30 usable items fails loudly instead of producing confident emptiness.
- Never padded evidence. Themes below 3 verified citations are demoted to "weak signals," visibly labeled.
- Never unattributed. Every quote carries author, sub, upvotes, age, and permalink, and stays excerpt-length.
- Never auto-published. Reports are private by default; publishing is a deliberate human act. One run's mental-health theme quoted suicide discussions, which is exactly why the publish boundary is human-gated and sensitive-content handling is specced for Phase 2.
- Never posts, replies, DMs, or lead-gens. Read-only research, full stop.
Build roadmap
Plan + adversarial critique
Full plan drafted, then torn apart by product, business, and GTM critique agents. 10+ findings resolved before build. Guardrails written down as non-negotiables.
Phase 1 pipeline
Discover, collect, extract, synthesize, report. SQLite cache with provenance, structured outputs throughout, offline test suite.
First real runs
Two live topics. The pipeline-critic's first audit caught the dead-engagement-signal incident; corpus floor and discovery noise filter added.
Hardening from production
Configurable timeouts with remote abort, truncation retry, silent-drop fallback tagging, the citation-padding prompt fix, 29 offline tests.
Acceptance bar (current)
5 real runs, every quote-to-theme attribution hand-graded against generated checklists. Findings feed the Phase 2 specs.
Specced, gated on acceptance
Adaptive research loop under hard code rails, depth presets, publish/compare/monitor commands, and a local dashboard to browse runs and grade attributions from the browser.
Where it goes next
Phase 2 ships once the acceptance bar clears: an orchestrator that reallocates budget mid-run (but can never raise it), --depth quick|standard|deep presets, and the local dashboard. Screenshots of the dashboard and the attribution-grading view land on this page when they exist. Until then, the HTML reports themselves are the demo.