How the Pipeline Works
Each stage is independent and resumable, which matters because it means a failure in one connector never corrupts everything downstream.
Last updated 2026-08-17
Ingest
Connectors fetch raw documents and write them with a content hash. The hash is computed on the prose, so the same complaint reposted on another platform is caught as a duplicate rather than double-counted.
Every run is logged with its status, counts and any error — including failures, which stay visible on the sources page rather than being quietly swallowed.
Extract
Documents are batched and read by Claude, which returns structured pain points: a normalised statement, a verbatim quote, the job to be done, the persona, severity, willingness to pay, and the workaround in use today.
Most documents yield zero or one pain point. Returning nothing is the correct answer for a document that contains no genuine friction, and the extraction prompt says so explicitly — inventing a pain point to fill a slot would poison everything downstream.
Cluster
Pain points are grouped by similarity. With an embeddings provider configured this uses cosine distance; without one it falls back to lexical similarity, which costs nothing and works well at small corpus sizes.
Clustering is incremental — new pain points join existing clusters rather than reshuffling everything, so an opportunity that already exists does not get rebuilt every run.
Synthesise and score
Each cluster of two or more pain points is sent to the model with all its evidence and any comparable companies we hold revenue data for. The output is one product idea — not a menu of options.
Scoring happens in the same call for efficiency, but can be re-run independently when weights change.