Ingestion

The crawl behind the index.

Every handle is set to the deepest crawl profile: full post and reel history, captions, hashtags, collaborator tags and engagement — then AI analysis into a creator thesis.

Handles0
Ingested00% complete
Posts stored0
CrawlerIdleTwo handles per run, locked
Pipeline
Stage 1

Profile

Bio, links, category, follower and following counts

Stage 2

Content

Full post and reel history with media, captions, hashtags, tagged brands

Stage 3

Engagement

Per-post likes, comments, views, saves proxy and posting cadence

Stage 4

Analysis

Aesthetic clustering, audience read, brand history, rate band, Circle thesis

Queue
HandleStagePosts

The crawler runs on a schedule with a single-run lock, retry limits and an automatic pause if the scraping account hits a billing or rate limit.