the new git-enabled periodic wasteof crawler is currently in development
i’m planning to implement two modes:
daily crawl: only crawls the explore page and checks for new (discovered) users and posts
full crawl: crawls from anything it can find
i’m not sure in what rate should i do a full crawl
just don’t ddos wasteof and you’re fine