the new git-enabled periodic wasteof crawler is currently in development

i’m planning to implement two modes:

• daily crawl: only crawls the explore page and checks for new (discovered) users and posts

• full crawl: crawls from anything it can find

i’m not sure in what rate should i do a full crawl