Hey — wanted to flag that Reddit URLs are returning 500s consistently for me. Looks like Reddit is blocking the crawler. Two suggestions: (1) it'd be helpful if the error response distinguished between 'Reddit blocked us' and a real internal error, and (2) maybe consider adding a Reddit-specific path that uses their JSON endpoints, since other crawl tools handle this. Not asking for anything — just thought you'd want the feedback.

Please authenticate to join the conversation.

Upvoters
Status

In Review

Board
💡

Feature Request

Date

16 days ago

Author

patel.anush

Subscribe to post

Get notified by email when there are changes.