django-dynamic-scraper - Documentation

Django Dynamic Scraper (DDS) is an app for Django build on top of the scraping framework Scrapy. While preserving many of the features of Scrapy it lets you dynamically create and manage spiders via the Django admin interface.

Note

Latest new features added :

  • Django 1.11/Scrapy 1.5 support
  • Python 3.5/3.6/3.7 (experimental) support
  • Javascript rendering
  • Scraping JSON content
  • More flexible ID and detail page URL(s) concept
  • Several checkers for a single scraper
  • Custom HTTP Header/Body, Cookies, GET/POST requests
  • Scrapy Meta attributes
  • Scraper/Checker Monitoring

See Release Notes for further details!

Features

  • Create and manage scrapers for your Django models in the Django admin interface
  • Many features of Scrapy like regular expressions, processors, pipelines (see Scrapy Docs)
  • Image/screenshot scraping
  • Dynamic scheduling depending on crawling success via Django Celery
  • Checkers to check if items once scraped are still existing

Indices and tables