The Goal

Example Test Cases

0 */9 * * * cd ~/development/scrapy/kickass &&  /usr/local/bin/scrapy crawl kickass -a category=books -a keywords='python' >> ~/scrapy.log 2>&1

*/50 * * * * cd ~/development/scrapy/kickass &&  /usr/local/bin/scrapy crawl kickass -a category=comics -a keywords='x-men,xmen,x men' >> ~/scrapy.log 2>&1

What we need

• sudo apt-get install python - Python 2.6 or 2.7
• Prerequisities for Scrapy
• sudo apt-get install python-dev - python dev libraries
• sudo apt-get install libxml2
• pip install Scrapy or easy_install Scrapy - Scrapy framework

Create a new Scrapy project

$ scrapy startproject kickass

Torrent Item

from scrapy.item import Item, Field

class TorrentItem(Item):
 title = Field() 
 url = Field()
 size = Field()
 sizeType = Field()
 age = Field()
 seed = Field()
 leech = Field()
 torrent = Field()
pass

Kickass Spider

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.http import Request
from scrapy.utils.response import get_base_url

from kickass.items import TorrentItem

class KickassSpider(BaseSpider):

 name = "kickass"

 allowed_domains = [
  "kat.ph"
 ]

 def __init__(self, *args, **kwargs): 
  super(KickassSpider, self).__init__(*args, **kwargs)
  self.keywords = kwargs['keywords'].split(',')
  self.category = kwargs['category']
  self.start_urls = [
   'http://kat.ph/usearch/category%3A' 
   + self.category 
   + '/?field=time_add&sorder=desc'
  ]

 def parse(self, response):
  hxs = HtmlXPathSelector(response)
  entries = hxs.select('//tr[starts-with(@id,"torrent_category")]')
  items = []
  for entry in entries:
   item = TorrentItem()
   item['title'] = entry.select('td[1]/div[2]/a[2]/text()').extract()
   item['url'] = entry.select('td[1]/div[2]/a[2]/@href').extract()
   item['torrent'] = entry.select('td[1]/div[1]/a[starts-with(@title,"Download torrent file")]/@href').extract()
   item['size'] = entry.select('td[2]/text()[1]').extract()
   item['sizeType'] = entry.select('td[2]/span/text()').extract()
   item['age'] = entry.select('td[4]/text()').extract()
   item['seed'] = entry.select('td[5]/text()').extract()
   item['leech'] = entry.select('td[6]/text()').extract()   
   for s in self.keywords:
    if s.lower() in item['title'][0].lower():
     items.append(item)
     break
  return items
  

Torrent Pipeline

import json
import subprocess
import time
import urllib2

from scrapy.http.request import Request

class TorrentPipeline(object):

 def process_item(self, item, spider):  
   print 'Downloading ' + item['title'][0]
   path = 'http:'+item['torrent'][0]     
   subprocess.call(['./curl_torrent.sh',path])
   time.sleep(10) # pause to prevent 502 eror
   return item

ITEM_PIPELINES = ['kickass.pipelines.TorrentPipeline']

CURLing for the Torrent

#!/bin/bash
# Downloads .torrent files from kickass.com links
# following redirects and getting the actual torrent
# filename

AGENT="'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4)'"

function usage(){
 echo "Usage: 
 

How to automatically search and download torrents with Python and Scrapy


 



The Goal
To automatically perform keyword based searches at one of  kickasstorrents categories, scrap relevant data that match our keywords and category, download the .torrent file and push it to transmission torrent client for auto downloading .

Setup a cron job to repeat the search at intervals, scraping and downloading torrents automatically.



Check out the code directly from Github.

 

Example Test Cases
Search and download newly posted python books every morning at 09:00:




view plaincopy to clipboardprint?



	
0 */9 * * * cd ~/development/scrapy/kickass &&  /usr/local/bin/scrapy crawl kickass -a category=books -a keywords='python' >> ~/scrapy.log 2>&1  



0 */9 * * * cd ~/development/scrapy/kickass &&  /usr/local/bin/scrapy crawl kickass -a category=books -a keywords='python' >> ~/scrapy.log 2>&1



Search and automatically download latest X Men comics posted at kickasstorrents under comics category, every fifty (50) minutes. Setup the following cron job:




view plaincopy to clipboardprint?



	
*/50 * * * * cd ~/development/scrapy/kickass &&  /usr/local/bin/scrapy crawl kickass -a category=comics -a keywords='x-men,xmen,x men' >> ~/scrapy.log 2>&1  



*/50 * * * * cd ~/development/scrapy/kickass &&  /usr/local/bin/scrapy crawl kickass -a category=comics -a keywords='x-men,xmen,x men' >> ~/scrapy.log 2>&1



 

 

What we need
Three classes and the Scrapy framework: 

TorrentItem class to store torrent information

KickassSpider classto scrap torrent data

Pipilene class to follow URL redirects invoking curl and download torrent files



But first, let's install python, python dev libraries, libxml2 and Scrapy.

 

	
sudo apt-get install python - Python 2.6 or 2.7
	
Prerequisities for Scrapy
	
sudo apt-get install python-dev - python dev libraries
	
sudo apt-get install libxml2
	
pip install Scrapy or easy_install Scrapy - Scrapy framework


 

 

Create a new Scrapy project
After installing scrapy, create a new project from the command line:




view plaincopy to clipboardprint?



	
$ scrapy startproject kickass  



$ scrapy startproject kickass

This will create all necessary directories and provide initial structure for our project with default settings and some basic template classes.

 
 

Torrent Item
We need a class to store torrent data such as title, url, size etc.

Edit the existing items.py file in directory kickass/kickass:




view plaincopy to clipboardprint?



	
from scrapy.item import Item, Field   
	
  
	
class TorrentItem(Item):   
	
 title = Field()    
	
 url = Field()   
	
 size = Field()   
	
 sizeType = Field()   
	
 age = Field()   
	
 seed = Field()   
	
 leech = Field()   
	
 torrent = Field()   
	
pass  



from scrapy.item import Item, Field

class TorrentItem(Item):
 title = Field() 
 url = Field()
 size = Field()
 sizeType = Field()
 age = Field()
 seed = Field()
 leech = Field()
 torrent = Field()
pass


 

Kickass Spider
Next we define the Spider, responsible for scraping data and storing TorrentItem information.

We instantiate it with two arguments, category and keywords. Create a new file kickass_spider.py in directory kickass/kickass/spiders:




view plaincopy to clipboardprint?



	
from scrapy.spider import BaseSpider   
	
from scrapy.selector import HtmlXPathSelector   
	
from scrapy.http import Request   
	
from scrapy.utils.response import get_base_url   
	
  
	
from kickass.items import TorrentItem   
	
  
	
class KickassSpider(BaseSpider):   
	
  
	
 name = "kickass"  
	
  
	
 allowed_domains = [   
	
  "kat.ph"  
	
 ]   
	
  
	
 def __init__(self, *args, **kwargs):    
	
  super(KickassSpider, self).__init__(*args, **kwargs)   
	
  self.keywords = kwargs['keywords'].split(',')   
	
  self.category = kwargs['category']   
	
  self.start_urls = [   
	
   'http://kat.ph/usearch/category%3A'    
	
   + self.category    
	
   + '/?field=time_add&sorder=desc'  
	
  ]   
	
  
	
 def parse(self, response):   
	
  hxs = HtmlXPathSelector(response)   
	
  entries = hxs.select('//tr[starts-with(@id,"torrent_category")]')   
	
  items = []   
	
  for entry in entries:   
	
   item = TorrentItem()   
	
   item['title'] = entry.select('td[1]/div[2]/a[2]/text()').extract()   
	
   item['url'] = entry.select('td[1]/div[2]/a[2]/@href').extract()   
	
   item['torrent'] = entry.select('td[1]/div[1]/a[starts-with(@title,"Download torrent file")]/@href').extract()   
	
   item['size'] = entry.select('td[2]/text()[1]').extract()   
	
   item['sizeType'] = entry.select('td[2]/span/text()').extract()   
	
   item['age'] = entry.select('td[4]/text()').extract()   
	
   item['seed'] = entry.select('td[5]/text()').extract()   
	
   item['leech'] = entry.select('td[6]/text()').extract()      
	
   for s in self.keywords:   
	
    if s.lower() in item['title'][0].lower():   
	
     items.append(item)   
	
     break  
	
  return items   
	
    



from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.http import Request
from scrapy.utils.response import get_base_url

from kickass.items import TorrentItem

class KickassSpider(BaseSpider):

 name = "kickass"

 allowed_domains = [
  "kat.ph"
 ]

 def __init__(self, *args, **kwargs): 
  super(KickassSpider, self).__init__(*args, **kwargs)
  self.keywords = kwargs['keywords'].split(',')
  self.category = kwargs['category']
  self.start_urls = [
   'http://kat.ph/usearch/category%3A' 
   + self.category 
   + '/?field=time_add&sorder=desc'
  ]

 def parse(self, response):
  hxs = HtmlXPathSelector(response)
  entries = hxs.select('//tr[starts-with(@id,"torrent_category")]')
  items = []
  for entry in entries:
   item = TorrentItem()
   item['title'] = entry.select('td[1]/div[2]/a[2]/text()').extract()
   item['url'] = entry.select('td[1]/div[2]/a[2]/@href').extract()
   item['torrent'] = entry.select('td[1]/div[1]/a[starts-with(@title,"Download torrent file")]/@href').extract()
   item['size'] = entry.select('td[2]/text()[1]').extract()
   item['sizeType'] = entry.select('td[2]/span/text()').extract()
   item['age'] = entry.select('td[4]/text()').extract()
   item['seed'] = entry.select('td[5]/text()').extract()
   item['leech'] = entry.select('td[6]/text()').extract()   
   for s in self.keywords:
    if s.lower() in item['title'][0].lower():
     items.append(item)
     break
  return items
  


The spider, simply parses the first page of torrents for a given category sorted by age - most recent first.

Then extracts torrent information and if a keyword matches a torrent title is added to a list of TorrentItems to be later processed by the pipeline defined in the next step.

The URL for a given category sorted buy time looks like this:

http://kat.ph/usearch/category%3Abooks/?field=time_add&sorder=desc



 
Torrent Pipeline
All TorrentItems that were scrapped by the spider by matching the keyword list are passed to this pipeline for further processing. In our case, the pipeline will be responsible for downloading the actual torrent files and invoking transmission torrent client. Edit the file pipelines.py in directory kickass/kickass:

 



view plaincopy to clipboardprint?



	
import json   
	
import subprocess   
	
import time   
	
import urllib2   
	
  
	
from scrapy.http.request import Request   
	
  
	
class TorrentPipeline(object):   
	
  
	
 def process_item(self, item, spider):     
	
   print 'Downloading ' + item['title'][0]   
	
   path = 'http:'+item['torrent'][0]        
	
   subprocess.call(['./curl_torrent.sh',path])   
	
   time.sleep(10) # pause to prevent 502 eror   
	
   return item  



import json
import subprocess
import time
import urllib2

from scrapy.http.request import Request

class TorrentPipeline(object):

 def process_item(self, item, spider):  
   print 'Downloading ' + item['title'][0]
   path = 'http:'+item['torrent'][0]     
   subprocess.call(['./curl_torrent.sh',path])
   time.sleep(10) # pause to prevent 502 eror
   return item


Next, we must declare the new pipeline in kickass/kickass/settings.py configuration file. Add the following entry:

ITEM_PIPELINES = ['kickass.pipelines.TorrentPipeline']
 

CURLing for the Torrent
The pipeline gets the URL path from the scrapped TorrentItem and calls script curl_torrent.sh.

The script follows the URL and the redirection to get the real filename of the torrent and donwloads it. Then, it runs transmission to start the download.

Place the script under your kickass/ directory.




view plaincopy to clipboardprint?



	
#!/bin/bash   
	
# Downloads .torrent files from kickass.com links   
	
# following redirects and getting the actual torrent   
	
# filename   
	
  
	
AGENT="'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4)'"  
	
  
	
function usage(){  
	
 echo "Usage: $0 [Kickass Torrent URL]"  
	
  exit 1   
	
}  
	
  
	
if [ ! -n "$1" ]  
	
then  
	
    usage   
	
fi  
	
  
	
name=`echo $1 | sed 's/.*kat.ph.//'`".torrent"  
	
curl --globoff --compressed -A '$AGENT' -L --post302 $1 > $name  
	
transmission -m $name  



#!/bin/bash
# Downloads .torrent files from kickass.com links
# following redirects and getting the actual torrent
# filename

AGENT="'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4)'"

function usage(){
 echo "Usage: $0 [Kickass Torrent URL]"
  exit 1
}

if [ ! -n "$1" ]
then
    usage
fi

name=`echo $1 | sed 's/.*kat.ph.//'`".torrent"
curl --globoff --compressed -A '$AGENT' -L --post302 $1 > $name
transmission -m $name


 

Schedule the Spider

To start the spider we run scrapy with the crawl command and the name of the spider, in our case kickass.

However, we need to supply two arguments. One for category, and a list of keywords.

 

For example: 

scrapy crawl kickass -a category=books -a keywords='python,java,scala topics'

 

To have the spider run every 10 minutes we can schedule a cron job

.

From the command line type crontab -e and add the following  line:

 

*/10 * * * * cd ~/development/scrapy/kickass &&  /usr/local/bin/scrapy crawl kickass -a category=books -a keywords='python,java,scala topics' >> ~/scrapy.log 2>&1

 

 


Considerations

Finally, is recommended to modify the setting.py file under kickass/kickass directory to tune the spiders behavior and adjust logging. The following settings introduce a download delay of 5 seconds per request and limit concurrent requests to 1. That is to prevent hammering. Here is the complete file:



# Scrapy settings for kickass project

BOT_NAME = 'kickass'

SPIDER_MODULES = ['kickass.spiders']
NEWSPIDER_MODULE = 'kickass.spiders'
ITEM_PIPELINES = ['kickass.pipelines.TorrentPipeline',]

# Download and traffic settings.
# Limit concurrent requests and add a 
# download delay to minimize hammering.
USER_AGENT = 'http://www.kickasstorrents.com)'
DOWNLOAD_DELAY = 5
RANDOMIZE_DOWNLOAD_DELAY = False
CONCURRENT_REQUESTS_PER_DOMAIN = 1 #  Default: 8
#SCHEDULER = 'scrapy.core.scheduler.Scheduler'

# Log Settings
LOG_ENABLED = True
LOG_LEVEL = 'INFO' # Levels: CRITICAL, ERROR, WARNING, INFO, DEBUG
LOG_FILE = './kickass.log'



 


This is my first attempt at doing anything with python so i guess some things could be done more efficiently. I am still experimenting with the language, and coming from a heavy Java background i can confess that i am fascinated. Also, i am pretty sure that spawning a new process with curl to fetch the torrent is not the most optimal way to do it.



Feel free to directly checkout the code at Github, and point out improvement / corrections.

I would very much appreciate that.

 

[출처] http://fuzz-box.blogspot.kr/2013/03/how-to-automatically-search-download-torrent-python-scrapy.html