SVN: toys/rsget.pl/README

[sparky]

Author: sparky
Date: Tue Oct  6 01:57:10 2009
New Revision: 10696

Modified:
   toys/rsget.pl/README
Log:
- get.list documentation


Modified: toys/rsget.pl/README
==============================================================================
--- toys/rsget.pl/README	(original)
+++ toys/rsget.pl/README	Tue Oct  6 01:57:10 2009
@@ -17,10 +17,108 @@
 For full list of command-line options check: rsget.pl --help. All those options
 may also be set in config file ( $HOME/.rsget.pl/config ). Check 'README.config'
 file for example config.
-Each option requires a value, which may be specified inmediately after '=' sign,
+Each option requires a value, which may be specified immediately after '=' sign,
 or as next argument. Underscores may be replaced with dashes.
 Following command-line declarations are equivalent:
  --use_svn=update
  --use-svn=update
  --use_svn update
  --use-svn update
+
+
+============================= URI list / get.list =============================
+
+Understanding URI list syntax may be very important for people unable or not
+willing to use HTTP interface. It also is crucial if writing any software
+for manipulating the list.
+
+Empty lines and lines starting with '#' are ignored. If line does not start
+with '#', the sign and text after it are treated as part of the line, and not
+as a comment.
+
+Long lines may be truncated between any two words in following manners:
+- Append '\' at the end of unfinished line:
+	word1 word2 \
+	word3 \
+	word4 word5
+- Start added line with '+' sign (preferred):
+	word1 word2
+	+ word3
+	+ word4 word5
+
+Each line specify one file to download, it is composed of following parts:
+
+[COMMAND:] [GLOBAL_OPTS] URI1 [URI1_OPTS] [URI2 [URI2_OPTS] [URI3 ...]]
+
+When introducing new source URI1 is the only part required. URIs must valid
+identifiers, as seen in location bar in your web browser; i.e. they may not
+contain spaces, replace them with '%20'.
+
+COMMAND may be one of: GET, DONE, STOP or ADD. Note the upper case, and colon
+sign (:) after command. If none specified GET is assumed. Their meanings are:
+- GET: download the file, or continue partially downloaded one
+- DONE: file download completed, don't try to download
+- STOP: file download aborted or never started, don't try to download
+- ADD: try to find other uris on the list with the same name and size and add
+    this uri as clone, if none found add it as new source. Command possibly
+    dangerous, SEE (1)
+
+GLOBAL_OPTS are options applied to all URIs for this file. Options specified
+after URI are applied only to the URI immediately proceeding them. Each option
+has syntax: 'name=value', where both name and value must be URI-encoded, in most
+cases it is enough to replace '%' sign and any white space with corresponding
+'%XX' code. There are many options saved by rsget.pl itself, but only a few
+which should be introduced, before starting download, by the user:
+- (global) dir=VALUE - directory within workdir and outdir where downloaded
+    file should be placed
+- (local) pass=VALUE - file password, sometimes required to download a file,
+    SEE (2)
+
+Some of automatic options (saved by rsget.pl):
+- (global) fname=VALUE - exact name of file, set after starting download
+- (global) fsize=VALUE - exact size of file, set after starting download
+- (local) error=VALUE - reason of last failure, URI won't be considered if it
+    contains error option, if all URIs for file have errors COMMAND will change
+    to STOP
+- (local) name, iname, aname, ainame - name of file as reported by the website
+- (local) size, asize - size of file as reported by the website
+
+rsget.pl always writes each line in the following style:
+	COMMAND: [GLOBAL_OPTS]
+	+ URI1 [URI1_OPTS]
+	+ URI2 [URI2_OPTS]
+	...
+
+NOTE to programmers:
+  rsget.pl rereads list file each time it is willing to change something,
+or file's mtime has changed. For that matter you should newer write directly to
+that file, unless you are appending only.
+  On the other hand, rsget.pl will not make any changes to the list file if lock
+file exists. There is one exception tho: if new sources are added from HTTP
+interface and lock file exists it is removed and new URIs are written.
+  Assuming your list_file is get.list, and lock file used is: get.list.lock,
+you should:
+	touch get.list.lock # this ensures consistency
+	read get.list
+	process
+	if needs update:
+	    write get.list.new
+	    if exists get.list.lock:
+	        rename get.list.new to get.list
+	    else: # get.list has changed
+	        restart from beginning
+	remove get.list.lock
+
+
+NOTES:
+ (1) Some download services (e.g. MegaUpload) abbreviate file names on their
+download pages, which may lead to incorrect clone guesses. If there is a three
+part rar, first two parts will have the same size, and same abbreviated name.
+ADD command will think both links specify the same file. Moreover MegaUpload
+allows continuing partial downloads, so after restarting rsget.pl you may end up
+with a broken file containing data chunks from both sources.
+
+ (2) It is not user password nor rar file password. Some sites (in fact
+MegaUpload is the only one) may require to introduce resource password to be
+able to download a file. That password is set by the uploader, separately for
+each resource, and it should be provided by the uploader along with the URIs.