fdb-spider

1.5 MiB

Author	SHA1	Message	Date
alpcentaur	0e58756600	added last resort exception for entry page downloading with wget, also implemented some further logic regarding getting the right links	10 months ago
alpcentaur	fbee5d6229	last commit in detached head	11 months ago
alpcentaur	d2324d265a	added pdf child text downloading and parse to json exceptions/cases for javascript entry data and normal data	11 months ago
alpcentaur	ec180bed0a	added flow for selenium grabbing popup instead of links for entries	11 months ago
alpcentaur	b4fd385c5d	did some changes to main.py for using sys.argv	11 months ago
alpcentaur	54daad8dfa	started sys arguments for main.py, to be able to control spider from interface	11 months ago
alpcentaur	a0075e429d	added further database in config.yaml, added new exception for downloading js generated html pages	11 months ago
alpcentaur	df4a8289b8	added pdf parser if entry link is direct pdf	11 months ago
alpcentaur	14ece9bceb	added functions for uniform and not uniform entry end points - non uniform endpoints are generally parsed as text from any paragraph xml element p	11 months ago
alpcentaur	42841ee650	added some exceptions for bad encoding and get errors	1 year ago
alpcentaur	ff23c22e3c	added working bund.de-bekanntmachungen config with new example of xpath contains	1 year ago
alpcentaur	c078ee4b1b	first function works, actuall xml parser has still problems with certain xml types	1 year ago
alpcentaur	59838bb8e1	added main.py importing and using the spider functions	1 year ago