32 Commits (onlinkgen)
 

Author SHA1 Message Date
  alpcentaur fbee5d6229 last commit in detached head 9 months ago
  alpcentaur 953f85ee5b added new lines to chromedriver, to make it work on other systems 9 months ago
  alpcentaur d2324d265a added pdf child text downloading and parse to json exceptions/cases for javascript entry data and normal data 9 months ago
  alpcentaur 885c210971 added selenium for pop up entry links 9 months ago
  alpcentaur ec180bed0a added flow for selenium grabbing popup instead of links for entries 9 months ago
  alpcentaur b4fd385c5d did some changes to main.py for using sys.argv 9 months ago
  alpcentaur 99c74dcbad updated requirements.txt 9 months ago
  alpcentaur 54daad8dfa started sys arguments for main.py, to be able to control spider from interface 9 months ago
  alpcentaur 89dcca2031 added further handling for javascript links not being urls, made config for giz work 9 months ago
  alpcentaur a0075e429d added further database in config.yaml, added new exception for downloading js generated html pages 9 months ago
  alpcentaur df4a8289b8 added pdf parser if entry link is direct pdf 10 months ago
  alpcentaur 677e54c0c2 added trafilatura to requirements 10 months ago
  alpcentaur 9ceaa28a82 Merge remote-tracking branch 'refs/remotes/origin/master' 10 months ago
  alpcentaur d3335f203b added trafilatura exception 10 months ago
  alpcentaur 61f9ba67fb update README.md 10 months ago
  alpcentaur 69c517292b Update 'README.md' 10 months ago
  alpcentaur 14ece9bceb added functions for uniform and not uniform entry end points - non uniform endpoints are generally parsed as text from any paragraph xml element p 10 months ago
  alpcentaur b2cf4b67ce added first config parameters for search on not uniform entries 10 months ago
  alpcentaur 42841ee650 added some exceptions for bad encoding and get errors 10 months ago
  alpcentaur 317ef99720 changed code in entrylist data2dictionary to handle empty or missing xml elements 10 months ago
  alpcentaur ff23c22e3c added working bund.de-bekanntmachungen config with new example of xpath contains 10 months ago
  alpcentaur 06fa81e549 added function find config parameter and changed core spider 10 months ago
  alpcentaur a846ce04cc specifying the links, new exception clause if soupparser does not work 10 months ago
  alpcentaur a99881796a first function works, actuall xml parser has still problems with certain xml types 10 months ago
  alpcentaur c078ee4b1b first function works, actuall xml parser has still problems with certain xml types 10 months ago
  alpcentaur 8b20bc178f added multi pages configuration and code 10 months ago
  alpcentaur 7aa903883b update to config.yaml 10 months ago
  alpcentaur 59838bb8e1 added main.py importing and using the spider functions 10 months ago
  alpcentaur 5ac07d151a added first config.yaml template and started creating folder structure 10 months ago
  alpcentaur b3011efc73 small change of naming in error message added 10 months ago
  alpcentaur 687d40f156 first change of naming, first commit for the actual spider based on importPEP 10 months ago
  alpcentaur 8783251133 first commit 10 months ago