 alpcentaur
|
ece5cf1301
|
added better logic for getting the right link of entry
|
11 months ago |
 alpcentaur
|
0e58756600
|
added last resort exception for entry page downloading with wget, also implemented some further logic regarding getting the right links
|
11 months ago |
|
alpcentaur
|
16199256e3
|
javascript on highest level done better
|
11 months ago |
|
alpcentaur
|
5627c80177
|
merged onlinkgen with master, and added more universal chrome driver initialization to the beginning of the javascript entries gothrough function in download_entry_list_pages_of_funding_databases()
|
11 months ago |
|
alpcentaur
|
14b8db7941
|
started adding javascript handling on highest spider level
|
11 months ago |
|
alpcentaur
|
fbee5d6229
|
last commit in detached head
|
11 months ago |
|
alpcentaur
|
953f85ee5b
|
added new lines to chromedriver, to make it work on other systems
|
11 months ago |
|
alpcentaur
|
d2324d265a
|
added pdf child text downloading and parse to json exceptions/cases for javascript entry data and normal data
|
11 months ago |
|
alpcentaur
|
885c210971
|
added selenium for pop up entry links
|
11 months ago |
|
alpcentaur
|
ec180bed0a
|
added flow for selenium grabbing popup instead of links for entries
|
11 months ago |
|
alpcentaur
|
b4fd385c5d
|
did some changes to main.py for using sys.argv
|
11 months ago |
|
alpcentaur
|
99c74dcbad
|
updated requirements.txt
|
11 months ago |
|
alpcentaur
|
54daad8dfa
|
started sys arguments for main.py, to be able to control spider from interface
|
11 months ago |
|
alpcentaur
|
89dcca2031
|
added further handling for javascript links not being urls, made config for giz work
|
11 months ago |
|
alpcentaur
|
a0075e429d
|
added further database in config.yaml, added new exception for downloading js generated html pages
|
1 year ago |
|
alpcentaur
|
df4a8289b8
|
added pdf parser if entry link is direct pdf
|
1 year ago |
|
alpcentaur
|
677e54c0c2
|
added trafilatura to requirements
|
1 year ago |
|
alpcentaur
|
9ceaa28a82
|
Merge remote-tracking branch 'refs/remotes/origin/master'
merging with a small README.md change
|
1 year ago |
|
alpcentaur
|
d3335f203b
|
added trafilatura exception
|
1 year ago |
|
alpcentaur
|
61f9ba67fb
|
update README.md
|
1 year ago |
|
alpcentaur
|
69c517292b
|
Update 'README.md'
|
1 year ago |
|
alpcentaur
|
14ece9bceb
|
added functions for uniform and not uniform entry end points - non uniform endpoints are generally parsed as text from any paragraph xml element p
|
1 year ago |
|
alpcentaur
|
b2cf4b67ce
|
added first config parameters for search on not uniform entries
|
1 year ago |
|
alpcentaur
|
42841ee650
|
added some exceptions for bad encoding and get errors
|
1 year ago |
|
alpcentaur
|
317ef99720
|
changed code in entrylist data2dictionary to handle empty or missing xml elements
|
1 year ago |
|
alpcentaur
|
ff23c22e3c
|
added working bund.de-bekanntmachungen config with new example of xpath contains
|
1 year ago |
|
alpcentaur
|
06fa81e549
|
added function find config parameter and changed core spider
|
1 year ago |
|
alpcentaur
|
a846ce04cc
|
specifying the links, new exception clause if soupparser does not work
|
1 year ago |
|
alpcentaur
|
a99881796a
|
first function works, actuall xml parser has still problems with certain xml types
|
1 year ago |
|
alpcentaur
|
c078ee4b1b
|
first function works, actuall xml parser has still problems with certain xml types
|
1 year ago |
|
alpcentaur
|
8b20bc178f
|
added multi pages configuration and code
|
1 year ago |
|
alpcentaur
|
7aa903883b
|
update to config.yaml
|
1 year ago |
|
alpcentaur
|
59838bb8e1
|
added main.py importing and using the spider functions
|
1 year ago |
|
alpcentaur
|
5ac07d151a
|
added first config.yaml template and started creating folder structure
|
1 year ago |
|
alpcentaur
|
b3011efc73
|
small change of naming in error message added
|
1 year ago |
|
alpcentaur
|
687d40f156
|
first change of naming, first commit for the actual spider based on importPEP
|
1 year ago |
|
alpcentaur
|
8783251133
|
first commit
|
1 year ago |
