fdb-spider

alpcentaur/fdb-spider

Fork 0

Commit graph

2aa1134b48 updated gitignore master alpcentaur 2024-03-05 14:56:03 +0000
a9c2346c04 first change to also click Accept Button in English if may come for js spidering functionality alpcentaur 2024-03-05 14:50:34 +0000
0808e5a42d main.py and config.yaml are left out from updates, only examples are provided. Change in Readme too alpcentaur 2024-03-05 14:42:30 +0000
4ec9f76080 added xorg-server-xephyr as dep to install alpcentaur 2024-03-05 14:52:02 +0100
10cdab6f60 updated README with new and working install order alpcentaur 2024-03-05 14:43:43 +0100
ccfe20044f added another tip to README.md, header for display and another tip added too alpcentaur 2024-03-05 12:34:32 +0100
0fa420d74c added explanation of display variable in the spiders code alpcentaur 2024-03-05 12:30:29 +0100
0d7728240e update var javascriptlink in README.md alpcentaur 2024-03-04 17:13:33 +0100
c52ea0cf0a added example1 for js configuration in README.md alpcentaur 2024-03-04 16:46:57 +0100
5000dca314 Update README.md with better explanation how to js spider alpcentaur 2024-03-04 16:30:31 +0100
0908ccf6e5 clarifications for javascript link and js link plus js iteration alpcentaur 2024-03-03 18:24:07 +0100
ff0fe5193d fixed the links for the clickable content summary alpcentaur 2024-03-03 17:57:48 +0100
49d5c2ffa9 third try ordering alpcentaur 2024-03-03 17:53:05 +0100
f489106ea0 second try ordering alpcentaur 2024-03-03 17:50:12 +0100
32fceffd01 searchable headers for step by step guide started alpcentaur 2024-03-03 17:48:52 +0100
eca77f9b63 Step by Step Guide continuation of describing the variables alpcentaur 2024-03-01 00:09:38 +0100
483eaec26e changed domain for new configuration dtvp alpcentaur 2024-02-29 14:19:45 +0100
c33dbc37e6 Merge remote-tracking branch 'refs/remotes/origin/master' Merging local changes to the code with changes to the README.md on gitea instance alpcentaur 2024-02-29 13:16:48 +0000
a07d2e93f6 changes for new database dtvp, new exceptions trying to click away cookie pop ups alpcentaur 2024-02-29 13:15:34 +0000
d284fef015 changes for new database dtvp, new exceptions trying to click away cookie pop ups alpcentaur 2024-02-29 13:15:01 +0000
5fd6b7f781 Part 2 of Step by Step Guide alpcentaur 2024-02-28 17:34:57 +0100
e4fa13d29d Start of Step by Step Guide alpcentaur 2024-02-28 17:17:27 +0100
7ba196b0c2 changed size of virtual window, added some scrolling and shortened the time for js lazy loading enforced slow downloading alpcentaur 2024-02-11 17:08:33 +0000
a56569712e another small change to config.yaml before pushing alpcentaur 2024-02-11 16:43:44 +0000
a0dd469f25 added new database ted.europe.eu, created new case of slow downloading, intergrated scrolling into entrylistpagesdownload alpcentaur 2024-02-09 18:38:49 +0000
094f092291 deleted fdb entry that was a ghost for syntax reasons, but same syntax should be in other fdb anyway alpcentaur 2024-01-23 17:17:40 +0100
d7d157bf42 added further dokumentation to README.md alpcentaur 2024-01-21 14:07:38 +0000
0500f5853d full working example from localhost alpcentaur 2024-01-15 21:08:23 +0000
0411d74936 deleted config.yaml.save alpcentaur 2024-01-15 19:12:04 +0000
cf3bb52684 corrected link glueing for pdf links for loop alpcentaur 2024-01-15 19:09:28 +0000
af8374f715 added other exception for unitrue var text not being found, before saving index 0 to variable produced error to whole execution alpcentaur 2024-01-10 15:28:41 +0000
20db0028e1 added first changes to fix js related bug for giz db alpcentaur 2024-01-10 15:18:36 +0100
dec60f9bf5 added changed logic for link addition regarding entry links alpcentaur 2023-12-18 21:26:53 +0000
5d17f4e421 corrected error which arised in logic of wget backup get alpcentaur 2023-12-15 14:36:08 +0100
92c238a2ed added instruction for downloading chromium driver for python selenium to README.md alpcentaur 2023-12-15 14:13:41 +0100
ece5cf1301 added better logic for getting the right link of entry alpcentaur 2023-12-15 13:34:23 +0100
0e58756600 added last resort exception for entry page downloading with wget, also implemented some further logic regarding getting the right links alpcentaur 2023-12-15 11:33:50 +0000
16199256e3 javascript on highest level done better alpcentaur 2023-12-14 23:37:10 +0000
5627c80177 merged onlinkgen with master, and added more universal chrome driver initialization to the beginning of the javascript entries gothrough function in download_entry_list_pages_of_funding_databases() alpcentaur 2023-12-14 12:38:14 +0000
14b8db7941 started adding javascript handling on highest spider level alpcentaur 2023-12-14 12:07:14 +0000
fbee5d6229 last commit in detached head onlinkgen alpcentaur 2023-12-13 16:20:27 +0100
953f85ee5b added new lines to chromedriver, to make it work on other systems alpcentaur 2023-12-13 16:05:26 +0100
d2324d265a added pdf child text downloading and parse to json exceptions/cases for javascript entry data and normal data alpcentaur 2023-12-06 13:46:54 +0000
885c210971 added selenium for pop up entry links alpcentaur 2023-12-05 22:19:00 +0000
ec180bed0a added flow for selenium grabbing popup instead of links for entries alpcentaur 2023-12-05 22:16:07 +0000
b4fd385c5d did some changes to main.py for using sys.argv alpcentaur 2023-12-05 18:23:57 +0100
99c74dcbad updated requirements.txt alpcentaur 2023-12-05 16:59:13 +0000
54daad8dfa started sys arguments for main.py, to be able to control spider from interface alpcentaur 2023-12-05 17:51:16 +0100
89dcca2031 added further handling for javascript links not being urls, made config for giz work v2 v1 alpcentaur 2023-11-28 15:27:39 +0000
a0075e429d added further database in config.yaml, added new exception for downloading js generated html pages alpcentaur 2023-11-27 15:10:11 +0000
df4a8289b8 added pdf parser if entry link is direct pdf alpcentaur 2023-11-22 17:03:15 +0000
677e54c0c2 added trafilatura to requirements alpcentaur 2023-11-22 00:07:59 +0000
9ceaa28a82 Merge remote-tracking branch 'refs/remotes/origin/master' merging with a small README.md change alpcentaur 2023-11-22 00:04:16 +0000
d3335f203b added trafilatura exception alpcentaur 2023-11-22 00:02:29 +0000
61f9ba67fb update README.md alpcentaur 2023-11-20 16:38:18 +0100
69c517292b Update 'README.md' alpcentaur 2023-11-20 16:37:27 +0100
14ece9bceb added functions for uniform and not uniform entry end points - non uniform endpoints are generally parsed as text from any paragraph xml element p alpcentaur 2023-11-20 15:28:04 +0000
b2cf4b67ce added first config parameters for search on not uniform entries alpcentaur 2023-11-15 17:27:54 +0000
42841ee650 added some exceptions for bad encoding and get errors alpcentaur 2023-11-14 14:38:45 +0000
317ef99720 changed code in entrylist data2dictionary to handle empty or missing xml elements alpcentaur 2023-11-14 10:22:26 +0000
ff23c22e3c added working bund.de-bekanntmachungen config with new example of xpath contains alpcentaur 2023-11-13 16:44:11 +0000
06fa81e549 added function find config parameter and changed core spider alpcentaur 2023-11-10 01:12:49 +0000
a846ce04cc specifying the links, new exception clause if soupparser does not work alpcentaur 2023-11-07 14:55:05 +0000
a99881796a first function works, actuall xml parser has still problems with certain xml types alpcentaur 2023-11-06 19:19:31 +0000
c078ee4b1b first function works, actuall xml parser has still problems with certain xml types alpcentaur 2023-11-06 19:17:45 +0000
8b20bc178f added multi pages configuration and code alpcentaur 2023-11-06 18:17:32 +0000
7aa903883b update to config.yaml alpcentaur 2023-11-03 12:23:04 +0000
59838bb8e1 added main.py importing and using the spider functions alpcentaur 2023-11-02 10:54:16 +0000
5ac07d151a added first config.yaml template and started creating folder structure alpcentaur 2023-10-31 17:41:44 +0000
b3011efc73 small change of naming in error message added alpcentaur 2023-10-30 16:43:32 +0000
687d40f156 first change of naming, first commit for the actual spider based on importPEP alpcentaur 2023-10-30 16:41:14 +0000
8783251133 first commit alpcentaur 2023-10-30 14:35:32 +0000