alpcentaur/fdb-spider
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
45 commits 2 branches 2 tags 805 KiB
Python 100%
Find a file
alpcentaur 0500f5853d full working example from localhost
2024-01-15 21:08:23 +00:00
spiders full working example from localhost 2024-01-15 21:08:23 +00:00
.gitignore added other exception for unitrue var text not being found, before saving index 0 to variable produced error to whole execution 2024-01-10 15:28:41 +00:00
main.py full working example from localhost 2024-01-15 21:08:23 +00:00
README.md added instruction for downloading chromium driver for python selenium to README.md 2023-12-15 14:13:41 +01:00
requirements.txt last commit in detached head 2023-12-13 16:20:27 +01:00

README.md 

  __     _ _                     _     _
 / _| __| | |__        ___ _ __ (_) __| | ___ _ __
| |_ / _` | '_ \ _____/ __| '_ \| |/ _` |/ _ | '__|
|  _| (_| | |_) |_____\__ | |_) | | (_| |  __| |
|_|  \__,_|_.__/      |___| .__/|_|\__,_|\___|_|
                          |_|

Configure fdb-spider in a yaml file. Spider Multi page databases of links. Filter and serialize content to json.

Filter either by xpath syntax. Or Filter with the help of Artificial Neural Networks (work in progress).

To run this, create a python3 virtualenv, pip install -r requirements, and

apt install xvfb
apt install chromium
apt install chromium-webdriver