alpcentaur
/
fdb-spider
1
0
Fork 0
Code Issues 0 Pull Requests 0 Projects 0 Releases 2 Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
42 Commits
2 Branches
1.5 MiB
 
Tree: af8374f715
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'af8374f715'
${ noResults }
 ZIP  TAR.GZ
alpcentaur af8374f715 added other exception for unitrue var text not being found, before saving index 0 to variable produced error to whole execution 10 months ago
spiders added other exception for unitrue var text not being found, before saving index 0 to variable produced error to whole execution 10 months ago
.gitignore added other exception for unitrue var text not being found, before saving index 0 to variable produced error to whole execution 10 months ago
README.md added instruction for downloading chromium driver for python selenium to README.md 10 months ago
main.py added other exception for unitrue var text not being found, before saving index 0 to variable produced error to whole execution 10 months ago
requirements.txt last commit in detached head 11 months ago

README.md 

  __     _ _                     _     _
 / _| __| | |__        ___ _ __ (_) __| | ___ _ __
| |_ / _` | '_ \ _____/ __| '_ \| |/ _` |/ _ | '__|
|  _| (_| | |_) |_____\__ | |_) | | (_| |  __| |
|_|  \__,_|_.__/      |___| .__/|_|\__,_|\___|_|
                          |_|

Configure fdb-spider in a yaml file. Spider Multi page databases of links. Filter and serialize content to json.

Filter either by xpath syntax. Or Filter with the help of Artificial Neural Networks (work in progress).

To run this, create a python3 virtualenv, pip install -r requirements, and

apt install xvfb
apt install chromium
apt install chromium-webdriver