| spiders | ||
| .gitignore | ||
| main.py | ||
| main.py_example | ||
| README.md | ||
| requirements.txt | ||
| spider.log | ||
__ _ _ _ _
/ _| __| | |__ ___ _ __ (_) __| | ___ _ __
| |_ / _` | '_ \ _____/ __| '_ \| |/ _` |/ _ | '__|
| _| (_| | |_) |_____\__ | |_) | | (_| | __| |
|_| \__,_|_.__/ |___| .__/|_|\__,_|\___|_|
|_|
Introduction
The fdb-spider was made to gather data from Websites in an automated way. The Website to be spidered has to be a list of Links. Which makes the fdb-spider a web spider for most Plattforms. The fdb-spider is to be configured in a .yaml file to make things easy. The output of the fdb-spider is in json format to make it easy to input the json to other programs.
At its core, the spider outputs tag search based entries
It works together with the fdb-spider-interface.
In Future, the spider will be extended by the model Sauerkraut. An !open source! Artificial Neural Network.
Installation
FIRST install systemwide requirements with your package manager
# apt based unixoids
apt install xvfb
apt install chromium
apt install chromium-webdriver
# pacman based unixoids
pacman -S xorg-server-xvfb
pacman -S xorg-server-xephyr
pacman -S chromium
THEN create a python3 virtualenv on your favourite UNIX Distribution with the command
git clone https://code.basabuuka.org/alpcentaur/fdb-spider
cd fdb-spider
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
Usage
Use it step by step. First care for the htmls of the lists of the links. Then care for getting the first json output from the first layer of html pages.
Copy the two examples to the file name, in which they will be input to the spider
cp main.py_example main.py
cp spiders/config.yaml_example spiders/config.yaml
Configuration File Syntax
The configuration file with working syntax template is
/spiders/config.yaml
Here you can configure new websites to spider, referred to as "databases".
link1 and link2 are the links to be iterated. The assumption is, that every list of links will have a loopable structure. If links are javascript links, specify js[domain,link[1,2],iteration-var-list]. Otherwise leave them out, but specify jsdomain as 'None'.
You will find parents and children of the entry list pages. Here you have to fill in the xpath of the entries to be parsed.
In the entry directive, you have to specify uniform to either TRUE or FALSE. Set it to TRUE, if all the entry pages have the same template, and you are able to specify xpath again to get the text or whatever variable you like to specify. In the entry_unitrue directive, you can specify new dimensions and the json will adapt to your wishes. Under the entry-list directive this feature has to be still implemented. So use name, link, javascript-link, info, period and sponsor by commenting in or out. If javascript-link is set (which means its javascript clickable), link will be ignored.
Set it to FALSE, if you have diverse pages behind the entries, and want to generally get the main text of all the different links. For your information, the library trafilature is used to gather the text generally for further processing.
Efficient Xpath Copying
When copying the Xpath, most modern Webbrowsers are of help. In Firefox (or Browsers build on it like the TOR Browser) you can use
strl-shift-c
to open the "Inspector" in "Pick an element" mode. When you click on the desired entry on the page, it opens the actual code of the clicked element in the html search box. Now make a right click on the code in the html search box, go on "Copy", and go on XPath. Now you have the XPath of the element in your clipboard. When pasting it into the config, try to replace some slashes with double slashes. That will make the spider more stable, in case the websites html/xml gets changed for maintenance or other reasons.
Step By Step Guide
Start with an old Configuration that is similar to what you need.
There are Three Types of Configurations:
The first Type is purely path based. An example is greenjobs.de. The second Type is a mixture of path and javascript functions, giz is an example for this Type. The third Type is purely javascript based. An example is ted.europe.eu.
Type 1:
Start with collecting every variable. From up to down.
var domain
domain is the variable for the root of the website. In case links are glued, they will be glued based on the root.
var entry list
Now come all the variables regarding the entry list pages.
var link and iteration
In Pseudo Code, whats happening with these three variables is
for n in iteration var list:
get(link1 glued to n glued to link2)
So if you are on the no javascript side of reality, you are lucky. Thats all needed to get the collection of links.
example1 link
Lets say we go on greenjobs.de. We go on search without search query. To get the biggest displayed output, in best case a table of everything the site has listed.
https://www.greenjobs.de/angebote/index.html?s=&loc=&countrycode=de&dist=10&lng=&lat= is the resulting url.
So now we navigate through the pages. In this case everything is displayed and scrollable on exactly this url. Which means, we leave link2 and iteration var list empty. And put the resulting url into link1.
example2 link
This time we go on giz. There we have https://ausschreibungen.giz.de/Satellite/company/welcome.do as our url for a general search. If I go on the "nextpage" button of the displayed table, a new url pattern appears being on the next page:
Going on the next side again, we get the url:
So now we already see the pattern, that any and every machine generated output cant hide.
RSULT=1 .... we put it in the url bar of the browser
and get to the first pages. Which leads to the following variables, considering that there were 6 pages:
- link1 = "https://ausschreibungen.giz.de/Satellite/company/welcome.do?method=showTable&fromSearch=1&tableSortPROJECT_RESULT=2&tableSortAttributePROJECT_RESULT=publicationDate&selectedTablePagePROJECT_RESULT="
- link2 = ""
- iteration-var-list = "[1,2,3,4,5,6]"
javascript
It happens, that it is not possible to see a pattern in the urls. Probably because the website hoster is not smart or just a thief in a bad sense. In this case you only get html gibberish. To get the desired info with the help of this program, you have the possibility to give the config.yaml paths of clickable items. The spider will open an axctual browser and click through the pages that start to exist.
var jsdomain
If jsdomain is "None" (And here it is important to use None and not NONE), it will try to get the domains and download elements based on the gets with a variety of libraries. But if you have a javascript situation, where the first html are already javascript generated kaos without xml to parse.. then you need to put an url here. By putting the url, the spider will open that website with a virtual graphical browser (using selenium), wait for the javascript to load, and by clicking on jslink, go through the pages.
In pseudo code that means, if you fill out the variable jsdomain in the config.yaml, the spider will do
for i in jsiteration-var-list:
click on the jslink string glued to i glued to jslink2
vars jslink and jsiteration
In jslink1 and jslink2 you have to put the xpath of the button that clicks to the next site of entry links to download.
Sometimes the xpath changes, after the new js content got loaded. That is where the jsiteration-var-list comes in. And like this, you define the button that gets klicked every site. Sometimes it stays the same, then you just need an occurence of the same number exactly the same times as in the var-iteration, which will define how many pages will be downloaded generally. The var iteration-var-list defines the folder structure of the json output files.
Which means we emulate a whole virtual "user" using a virtual "browser" on his or her virtual "screen". In the end the clickable elements are defined by xpath too, so put these accordingly in the jslink and jsiteration variables.
You can run the spider with display=1 instead of display=0 in the python line of the virtual display the chromium driver is running on. I will put that in the initialization of the spider. How to do this, in general if you use any of the js related variables instead of setting "NONE" (important: and not "None"), will be described in the paragraph display. It is very useful to debug js related configs.
With running the spider while watching the automated mouse moves and clicks, you will be able to find the right xpath for every step and element.
example1 jslink and jsiteration
So let us consider evergabe-online as an example.
evergabe-online:
domain: 'https://www.evergabe-online.de/'
entry-list:
link1: 'https://www.evergabe-online.de/search.html?101-1.-searchPanel>
link2: '-pageLink'
jsdomain: 'https://www.evergabe-online.de/search.html'
jslink1: '/html/body/div[8]/main/div[4]/div/div/div[2]/table/thead/tr[1]/td/div[2]/div/span['
jslink2: ']'
jsiteration-var-list: "[1,2, 3, 4, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
iteration-var-list: "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
Go on jsdomain https://www.evergabe-online.de/search.html. You will see the table we want to spider. Open the inspector, and have a look at the button to get to the next site. Its xpath is '/html/body/div[8]/main/div[4]/div/div/div[2]/table/thead/tr[1]/td/div[2]/div/span[1]'
Now we click on it. On page two, the button to click us to page three has the xpath: '/html/body/div[8]/main/div[4]/div/div/div[2]/table/thead/tr[1]/td/div[2]/div/span[2]'
From page 5 on, the button to get to the next pages stays '/html/body/div[8]/main/div[4]/div/div/div[2]/table/thead/tr[1]/td/div[2]/div/span[6]'
until the end.
display
When you run the spider with js spidering enabled, in fdb_spider.py a display will get created. If you open nano, and press crtl+w, you can type display and enter. This will bring you to the lines of code, generating the display. Watch out for the line
display = Display(visible=0, size=(1200, 800))
If you change visible=0 to visible=1 here, the spider will run with actually open a viewable browser on the workspace.
This line is present two times in the code. One for downloading the pages with the links, and one for downloading the pages of/behind the links.
After finding and jumping to "display" with ctrl-w, issue alt+w to get to the next display variable definition.
another tip
In main.py, where the spiders code gets loaded, you have also a function available called
spider.find_config_parameter(list_of_fdbs)
This function helps you to find the right config parameters, because it shows you what you get. That can be a bit tricky, because if you get nothing, it does not really help you. But when you have the first ones running, it shows you exactly that: That its always possible and no magic needed.
var parent
The parent stands for the last xml element which contains the entry links. Go with the Inspector on the entry respectively one of the links, klick on it and klick in the code view afterwards. Now use the arrow up key to get to the last child before it comes to the parent. You can see it on the rendered html blue contained.
From the last child, when it is selected in the code view of the Inspector, do a left click on it. Select copy --> full xpath. Now copy paste the xpath to the parent variable, and put a double slash in front of the last element in the route.
example1 parent
For the List of the Gesellschaft für internationale Zusammenarbeit the last child before the parent is
//html//body//div//div//table[contains(@class, 'csx-new-table')]//tbody//tr
a tr which kind of stands for row in html. This, because GIZ is cooperative and ordered. You need the //tr because that says "search for any tr" which means we are reffering to a list of elements.
vars children
for the children it is about to define in xpath syntax, where the text lies that we want to be parsed automatically. child-name, child-link, child-period, child-info and child-sponsor are defined until now. In future it will be possible to define any variables anywhere and get that fed into the json output.
var javascript link
In case the whole website to spider is javascript generated gibberish, there is another possibility for you. To find out if the website is generated gibberish not containing your payload, just search in the outputed pages for the child name etc. If you do not find them, or directly see on the html pages no real xml, try to download your sites with jsdomain, and go up again to the paragraphs before. For the actual link child, the spider can javascript style clicking download the htmls behind the links already while downloading the entry list htmls with javascript.
For that to happen, you can define the javascript link that needs to be clicked in xpath syntax. If it becomes a pop up, which source code needs to be processed, or if it becomes an actual page the source code will be parsed, or if the clickable link refers to a pdf, the spider will handle all situations and output the resulting text in json under spiders/output.
var slow downlading
slow downloading comes into take when a website uses lazy loading and / or is used by too many users resulting in loading very slow. In this case, the selenium part, as any normal user, runs into problems of timing. Which leads to much longer processing time, when harvesting the lists. Depending on the configuration of the firewall of the server you are harvesting, there may even be a limit by time. And again, we just can act as a lot of different instances and everyone just gets a part of it to download. In this sense, even for future totalitarist system that may come, the freeing of information from big platforms is always and will always be possible in peer to peer systems.