alpcentaur
/
wd_importPEP
1
0
Fork 0
Code Issues 0 Pull Requests 0 Projects 0 Releases 0 Wiki Activity
automated Pipeline for parsing profiles of politically exposed persons (PEP) into Wikidata
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
10 Commits
1 Branch
329 KiB
Tree: ee1d5d8919
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'ee1d5d8919'
${ noResults }
wd_importPEP/crawlers/useMembersParliamentCrawler.py

18 lines
376 B
Raw Normal View History

added first two functions of class MembersOfParliamentCrawler
1 year ago
changed dir and import structure to be used in root of project
1 year ago
added first two functions of class MembersOfParliamentCrawler
1 year ago
​
​
changed name countries.yaml to config.yaml
1 year ago
added first two functions of class MembersOfParliamentCrawler
1 year ago
​
added function parseMemberData2dictionary()
1 year ago
added first two functions of class MembersOfParliamentCrawler
1 year ago
​
added function parseMemberData2dictionary()
1 year ago
​
added memberData function with first step loading json, renamed memberListData function
1 year ago
​
added function parseMemberData2dictionary() for first property political party
1 year ago
​
added function parseMemberData2dictionary()
1 year ago
 
  1. 

  2. from crawlers.MembersParliamentCrawler import *
  3. 

  4. 

  5. 

  6. config = 'config.yaml'
  7. listOfCountries = ['nicaragua']
  8. 

  9. 

  10. Crawler = membersParliamentCrawler(config)
  11. 

  12. #Crawler.downloadMemberListPagesOfCountries(listOfCountries)
  13. 

  14. #Crawler.parseMemberListData2dictionary(listOfCountries)
  15. 

  16. #Crawler.downloadMemberDataHtmls(listOfCountries)
  17. 

  18. Crawler.parseMemberData2dictionary(listOfCountries)