web-scraping, regex and iteration in python -


i have following url 'http://www.alriyadh.com/file/278?&page=1' write regex access urls page=2 till page=12

for example, url needed 'http://www.alriyadh.com/file/278?&page=4', not page = 14

i reckon work function iterate specified 10 pages access urls within them. have tried regex not work '.*?=[2-9]'

my aim content urls using newspaper package. want data research

thanks in advance

does not require regex, simple preset loop do.

import requests bs4 import beautifulsoup bs  url = 'http://www.alriyadh.com/file/278?&page='  page in range(2,13):     html = requests.get(url+str(page)).text     soup = bs(html)

Comments

Post a Comment

Popular posts from this blog

gcc - MinGW's ld cannot perform PE operations on non PE output file -

i know there other similar questions out there, stackoverflow or not. i've researched lot this, , still didn't find single solution. i'm doing operative system side project. i've been doing in assembly, wanna join c code. test, made assembly code file (called test.asm): [bits 32] global _a section .text _a: jmp $ then made c file (called main.c): extern void a(void); int main(void) { a(); } to link, used file (called make.bat): "c:\mingw\bin\gcc.exe" -ffreestanding -c -o c.o main.c nasm -f coff -o asm.o test.asm "c:\mingw\bin\ld.exe" -ttext 0x100000 --oformat binary -o out.bin c.o asm.o pause i've been researching ages, , i'm still struggling find answer. hope won't flagged duplicate. acknowledge existence of similar questions, have different answers, , none work me. question: what doing wrong? old mingw versions had problem "ld" not able create non-pe files @ all. maybe current versions...
Read more

How to connect android app to App engine -

i'm trying connect app in android server i've opened on appengine. my server's name : dddd-daniel-2345 , website server : http://dddd-daniel-2345.appspot.com/ this how connect android app server in app engine: app.yaml: application: dddd-daniel-2345 version: 1 runtime: python27 api_version: 1 threadsafe: yes handlers: - url: .* script: main.app libraries: - name: webapp2 version: "2.5.2" main.py: #!/usr/bin/env python # # copyright 2007 google inc. google.appengine.api import users import webapp2 class userhandler(webapp2.requesthandler): def get(self): user = users.get_current_user() if user: greeting = ('%s,%s,%s' % (user.email(),user.user_id(),user.nickname())) else: greeting = ('not logged in') self.response.out.write(greeting) app = webapp2.wsgiapplication([('/', userhandler),], debug=true) my applications code : add manifast main....
Read more

hadoop - Running Map Reduce Job shows error - Mkdirs failed to create /var/folders/ -

when running map reduce job in mac terminal as: pawandeepsingh1$ hadoop jar maximumtemperature.jar exception in thread "main" java.io.ioexception: mkdirs failed create /var/folders/v1/lyx_f0rj615cy8s54_bk053h0000gp/t/hadoop-unjar3698429834837790177/meta-inf/license @ org.apache.hadoop.util.runjar.ensuredirectory(runjar.java:128) @ org.apache.hadoop.util.runjar.unjar(runjar.java:104) @ org.apache.hadoop.util.runjar.unjar(runjar.java:81) @ org.apache.hadoop.util.runjar.run(runjar.java:209) @ org.apache.hadoop.util.runjar.main(runjar.java:136) i have seen similar question says don't have permissions run job. can give me step wise solution. thank in advance you don't have permission on hdfs filesystem create dirs job. submitting cluster or running local testing environment. can su user has perms /var folders on hdfs? if aren't admin on cluster need have admin either add group ( hdfs, hadoop) have permissions on hdfs or give p...
Read more