http - Download only the text from a webpage content in Python -


how can download only text/html/javascript of webpage in python?

i'm trying statistics text written authors of blogs. needing text, want increase program speed avoiding download of images, etc.

i'm able separate text html markup language. intention avoiding downloading aditional content in webpage (like images, .swf or like)

so far use:

user_agent = 'mozilla/5.0 (macintosh; u; intel mac os x 10_6_4; en-us) applewebkit/534.3 (khtml, gecko) chrome/6.0.472.63 safari/534.3'         headers = {'user-agent': user_agent} req = urllib2.request(url, none, headers) response = urllib2.urlopen(req, timeout=60) content_type = response.info().getheader('content-type') if 'text/html' in content_type:    return response.read()

but i'm not sure if i'm doing right thing (i.e. downloading text only)

python beautifulsoup 1 of best parsing webpages

import bs4 import urllib.request  webpage=str(urllib.request.urlopen(link).read()) soup = bs4.beautifulsoup(webpage)  print(soup.get_text())

Comments

Post a Comment

Popular posts from this blog

gcc - MinGW's ld cannot perform PE operations on non PE output file -

i know there other similar questions out there, stackoverflow or not. i've researched lot this, , still didn't find single solution. i'm doing operative system side project. i've been doing in assembly, wanna join c code. test, made assembly code file (called test.asm): [bits 32] global _a section .text _a: jmp $ then made c file (called main.c): extern void a(void); int main(void) { a(); } to link, used file (called make.bat): "c:\mingw\bin\gcc.exe" -ffreestanding -c -o c.o main.c nasm -f coff -o asm.o test.asm "c:\mingw\bin\ld.exe" -ttext 0x100000 --oformat binary -o out.bin c.o asm.o pause i've been researching ages, , i'm still struggling find answer. hope won't flagged duplicate. acknowledge existence of similar questions, have different answers, , none work me. question: what doing wrong? old mingw versions had problem "ld" not able create non-pe files @ all. maybe current versions...
Read more

timeout - Handshake_timeout on RabbitMQ using python and pika from remote vm -

i have several vm's around world, identical , same python code (and same library version of pika 0.9.13). 1 of them failing because of handshake_timeout: =error report==== 17-jun-2015::12:27:07 === closing amqp connection <0.290.0> (yy.yy.yy.yy:47232 -> xx.xx.xx.xx:5672): {handshake_timeout,handshake} that taken rabbitmq log, in client code see like error:pika.adapters.base_connection:connection xx.xx.xx.xx:5672 failed: timeout unhandled exception in thread started <function listen_for_start_download_message @ 0x1e5bcf8> i have tested connection telnet following guide: http://rubybunny.info/articles/troubleshooting.html i increased handshake_timeout 40000 milliseconds , ssl_handshake_timeout 20000 milliseconds , error persists, ping rabbitmq server machine little bit higher in other machines nothing abnormal (136ms). has found similar problem sometime or can recommend way test rabbitmq connection? i'm not sure if solution problem in case pr...
Read more

c# - Search and Add Comment with OpenXML for Word -

i'm newbie openxml sdk. i need perform following tasks on word docx on server side, , believe openxml sdk first place should into. so requirements follows: look specific text in document add comment in location of found text. is possible openxml sdk? tried go through docs, , found way search , replace text, couldn't find way add comment in appropriate location. thanks! the link ron gave indicate how insert comment. insert want, can find paragraph like paragraph p = body.descendants<paragraph>().where(p.innertext.contains(searchstring)).firstordefault(); if (p != null) { // here follow link ron post create comment // insert comment p p.insertafter() said in link } maybe have adapt bit depending matching criteria
Read more