How do i setup Pyspark in Python 3 with spark-env.sh.template -


because have issue in ipython3 notebook, guess have change "spark-env.sh.template" somehow.

exception: python in worker has different version 2.7 in driver 3.4, pyspark cannot run different minor versions

spark not yet work python 3.if wish use python api need python interpreter (version 2.6 or newer).

i had same issue when running ipython=1 ./pyspark.

ok quick fix

edit vim pyspark , change pyspark_driver_python="ipython" line 

pyspark_driver_python="ipython2"

that's it.

if want check dose ipython points to,

type which ipython in terminal , bet that'll be

/library/frameworks/python.framework/versions/3.4/bin/ipython

**updated**

the latest version of spark works python 3. may not need latest version.

just set environment variable:

export pyspark_python=python3

in case want change permanent add line pyspark script


Comments

Post a Comment

Popular posts from this blog

timeout - Handshake_timeout on RabbitMQ using python and pika from remote vm -

i have several vm's around world, identical , same python code (and same library version of pika 0.9.13). 1 of them failing because of handshake_timeout: =error report==== 17-jun-2015::12:27:07 === closing amqp connection <0.290.0> (yy.yy.yy.yy:47232 -> xx.xx.xx.xx:5672): {handshake_timeout,handshake} that taken rabbitmq log, in client code see like error:pika.adapters.base_connection:connection xx.xx.xx.xx:5672 failed: timeout unhandled exception in thread started <function listen_for_start_download_message @ 0x1e5bcf8> i have tested connection telnet following guide: http://rubybunny.info/articles/troubleshooting.html i increased handshake_timeout 40000 milliseconds , ssl_handshake_timeout 20000 milliseconds , error persists, ping rabbitmq server machine little bit higher in other machines nothing abnormal (136ms). has found similar problem sometime or can recommend way test rabbitmq connection? i'm not sure if solution problem in case pr...
Read more

gcc - MinGW's ld cannot perform PE operations on non PE output file -

i know there other similar questions out there, stackoverflow or not. i've researched lot this, , still didn't find single solution. i'm doing operative system side project. i've been doing in assembly, wanna join c code. test, made assembly code file (called test.asm): [bits 32] global _a section .text _a: jmp $ then made c file (called main.c): extern void a(void); int main(void) { a(); } to link, used file (called make.bat): "c:\mingw\bin\gcc.exe" -ffreestanding -c -o c.o main.c nasm -f coff -o asm.o test.asm "c:\mingw\bin\ld.exe" -ttext 0x100000 --oformat binary -o out.bin c.o asm.o pause i've been researching ages, , i'm still struggling find answer. hope won't flagged duplicate. acknowledge existence of similar questions, have different answers, , none work me. question: what doing wrong? old mingw versions had problem "ld" not able create non-pe files @ all. maybe current versions...
Read more

c# - Search and Add Comment with OpenXML for Word -

i'm newbie openxml sdk. i need perform following tasks on word docx on server side, , believe openxml sdk first place should into. so requirements follows: look specific text in document add comment in location of found text. is possible openxml sdk? tried go through docs, , found way search , replace text, couldn't find way add comment in appropriate location. thanks! the link ron gave indicate how insert comment. insert want, can find paragraph like paragraph p = body.descendants<paragraph>().where(p.innertext.contains(searchstring)).firstordefault(); if (p != null) { // here follow link ron post create comment // insert comment p p.insertafter() said in link } maybe have adapt bit depending matching criteria
Read more