hadoop - Java Spark spilling in-memory map OutOfMemoryError error -


i'm running following code (a contains 10000 rows):

indexedrowmatrix qtmp = new indexedrowmatrix(a.rows()); indexedrowmatrix qtmpt = qtmp.tocoordinatematrix().transpose().toindexedrowmatrix();

i following error:

15/06/19 10:08:52 info externalappendonlymap: thread 66 spilling in-memory map of 24.8 mb disk (1 time far) 15/06/19 10:08:54 info externalappendonlymap: thread 66 spilling in-memory map of 24.8 mb disk (2 times far) 15/06/19 10:08:57 info externalappendonlymap: thread 66 spilling in-memory map of 24.8 mb disk (3 times far) 15/06/19 10:09:02 error executor: exception in task 0.0 in stage 9.0 (tid 9) java.lang.outofmemoryerror: java heap space

i have tried tuning memory various parameters well:

sparkconf conf = new sparkconf(); conf.set("spark.serializer", "org.apache.spark.serializer.kryoserializer"); conf.setappname("test"); conf.setmaster("local[2]"); conf.set("spark.executor.memory", "6g"); conf.set("spark.driver.maxresultsize", "1g"); conf.set("spark.shuffle.spill", "false"); conf.set("spark.storage.memoryfraction", "0.3"); conf.set("spark.eventlog.enabled", "false"); conf.set("driver-memory", "4g")`; javasparkcontext sc = new javasparkcontext(conf);

is there reason i'm consistently getting spilling error?

you should run application overridden java options.

start program -xms=[size] -xmx -xx:maxpermsize=[size] -xx:maxnewsize=[size]

for example -

-xms512m -xmx1152m -xx:maxpermsize=256m -xx:maxnewsize=256m


Comments

Post a Comment

Popular posts from this blog

How to connect android app to App engine -

i'm trying connect app in android server i've opened on appengine. my server's name : dddd-daniel-2345 , website server : http://dddd-daniel-2345.appspot.com/ this how connect android app server in app engine: app.yaml: application: dddd-daniel-2345 version: 1 runtime: python27 api_version: 1 threadsafe: yes handlers: - url: .* script: main.app libraries: - name: webapp2 version: "2.5.2" main.py: #!/usr/bin/env python # # copyright 2007 google inc. google.appengine.api import users import webapp2 class userhandler(webapp2.requesthandler): def get(self): user = users.get_current_user() if user: greeting = ('%s,%s,%s' % (user.email(),user.user_id(),user.nickname())) else: greeting = ('not logged in') self.response.out.write(greeting) app = webapp2.wsgiapplication([('/', userhandler),], debug=true) my applications code : add manifast main....
Read more

gcc - MinGW's ld cannot perform PE operations on non PE output file -

i know there other similar questions out there, stackoverflow or not. i've researched lot this, , still didn't find single solution. i'm doing operative system side project. i've been doing in assembly, wanna join c code. test, made assembly code file (called test.asm): [bits 32] global _a section .text _a: jmp $ then made c file (called main.c): extern void a(void); int main(void) { a(); } to link, used file (called make.bat): "c:\mingw\bin\gcc.exe" -ffreestanding -c -o c.o main.c nasm -f coff -o asm.o test.asm "c:\mingw\bin\ld.exe" -ttext 0x100000 --oformat binary -o out.bin c.o asm.o pause i've been researching ages, , i'm still struggling find answer. hope won't flagged duplicate. acknowledge existence of similar questions, have different answers, , none work me. question: what doing wrong? old mingw versions had problem "ld" not able create non-pe files @ all. maybe current versions...
Read more

php - display validation error message next to the textbox in codeigniter -

Image
i'm newbie codeigniter.i want display error message next text box.my view file create.php contains following code. <?php echo validation_errors(); //validation echo form_open('news/create'); //create form ?> <div class="boxes"> <div class="common"> <div class="lables"> <label for="name">name</label> </div> <div class="text_boxes"> <input type="text" name="uname" /><div id="infomessage"><?php echo $message;?></br> </div> </div>` and below controller model: public function create() { $this->load->helper('form'); $this->load->library('form_validation'); $data['title'] = 'insert items'; $this->form_validation->set_rules('uname','required|min_length[6]|m...
Read more