window - How many RDD in the resulting DStream of reduceByKeyAndWindow -


i working on small spark job compute stock correlation matrix dstream.

from dstream[(time, quote)], need aggregate quotes (double) time (long) among multiple rdds, before computing correlations (considering quotes of rdds) 

dstream.reducebykeyandwindow{./*aggregate quotes in vectors*/..}         .foreachrdd {rdd => statistics.corr(rdd[vector])}

to mind, solution if resulting dstream (from reducebykeyandwindow) contains 1 rdd aggregated quotes.

but not sure. how data distributed after reducebykeyandwindow? there way merge rdds in dstream?


Comments

Post a Comment

Popular posts from this blog

gcc - MinGW's ld cannot perform PE operations on non PE output file -

i know there other similar questions out there, stackoverflow or not. i've researched lot this, , still didn't find single solution. i'm doing operative system side project. i've been doing in assembly, wanna join c code. test, made assembly code file (called test.asm): [bits 32] global _a section .text _a: jmp $ then made c file (called main.c): extern void a(void); int main(void) { a(); } to link, used file (called make.bat): "c:\mingw\bin\gcc.exe" -ffreestanding -c -o c.o main.c nasm -f coff -o asm.o test.asm "c:\mingw\bin\ld.exe" -ttext 0x100000 --oformat binary -o out.bin c.o asm.o pause i've been researching ages, , i'm still struggling find answer. hope won't flagged duplicate. acknowledge existence of similar questions, have different answers, , none work me. question: what doing wrong? old mingw versions had problem "ld" not able create non-pe files @ all. maybe current versions...
Read more

How to connect android app to App engine -

i'm trying connect app in android server i've opened on appengine. my server's name : dddd-daniel-2345 , website server : http://dddd-daniel-2345.appspot.com/ this how connect android app server in app engine: app.yaml: application: dddd-daniel-2345 version: 1 runtime: python27 api_version: 1 threadsafe: yes handlers: - url: .* script: main.app libraries: - name: webapp2 version: "2.5.2" main.py: #!/usr/bin/env python # # copyright 2007 google inc. google.appengine.api import users import webapp2 class userhandler(webapp2.requesthandler): def get(self): user = users.get_current_user() if user: greeting = ('%s,%s,%s' % (user.email(),user.user_id(),user.nickname())) else: greeting = ('not logged in') self.response.out.write(greeting) app = webapp2.wsgiapplication([('/', userhandler),], debug=true) my applications code : add manifast main....
Read more

hadoop - Running Map Reduce Job shows error - Mkdirs failed to create /var/folders/ -

when running map reduce job in mac terminal as: pawandeepsingh1$ hadoop jar maximumtemperature.jar exception in thread "main" java.io.ioexception: mkdirs failed create /var/folders/v1/lyx_f0rj615cy8s54_bk053h0000gp/t/hadoop-unjar3698429834837790177/meta-inf/license @ org.apache.hadoop.util.runjar.ensuredirectory(runjar.java:128) @ org.apache.hadoop.util.runjar.unjar(runjar.java:104) @ org.apache.hadoop.util.runjar.unjar(runjar.java:81) @ org.apache.hadoop.util.runjar.run(runjar.java:209) @ org.apache.hadoop.util.runjar.main(runjar.java:136) i have seen similar question says don't have permissions run job. can give me step wise solution. thank in advance you don't have permission on hdfs filesystem create dirs job. submitting cluster or running local testing environment. can su user has perms /var folders on hdfs? if aren't admin on cluster need have admin either add group ( hdfs, hadoop) have permissions on hdfs or give p...
Read more