window - How many RDD in the resulting DStream of reduceByKeyAndWindow -


i working on small spark job compute stock correlation matrix dstream.

from dstream[(time, quote)], need aggregate quotes (double) time (long) among multiple rdds, before computing correlations (considering quotes of rdds)

dstream.reducebykeyandwindow{./*aggregate quotes in vectors*/..}         .foreachrdd {rdd => statistics.corr(rdd[vector])} 

to mind, solution if resulting dstream (from reducebykeyandwindow) contains 1 rdd aggregated quotes.

but not sure. how data distributed after reducebykeyandwindow? there way merge rdds in dstream?


Comments

Popular posts from this blog

twig - Using Twigbridge in a Laravel 5.1 Package -

jdbc - Not able to establish database connection in eclipse -

Kivy: Swiping (Carousel & ScreenManager) -