rest - java service - spark communication -


i pretty new spark , looking guidance :)

i have java service acts backend application. adhoc queries ui , figured out of these queries taking lot of time. so, decided move spark these tasks done. stuck how establish communication apache spark java.

i saw other questions , seems spark job server ooyala solves problem. curious know if there other options solve problem.

here sample code provided work on spark in java, can write code after input taken in spark, since i've used here pre-built library of kmean clustering, can take yours.

import java.util.regex.pattern; import org.apache.spark.sparkconf; import org.apache.spark.api.java.javardd; import org.apache.spark.api.java.javasparkcontext; import org.apache.spark.api.java.function.function; import org.apache.spark.mllib.clustering.kmeans; import org.apache.spark.mllib.clustering.kmeansmodel; import org.apache.spark.mllib.linalg.vector; import org.apache.spark.mllib.linalg.vectors;  public final class spark_kmeans { private static class parsepoint implements function<string, vector> { private static final pattern space = pattern.compile(" ");  @override public vector call(string line) { string[] tok = space.split(line); double[] point = new double[tok.length]; (int = 0; < tok.length; ++i) { point[i] = double.parsedouble(tok[i]); } return vectors.dense(point); } }  public static void main(string[] args1) {  string[] args = { "/usr/spark_pack/spark_input", "3", "5" };  if (args.length < 3) { system.err .println("usage: javakmeans <input_file> <k> <max_iterations> [<runs>]"); system.exit(1); } string inputfile = args[0]; int k = integer.parseint(args[1]); int iterations = integer.parseint(args[2]); int runs = 1; if (args.length >= 4) { runs = integer.parseint(args[3]); }  string sparkhome = "/usr/spark_pack/spark-1.3.0-bin-hadoop2.4/"; string sparkmasterurl = "spark://master:7077";  string jarfile1 = "/usr/spark_pack/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.3.0-hadoop2.4.0.jar"; string jarfile2 = "/usr/spark_pack/spark_jar/spark_mlib.jar";  /* * javastreamingcontext ssc = new javastreamingcontext(sparkmasterurl, * "kshitij stream engine", new duration(1000), sparkhome); */ sparkconf conf = new sparkconf().setappname("log analyzer sql") .setmaster(sparkmasterurl).setsparkhome(sparkhome) .setjars(new string[] { jarfile1, jarfile2 }) .setappname("javakmeans");  javasparkcontext sc = new javasparkcontext(conf);  /* * sparkconf sparkconf = new sparkconf().setappname("javakmeans"); * javasparkcontext sc = new javasparkcontext(sparkconf); */  javardd<string> lines = sc.textfile(inputfile);  //****this portion algo part************************* javardd<vector> points = lines.map(new parsepoint()); kmeansmodel model = kmeans.train(points.rdd(), k, iterations, runs, kmeans.k_means_parallel()); system.out.println("cluster centers:"); (vector center : model.clustercenters()) { system.out.println(" " + center); } double cost = model.computecost(points.rdd()); system.out.println("cost: " + cost); sc.stop(); } }

Comments

Post a Comment

Popular posts from this blog

How to connect android app to App engine -

i'm trying connect app in android server i've opened on appengine. my server's name : dddd-daniel-2345 , website server : http://dddd-daniel-2345.appspot.com/ this how connect android app server in app engine: app.yaml: application: dddd-daniel-2345 version: 1 runtime: python27 api_version: 1 threadsafe: yes handlers: - url: .* script: main.app libraries: - name: webapp2 version: "2.5.2" main.py: #!/usr/bin/env python # # copyright 2007 google inc. google.appengine.api import users import webapp2 class userhandler(webapp2.requesthandler): def get(self): user = users.get_current_user() if user: greeting = ('%s,%s,%s' % (user.email(),user.user_id(),user.nickname())) else: greeting = ('not logged in') self.response.out.write(greeting) app = webapp2.wsgiapplication([('/', userhandler),], debug=true) my applications code : add manifast main....
Read more

gcc - MinGW's ld cannot perform PE operations on non PE output file -

i know there other similar questions out there, stackoverflow or not. i've researched lot this, , still didn't find single solution. i'm doing operative system side project. i've been doing in assembly, wanna join c code. test, made assembly code file (called test.asm): [bits 32] global _a section .text _a: jmp $ then made c file (called main.c): extern void a(void); int main(void) { a(); } to link, used file (called make.bat): "c:\mingw\bin\gcc.exe" -ffreestanding -c -o c.o main.c nasm -f coff -o asm.o test.asm "c:\mingw\bin\ld.exe" -ttext 0x100000 --oformat binary -o out.bin c.o asm.o pause i've been researching ages, , i'm still struggling find answer. hope won't flagged duplicate. acknowledge existence of similar questions, have different answers, , none work me. question: what doing wrong? old mingw versions had problem "ld" not able create non-pe files @ all. maybe current versions...
Read more

php - display validation error message next to the textbox in codeigniter -

Image
i'm newbie codeigniter.i want display error message next text box.my view file create.php contains following code. <?php echo validation_errors(); //validation echo form_open('news/create'); //create form ?> <div class="boxes"> <div class="common"> <div class="lables"> <label for="name">name</label> </div> <div class="text_boxes"> <input type="text" name="uname" /><div id="infomessage"><?php echo $message;?></br> </div> </div>` and below controller model: public function create() { $this->load->helper('form'); $this->load->library('form_validation'); $data['title'] = 'insert items'; $this->form_validation->set_rules('uname','required|min_length[6]|m...
Read more