Cannot read cassandra column family on a spark cluster using the spark-cassandra-connector in java -


here code read column family using spark cassandra connector

import static com.datastax.spark.connector.japi.cassandrajavautil.*;  import com.datastax.spark.connector.japi.sparkcontextjavafunctions; import org.apache.spark.sparkconf; import org.apache.spark.sparkcontext; import org.apache.spark.api.java.javardd;  public class main {     private static final string host = "spark://sparkmaster:7077"; //    private static final string host = "local[4]";      private static final string app_name = "cassandra spark wordcount";      public static void main (string... args) {         string[] jars = {                 "./build/libs/cassandrasparkmapreduce-1.0-snapshot.jar"         };          sparkconf conf = new sparkconf(true)                 .set("spark.cassandra.connection.host", "107.108.214.154")                 .set("spark.executor.userclasspathfirst", "true")                 .setjars(jars);          sparkcontext sc = new sparkcontext(host, app_name, conf);         sparkcontextjavafunctions context = javafunctions(sc);          javardd<string> rdd = context.cassandratable("wordcount", "input")                 .map(row -> row.tostring());          system.out.println(rdd.toarray());     } }

and here build.gradle file build , run application

group 'in.suyash.tests' version '1.0-snapshot'  apply plugin: 'java' apply plugin: 'application'  sourcecompatibility = 1.8  repositories {     mavencentral() }  dependencies {     compile group: 'org.apache.spark', name: 'spark-core_2.10', version: '1.4.0'      compile group: 'com.datastax.spark', name: 'spark-cassandra-connector_2.10', version: '1.4.0-m1'     compile group: 'com.datastax.spark', name: 'spark-cassandra-connector-java_2.10', version: '1.4.0-m1'      testcompile group: 'junit', name: 'junit', version: '4.11' }  sourcesets {     main {         java {             srcdir './'         }     } }  mainclassname = 'main'  // http://stackoverflow.com/a/14441628/3673043 jar {     dofirst {         {             configurations.compile.collect {                 it.isdirectory() ? : ziptree(it)             }         }     }     exclude 'meta-inf/*.rsa', 'meta-inf/*.sf','meta-inf/*.dsa' }

i execute job first doing gradle build build jar, , gradle run. job fails, , looking @ stderr in executors, getting following exception

java.lang.classcastexception: org.apache.spark.scheduler.resulttask cannot cast org.apache.spark.scheduler.task     @ org.apache.spark.executor.executor$taskrunner.run(executor.scala:194)     @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142)     @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617)     @ java.lang.thread.run(thread.java:745)

i have 3 node setup, in 1 node acts spark master node, while other 2 spark worker nodes , form cassandra ring. can execute job locally, if change spark host, on cluster getting weird exception, cannot find anywhere else. versions:

  • spark: 1.4.0
  • cassandra: 2.1.6
  • spark-cassandra-connector: 1.4.0-m1


edit

i can't answer how resolved this, removed java installations nodes, restarted , installed fresh copy of jdk1.8.0_45, started cluster again, , job completes successfully. explanations behaviour welcome.


Comments

Post a Comment

Popular posts from this blog

symfony - TEST environment only: The database schema is not in sync with the current mapping file -

there similar questions regarding issue haven't seen in particular test environment , sqlite. cannot validate schema test environment uses sqlite can validate schemas dev environment uses mysql. know reason , possible solution? whole application works fine in both dev , test environments though. phing , ci build broken. php app/console doctrine:schema:update --force --env=test updating database schema... database schema updated successfully! "9" queries executed php app/console doctrine:schema:validate --env=test [mapping] ok - mapping files correct. [database] fail - database schema not in sync current mapping file. update i'm adding sqldump result see if spots wrong it. thing can is, have 2 bundles. country below coming frontendbundle user entity missing defined in backendbundle . php app/console doctrine:schema:update --dump-sql --env=test drop index uniq_5373c9665e237e06; drop index uniq_5373c966d1f4eb9a; create temporary table __temp__country se...
Read more

twig - Using Twigbridge in a Laravel 5.1 Package -

i'm trying create laravel 5.1 package uses twigbridge instead of .php views. can normal .php views display fine, when try make use of .twig templates i'm met error. possible use twigbridge templates when creating views laravel package or pretty stuck blade , php ? edit 1: loading twig template without extending works. it's when start using {% extends 'file' %} error comes out. i error error loading /virtual/laravel/packages/username/mypackage/src/views/sample.twig: template "templates.master" not defined () in "/virtual/laravel/packages/username/mypackage/src/views/sample.twig" @ line 1. my folders username |_ mypackage |_ src |_ controllers - samplecontroller.php |_ views |_ templates - master.twig - sample.twig twig template sample.twig {% extends 'templates.master' %} {% block content %} {{ parent() ...
Read more

jdbc - Not able to establish database connection in eclipse -

i want make new database connection (db2) in eclipse purpose of using jpa. in jpa perspective, in data source explorer view, when try create new db connection, receive following logs when try ping server. org.eclipse.datatools.connectivity.exceptions.dbnotstartexception: no start database command issued. errorcode = -4499, sqlstate = 08001. @ org.eclipse.datatools.enablement.ibm.db2.internal.luw.jdbcluwjdbcconnection.getconnectexception(jdbcluwjdbcconnection.java:74) @ org.eclipse.datatools.connectivity.ui.pingjob.gettestconnectionexception(pingjob.java:81) @ org.eclipse.datatools.connectivity.ui.pingjob.run(pingjob.java:63) @ org.eclipse.core.internal.jobs.worker.run(worker.java:54) caused by: com.ibm.db2.jcc.am.ro: [jcc][t4][2043][11550][4.8.87] exception java.net.connectexception: error opening socket server example.example.com/51.37.93.117 on port 50,000 message: connection timed out: connect. errorcode=-4499, sqlstate=08001 @ com.ibm.db2.jcc.am.gd.a(gd.j...
Read more