javadockerredisdocker-composestreamsets

StreamSets processor cant find the Redis libs


I have a two containers with Redis and StreamSets. I want to write custom processor in Java and put to pipeline. But when I add code from tutorial to processor, send jar to lib and try to start I got an exceptions. Could you help me please?

Processor tutorials - https://github.com/streamsets/tutorials/blob/master/tutorial-processor/readme.md

Redis tutorial - https://dzone.com/articles/quickstart-how-to-use-redis-on-java

  redis:
    image: redis
    ports:
      - 6379:6379
    command: redis-server


  streamsets:
    build:
      context: "."
      args:
        SDC_LIBS: streamsets-datacollector-apache-kafka_2_0-lib                ports:
      - 18630:18630
    volumes:
      - ./user-libs:/opt/streamsets-datacollector-3.9.0/user-libs

Initiation in sampleprocessor:

  @Override
  protected List<ConfigIssue> init() {
      ....
      Config config = new Config();
      config.useSingleServer().setAddress("redis://redis:6379");
      redisson = Redisson.create(config);
      ... }



  <dependency>
      <groupId>org.redisson</groupId>
      <artifactId>redisson</artifactId>
      <version>3.10.4</version>
      <scope>provided</scope>
  </dependency>

The tests of code of processor successfully works.

Logs

streamsets_1  | 2020-03-21 14:00:19,954 [user:] [pipeline:] [runner:] [thread:main] [stage:] ERROR Main - Abnormal exit: java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;

streamsets_1  | java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;
streamsets_1  |         at java.lang.Class.getDeclaredFields0(Native Method)
streamsets_1  |         at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2614)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2629)
streamsets_1  |         at java.lang.Class.getFields(Class.java:1557)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:82)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:65)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.validate(StageDefinitionExtractor.java:141)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.extract(StageDefinitionExtractor.java:197)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.loadStages(ClassLoaderStageLibraryTask.java:557)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.initTask(ClassLoaderStageLibraryTask.java:278)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.CompositeTask.initTask(CompositeTask.java:44)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.TaskWrapper.init(TaskWrapper.java:40)
streamsets_1  |         at com.streamsets.datacollector.main.Main.lambda$doMain$1(Main.java:123)
streamsets_1  |         at java.security.AccessController.doPrivileged(Native Method)
streamsets_1  |         at com.streamsets.datacollector.security.SecurityUtil.doAs(SecurityUtil.java:92)
streamsets_1  |         at com.streamsets.datacollector.main.Main.doMain(Main.java:158)
streamsets_1  |         at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
streamsets_1  |         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
streamsets_1  |         at java.lang.reflect.Method.invoke(Method.java:498)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
streamsets_1  | Caused by: java.lang.ClassNotFoundException: org.redisson.api.RedissonClient
streamsets_1  |         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
streamsets_1  |         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         ... 26 more
streamsets_1  |
streamsets_1  | Abnormal exit: java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;Check STDERR for more details
streamsets_1  |
streamsets_1  | java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;
streamsets_1  |         at java.lang.Class.getDeclaredFields0(Native Method)
streamsets_1  |         at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2614)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2629)
streamsets_1  |         at java.lang.Class.getFields(Class.java:1557)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:82)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:65)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.validate(StageDefinitionExtractor.java:141)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.extract(StageDefinitionExtractor.java:197)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.loadStages(ClassLoaderStageLibraryTask.java:557)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.initTask(ClassLoaderStageLibraryTask.java:278)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.CompositeTask.initTask(CompositeTask.java:44)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.TaskWrapper.init(TaskWrapper.java:40)
streamsets_1  |         at com.streamsets.datacollector.main.Main.lambda$doMain$1(Main.java:123)
streamsets_1  |         at java.security.AccessController.doPrivileged(Native Method)
streamsets_1  |         at com.streamsets.datacollector.security.SecurityUtil.doAs(SecurityUtil.java:92)
streamsets_1  |         at com.streamsets.datacollector.main.Main.doMain(Main.java:158)
streamsets_1  |         at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
streamsets_1  |         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
streamsets_1  |         at java.lang.reflect.Method.invoke(Method.java:498)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
streamsets_1  | Caused by: java.lang.ClassNotFoundException: org.redisson.api.RedissonClient
streamsets_1  |         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
streamsets_1  |         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         ... 26 more

Solution

  • The problem is likely your dependency scope. You have specified provided, which is defined as:

    This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.

    You should specify compile scope.