I have a two containers with Redis and StreamSets. I want to write custom processor in Java and put to pipeline. But when I add code from tutorial to processor, send jar to lib and try to start I got an exceptions. Could you help me please?
Processor tutorials - https://github.com/streamsets/tutorials/blob/master/tutorial-processor/readme.md
Redis tutorial - https://dzone.com/articles/quickstart-how-to-use-redis-on-java
redis:
image: redis
ports:
- 6379:6379
command: redis-server
streamsets:
build:
context: "."
args:
SDC_LIBS: streamsets-datacollector-apache-kafka_2_0-lib ports:
- 18630:18630
volumes:
- ./user-libs:/opt/streamsets-datacollector-3.9.0/user-libs
Initiation in sampleprocessor:
@Override
protected List<ConfigIssue> init() {
....
Config config = new Config();
config.useSingleServer().setAddress("redis://redis:6379");
redisson = Redisson.create(config);
... }
<dependency>
<groupId>org.redisson</groupId>
<artifactId>redisson</artifactId>
<version>3.10.4</version>
<scope>provided</scope>
</dependency>
The tests of code of processor successfully works.
Logs
streamsets_1 | 2020-03-21 14:00:19,954 [user:] [pipeline:] [runner:] [thread:main] [stage:] ERROR Main - Abnormal exit: java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;
streamsets_1 | java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;
streamsets_1 | at java.lang.Class.getDeclaredFields0(Native Method)
streamsets_1 | at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
streamsets_1 | at java.lang.Class.privateGetPublicFields(Class.java:2614)
streamsets_1 | at java.lang.Class.privateGetPublicFields(Class.java:2629)
streamsets_1 | at java.lang.Class.getFields(Class.java:1557)
streamsets_1 | at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:82)
streamsets_1 | at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:65)
streamsets_1 | at com.streamsets.datacollector.definition.StageDefinitionExtractor.validate(StageDefinitionExtractor.java:141)
streamsets_1 | at com.streamsets.datacollector.definition.StageDefinitionExtractor.extract(StageDefinitionExtractor.java:197)
streamsets_1 | at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.loadStages(ClassLoaderStageLibraryTask.java:557)
streamsets_1 | at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.initTask(ClassLoaderStageLibraryTask.java:278)
streamsets_1 | at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1 | at com.streamsets.datacollector.task.CompositeTask.initTask(CompositeTask.java:44)
streamsets_1 | at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1 | at com.streamsets.datacollector.task.TaskWrapper.init(TaskWrapper.java:40)
streamsets_1 | at com.streamsets.datacollector.main.Main.lambda$doMain$1(Main.java:123)
streamsets_1 | at java.security.AccessController.doPrivileged(Native Method)
streamsets_1 | at com.streamsets.datacollector.security.SecurityUtil.doAs(SecurityUtil.java:92)
streamsets_1 | at com.streamsets.datacollector.main.Main.doMain(Main.java:158)
streamsets_1 | at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
streamsets_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
streamsets_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
streamsets_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
streamsets_1 | at java.lang.reflect.Method.invoke(Method.java:498)
streamsets_1 | at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
streamsets_1 | at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
streamsets_1 | Caused by: java.lang.ClassNotFoundException: org.redisson.api.RedissonClient
streamsets_1 | at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
streamsets_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
streamsets_1 | at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
streamsets_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
streamsets_1 | at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1 | at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1 | at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1 | at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1 | ... 26 more
streamsets_1 |
streamsets_1 | Abnormal exit: java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;Check STDERR for more details
streamsets_1 |
streamsets_1 | java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;
streamsets_1 | at java.lang.Class.getDeclaredFields0(Native Method)
streamsets_1 | at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
streamsets_1 | at java.lang.Class.privateGetPublicFields(Class.java:2614)
streamsets_1 | at java.lang.Class.privateGetPublicFields(Class.java:2629)
streamsets_1 | at java.lang.Class.getFields(Class.java:1557)
streamsets_1 | at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:82)
streamsets_1 | at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:65)
streamsets_1 | at com.streamsets.datacollector.definition.StageDefinitionExtractor.validate(StageDefinitionExtractor.java:141)
streamsets_1 | at com.streamsets.datacollector.definition.StageDefinitionExtractor.extract(StageDefinitionExtractor.java:197)
streamsets_1 | at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.loadStages(ClassLoaderStageLibraryTask.java:557)
streamsets_1 | at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.initTask(ClassLoaderStageLibraryTask.java:278)
streamsets_1 | at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1 | at com.streamsets.datacollector.task.CompositeTask.initTask(CompositeTask.java:44)
streamsets_1 | at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1 | at com.streamsets.datacollector.task.TaskWrapper.init(TaskWrapper.java:40)
streamsets_1 | at com.streamsets.datacollector.main.Main.lambda$doMain$1(Main.java:123)
streamsets_1 | at java.security.AccessController.doPrivileged(Native Method)
streamsets_1 | at com.streamsets.datacollector.security.SecurityUtil.doAs(SecurityUtil.java:92)
streamsets_1 | at com.streamsets.datacollector.main.Main.doMain(Main.java:158)
streamsets_1 | at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
streamsets_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
streamsets_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
streamsets_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
streamsets_1 | at java.lang.reflect.Method.invoke(Method.java:498)
streamsets_1 | at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
streamsets_1 | at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
streamsets_1 | Caused by: java.lang.ClassNotFoundException: org.redisson.api.RedissonClient
streamsets_1 | at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
streamsets_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
streamsets_1 | at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
streamsets_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
streamsets_1 | at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1 | at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1 | at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1 | at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1 | ... 26 more
The problem is likely your dependency scope. You have specified provided
, which is defined as:
This is much like
compile
, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scopeprovided
because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.
You should specify compile
scope.