I have read the Heron Documents about Implementing a Custom Scheduler. And I have known that I should implement some interfaces to implement a Custom Scheduler, such as ILauncher, IPacking, IScheduler and IUploader.
I have realized my CustomScheduler that implemented IScheduler interface, and I want to using LocalLauncher, LocalUploader and default Packing algorithm with my custom scheduler.
What's more, I modified the heron configuration file named scheduler.yaml that located at conf/local/ to use the custom scheduler. At the same time, I added the CustomScheduler.jar to heron-core/lib/scheduler/. However, there is something wrong as logs shows:
[2018-04-15 20:44:27 -0700] [STDERR] stderr: Exception in thread "main"
[2018-04-15 20:44:27 -0700] [STDERR] stderr: com.twitter.heron.spi.scheduler.SchedulerException: Failed to instantiate scheduler using class 'com.zyt.heron.custom.scheduler.CustomScheduler'
[2018-04-15 20:44:27 -0700] [STDERR] stderr: at com.twitter.heron.scheduler.utils.LauncherUtils.getSchedulerInstance(LauncherUtils.java:120)
[2018-04-15 20:44:27 -0700] [STDERR] stderr: at com.twitter.heron.scheduler.SchedulerMain.runScheduler(SchedulerMain.java:382)
[2018-04-15 20:44:27 -0700] [STDERR] stderr: at com.twitter.heron.scheduler.SchedulerMain.main(SchedulerMain.java:218)
what should I do to fix this problem? Thanks for your help!
The key is to provide your compiled .jar files to heron by copying your jar files into the .heron/lib/ folder.
I want to implement a custom scheduler and packing algorithm without recompiling Heron.
I have a project called "packing" that has the maven dependencies heron.api and heron.spi as requested.
My jar is called "packing-0.0.1-SNAPSHOT-jar-with-dendencies.jar built with maven. What works for me are the following steps summarized in a small shell-script. I run it from within my packing-project folder to the assembly-command works:
#!/bin/bash
mvn assembly:assembly
cp target/packing-0.0.1-SNAPSHOT-jar-with-dependencies.jar $HOME/.heron/lib/packing/
cp target/packing-0.0.1-SNAPSHOT-jar-with-dependencies.jar $HOME/.heron/lib/scheduler/
echo "updated packing plan and scheduler"
Then, simply add your custom packing or scheduler to your config:
Example from the packing.yaml:
heron.class.packing.algorithm: com.hcep.packing.CustomPacking
with com.hcep.packing being my package structure in my packing-project and CustomPacking my packing class.