Background

I've inherited an old, extremely complicated Gradle-based Java project with many sub-projects. Many of these have closed-source *.jar libraries that should not leak to other (sub-)projects in order to avoid namespace clashes when multiple projects use different versions of the "same" library.

I'm trying to update the project to use some new versions of some of these libraries and in general clean up the dependencies, since development has become extremely complicated. To make things worse, one project is nested as a Git submodule, which is supposed to be able to be built either independently or as a subproject of the main Gradle project I'm working with.

The Git submodule

Here is a simplified structure analogous to the Git submodule/Gradle project/Gradle sub-project:

robotcontroller
+--.git
+--gradle
+--libs
+--planning
|  +--src
|  +--build.gradle
+--plugins
|  +--nao
|  |  +--libs
|  |  |  +--robocommons.jar
|  |  +--src
|  |  +--build.gradle
|  +--pepper
|  |  +--libs
|  |  |  +--graphadapter.jar
|  |  +--src
|  |  +--build.gradle
+--build.gradle
+--gradlew
+--gradlew.bat
+--settings.gradle

plugins/nao/build.gradle:

dependencies {
    compile project(':planning')
    // This is inside the subproject's "libs/" dir
    compile name: 'robocommons'
    compile group: 'org.apache.httpcomponents', name: 'httpclient', version: '4.5.3'
}

[rootProject, this].each {
    it.repositories {
        flatDir {
            dirs "${rootDir}/plugins/nao/libs/"
        }
    }
}

plugins/pepper/build.gradle:

dependencies {
    compile project(':planning')
    // This is inside the subproject's "libs/" dir
    compile name: 'graphadapter'
    compile group: 'com.google.guava', name: 'guava', version: '19.0'
}

[rootProject, this].each {
    it.repositories {
        maven {
            // Some proprietary, closed-source repo
        }
        flatDir {
            dirs "${rootDir}/plugins/pepper/libs/"
        }
    }
}   

plugins/build.gradle:

dependencies {
    compile project(':plugins:nao')
    compile project(':plugins:pepper')
}

build.gradle:

// Stuff like plugins here
...

allprojects {
  repositories {
    flatDir {
      dirs "${rootDir}/libs"
    }
    jcenter()

    maven {
        // Some other proprietary repo here
    }
  }

  dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.12'
  }
}

dependencies {
    compile subprojects
}

// Stuff like javadoc and sourceJar tasks
...

settings.gradle:

include 'planning'
include 'plugins'
include 'plugins:nao'
include 'plugins:pepper'

The "root" Git repository

The Gradle project and all of its sub-projects above not only need to be able to be built on their own, but the project is itself a sub-project of another Gradle project, which is in a different Git repository:

complicatedrobotproject
+--.git
+--gradle
+--robotcontroller@SOMEREVISION // This is managed by the ".gitmodules" file
+--robocommons // A different, partially-compatible version of the source code used to make "robocommons.jar" is here
|  +--src
|  +--build.gradle
+--.gitmodules
+--build.gradle
+--gradlew
+--gradlew.bat
+--settings.gradle

build.gradle:

allprojects {
    repositories {
        flatDir {
            dirs "${rootDir}/robotcontroller/libs"
        }
    }
}

dependencies {
    // some external dependencies
    ...
    compile project(":robotcontroller")
    compile project(":robocommons")

    // TEST
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

settings.gradle:

include 'robocommons'
include 'robotcontroller:planning'
include 'robotcontroller:plugins'
include 'robotcontroller:plugins:nao'
include 'robotcontroller:plugins:pepper'

---

Problem

In the example above, my immediate goal is to replace the source code found at complicatedrobotproject/robocommons with robocommons.jar. However, the only way to build the project is with a modified version of the Git submodule robotcontroller that actually depends on that:

robotcontroller/plugins/nao/build.gradle from revision SOMEREVISION:

dependencies {
    compile name ':robocommons' // When building "complicatedrobotproject", this in fact links "robocommons" the directory, not "robocommons.jar"!
    compile project(':robotcontroller:planning')
    compile group: 'org.apache.httpcomponents', name: 'httpclient', version: '4.5.3'
}

By adding :robotcontroller to the dependencies, the project complicatedrobotproject can be built, but robotcontroller cannot be built as a standalone Gradle project. Taking that qualifier then makes the latter buildable at the expense of the former. The classes present in robocommons.jar that are also present in the directory robocommons are largely but not completely equivalent. Ultimately, I want to replace both with a proper external dependency to e.g. compile group: 'com.complicatedrobots', name: 'robocommons', version: '2.0.3', but that will require a lot of re-implementation, and I first need to sort out this dependency nightmare.

My task is not to make the project make sense; As much as this makes my head hurt, I just need both Git repositories to be buildable and also using a single version of robocommons across all Gradle projects, regardless of which Git repository is being built. I can't even think about the problem straight because every single possibility I think of blocks out some other thing that needs to work.

I can't say I understand what's happening in that build and I feel your pain. The first thing I would say is that composite builds may offer a pathway to a "normal", maintainable build, allowing you to keep robotcontroller as a separate project. If you go down this route, you will have to remove the `include ":robotcontroller" from the parent's settings.gradle file and replace it with the equivalent composite build configuration.

That said, I think you can get this working fairly easily, because most of the pieces appear to be in place. I would do this:

1. Move all of those JARs currently in plugins/*/libs into robotcontroller/libs, adding a version to their file names

2. Update all the dependencies on those JARs so that they include the appropriate version

3. Remove the /robocommons directory

You see, robotcontroller/libs appears to already be set up as a flat directory repository in both robotcontroller and the parent project. It seems to me that the only reason the problematic libraries are in project-specific directories is because they don't have version numbers in their names to distinguish them. By adding the version number, you can keep them side by side in the same directory.

Does that make sense? Hope it helps anyway!