How do I manage multi-projects dependency (Directed acyclic path) with IBM RAD ant?

I am working on an ant script to build java prjects developed with IBM RAD 7.5.

The an script is calling IBM RAD ant extenstion API. I am using Task to load the project set file(*.psf) into the memory, and calling Task to compile the projects listed in the projectSetImport.

The problem is the projects listed in psf file is not ordered by project dependency, when compiles, it fails because the depency is incorrect.

Is there any API or method to manage the dependency automatically? the psf files Iam handling is quite big, with 200+ projects in each file and it is constanly changing(e.g. some projects get removed and some new projects added in each week)

here is a detailed description for the question: The project dependency is like: 1) project A depends on B and D. 2) project B depends on C 3) project E depends on F

A -> B -> C
A -> D
E-> F

The sample.psf file just list all projects:

A
B
C
D
E
F

loads sample.psf, which have a project list [A,B,C,D,E,F] build project list from the build fail at A, because A need B and D to be build first.

My current solution is to rebuild the sample.psf manually, e.g. sample.psf file:

C
B
D
A
F
E

but this is hard to maintain, because there are 200+ projects in a psf file and they are constanly changing.

One way to attack this issue is to write a parser to read the .project file for each project, the dependency projects are listed in "projects" tag. Then implement a Directed acyclic path algorithm to reorder the dependency. This approach might be over kill. This must be a common issue in teams build IBM java projects, is there a solution?

Solution

Finally, I wrote some python code to compute the dependency. I Listed the logic below:

read the psf file into an list, the psf file is a xml file, and the project name is in tag.
for each project in the list, go to project source code and read the .project file and .classpath file, these two files contains the dependency project. for .project file(xml), fetch the project name from tag, for .classpath file. fetch the line with attribute kind='src'
now you got [source]->[dependened_project_list], implement a Directed acyclic map.(see attached code)
load the [source]->[dependened_project] in to the AdjecentListDigraph, and call topoSort() to return the dependency.

generate a new ordered psf file.

/////////////////////// dap_graph.py/////////////////////////////
# -*- coding: utf-8 -*-

'''Use directed acyclic path to calculate the dependency''' class Vertex: def init(self, name): self._name = name self.visited = True

class InValidDigraphError(RuntimeError): def init(self, arg): self.args = arg

class AdjecentListDigraph: '''represent a directed graph by adjacent list'''

def __init__(self):
    '''use a table to store edges,
    the key is the vertex name, value is vertex list 
    '''
    self._edge_table = {}
    self._vertex_name_set = set()

def __addVertex(self, vertex_name):
    self._vertex_name_set.add(vertex_name)

def addEdge(self, start_vertex, end_vertex):
    if not self._edge_table.has_key(start_vertex._name):
        self._edge_table[start_vertex._name] = []
    self._edge_table[start_vertex._name].append(end_vertex)
    # populate vertex set
    self.__addVertex(start_vertex._name)
    self.__addVertex(end_vertex._name)

def getNextLeaf(self, vertex_name_set, edge_table):
    '''pick up a vertex which has no end vertex. return vertex.name.
    algorithm:
    for v in vertex_set:
        get vertexes not in edge_table.keys()
        then get vertex whose end_vertex is empty  
    '''

print 'TODO: validate this is a connected tree'

    leaf_set = vertex_name_set - set(edge_table.keys())
    if len(leaf_set) == 0: 
        if len(edge_table) > 0:
            raise InValidDigraphError("Error: Cyclic directed graph")
    else:
        vertex_name = leaf_set.pop()
        vertex_name_set.remove(vertex_name)
        # remove any occurrence of vertext_name in edge_table
        for key, vertex_list in edge_table.items():
            if vertex_name in vertex_list:
                vertex_list.remove(vertex_name)
            # remove the vertex who has no end vertex from edge_table
            if len(vertex_list) == 0:
                del edge_table[key]
        return vertex_name

def topoSort(self):
    '''topological sort, return list of vertex. Throw error if it is 
    a cyclic graph'''
    sorted_vertex = []
    edge_table = self.dumpEdges()
    vertex_name_set = set(self.dumpVertexes())

    while len(vertex_name_set) > 0:
        next_vertex = self.getNextLeaf(vertex_name_set, edge_table)
        sorted_vertex.append(next_vertex)
    return sorted_vertex

def dumpEdges(self):
    '''return the _edge_list for debugging'''
    edge_table = {}
    for key in self._edge_table:
        if not edge_table.has_key(key): 
            edge_table[key] = []
        edge_table[key] = [v._name for v in self._edge_table[key]]
    return edge_table

def dumpVertexes(self):
    return self._vertex_name_set

//////////////////////projects_loader.py///////////////////////

-- coding: utf-8 --

''' This module will load dependencies from every projects from psf, and compute the directed acyclic path.

Dependencies are loaded into a map structured as below: dependency_map{"project_A":set(A1,A2,A3), "A1:set(B1,B2,B3)}

The algorithm is: 1) read 2) call readProjectDependency(project_name) ''' import os, xml.dom.minidom from utils.setting import configuration

class ProjectsLoader:

def __init__(self, application_name):
    self.dependency_map = {}
    self.source_dir = configuration.get('Build', 'base.dir')
    self.application_name = application_name
    self.src_filter_list = configuration.getCollection('psf',\
                                                        'src.filter.list')

def loadDependenciesFromProjects(self, project_list):
    for project_name in project_list:
        self.readProjectDependency(project_name)

def readProjectDependency(self, project_name):
    project_path = self.source_dir + '\\' + self.application_name + '\\'\
        + project_name
    project_file_path = os.path.join(project_path,'.project')
    projects_from_project_file = self.readProjectFile(project_file_path)

    classpath_file_path = os.path.join(project_path,'.classpath')
    projects_from_classpath_file = self.\
        readClasspathFile(classpath_file_path)

    projects = (projects_from_project_file | projects_from_classpath_file)
    if self.dependency_map.has_key(project_name):
        self.dependency_map[project_name] |= projects
    else:
        self.dependency_map[project_name] = projects

def loadDependencyByProjectName(self, project_name):
    project_path = self.source_dir + '\\' + self.application_name + '\\'\
        + project_name
    project_file_path = os.path.join(project_path,'.project')
    projects_from_project_file = self.readProjectFile(project_file_path)

    classpath_file_path = os.path.join(project_path,'.classpath')
    projects_from_classpath_file = self.\
        readClasspathFile(classpath_file_path)

    projects = list(set(projects_from_project_file\
                         + projects_from_classpath_file))
    self.dependency_map[project_name] = projects
    for project in projects:
        self.loadDependencyByProjectName(project)

def readProjectFile(self, project_file_path):
    DOMTree = xml.dom.minidom.parse(project_file_path)
    projects = DOMTree.documentElement.getElementsByTagName('project')
    return set([project.childNodes[0].data for project in projects])

def readClasspathFile(self, classpath_file_path):
    dependency_projects = set([])
    if os.path.isfile(classpath_file_path):
        DOMTree = xml.dom.minidom.parse(classpath_file_path)
        projects = DOMTree.documentElement.\
            getElementsByTagName('classpathentry')
        for project in projects:
            if project.hasAttribute('kind') and project.getAttribute\
                ('kind') == 'src' and project.hasAttribute('path') and \
                project.getAttribute('path') not in self.src_filter_list:
                    project_name = project.getAttribute('path').lstrip('/')
                    dependency_projects.add(project_name)

    return dependency_projects

def getDependencyMap(self):
    return self.dependency_map