sconsbuild-dependenciesincremental-build

How can I get all the source dependencies SCons computed for a given target?


I want to do this programmatically right after a given target is built, during SCons build run, not with --tree or any other command to scons. I have a target node. It might have had some explicit dependencies, used scanners, file extension-based scanners, and whatever else SCons calculated. So like:

all_source_nodes = tgt_node.get_all_sources(...)

I searched the docs and the APIs. Only saw get_stored_implicit on FS nodes. I get None on that and also for .prerequisites and .implicit Node members.

I also found that .sources Node member shows direct sources that were passed into the builder. That's not enough too, of course, because I need essentially all the nodes of the dependency sub-tree, which is a lot more.


Solution

  • Eureka! :) A combination of build info data and recursive scanner application got me everything scons --tree=prune reports. Full code below, along with some informational messages.

    __FULL_DEPS = {}    # all dependencies for the node and its children recursively
    __DIRECT_DEPS = {}  # only the dependencies obtained from SCons for the node itself (direct and scanned)
    
    # Find scons deps for this target - these are all in the subtree (same as __FULL_DEPS[tgt])
    scons_deps = find_deps(tgt, None, env)
    tgt_deps = set(map(str, scons_deps))    # Some nodes refer to the same paths
    
    def find_deps(node, children_func, env):
        """Recursively traverses children dependencies from build info and by applying 
    children_func (scanner) to collect the full set of all dependencies 
    starting with 'node' as the dependency tree root.
    :Parameters:
        - node          the current target we check (root node of the sub-tree)
        - children_func the function called to get the children of a node. May be None.
        - env           the current environment
    :Return:
        The set of all source nodes in the dependency sub-tree
        While the node objects are unique, the underlying files/dirs may repeat on rare occasions.
        Collects the global __DIRECT_DEPS and __FULL_DEPS caches
        """
        if node in __FULL_DEPS:
            print("_find_deps '{}' deps - from CACHE: {}."
                                .format(str(node), len(__FULL_DEPS[node])))
            return __FULL_DEPS[node]
        children = set(node.all_children())  # children may repeat
        
        # Apply children func, if available and merge in the results
        scanned_deps = set()
        if children_func:
            scanned_deps = set(children_func(node))
            children |= scanned_deps
        __DIRECT_DEPS[node] = children
    
        # Iterate all the unvisited children and recursively call find_deps with children_func 
        # specific to each child.
        # FYI: Scanner use code is based on SCons code in Node.render_include_tree()
        subtree_deps = set()
        for child in sorted(children):
            scanner = node.get_source_scanner(child)
            # Note: get_build_scanner_path fails on nodes without executor
            scanner_path = node.get_build_scanner_path(scanner) if (scanner and node.get_executor()) else None
            def children_func(node, env=env, scanner=scanner, path=None):
                return node.get_found_includes(env, scanner, path)
        
            subtree_deps |= find_deps(child, children_func, env)
            __FULL_DEPS[node] = children | subtree_deps
    
        print("_find_deps '{}' deps: {} children ({} scanned) + {} from sub-tree"
            .format(str(node), len(children), len(scanned_deps), len(subtree_deps)))
    
    return __FULL_DEPS[node]
    

    Getting 700+ dependencies in about a second on a high performance machine. This should be called in a post-action, i.e. after the node is built. Probably will still work fine when called just before the node is built.