Managing multiple interdependent git repositories - unification vs symbolic links

Up until today, I was able to manage seven repositories using a single view of the Source Control tab of VS Code.

Basically, I have 6 embedded firmware projects that run on different platforms, each in their own repository.

I then have a single shared repository where I keep all of the header files/drivers that are shared by all of these projects.

Each project repository contains a symbolic link to the place where the shared repository is cloned on my local machine so that the application structure is like:

• Application
  • Symlink to shared repository

symlink in folder tree

Today when I opened up VS code, I see that this is no longer working.

I used to be able to force VS code to "add the repository" to the source control view using Git Graph: Add Git Repository... but that now throws errors:

The folder "c:/PARTICLE_LOCAL/jaza_embedded" is not within the opened Visual Studio Code workspace, and therefore could not be added to Git Graph.

I can get the symlinked repo to show up in the source code management tab if I manually add the local folder to the workspace though:

shared folder added to vs code workspace redundantly at workspace root

both repositories now showing in Git management tab

I tried:

• Checking if any extensions got updated in the past few days that could be causing this issue, and rolled back to previous versions of extensions
• Checked that VS code itself did not have any updates recently

I expected:

• To find that some recent change to VS Code or its extensions has caused this to stop working.

What resulted:

• Could not find any cause of this new problem

What I am considering and would like help with:

I am considering unifying all of these git repositories into a single one, since I need the shared portions of the repositories to be the same for each project in the repo whenever I am compiling. My concern is about how to do this, and what "gotchas" or downsides there are in unifying 7 repositories into a single one.

First, I've tested git operations with POSIX symlinks. They work if the appropriate option is available (git config core.symlinks true, git config --get core.symlinks to test it). The symlinks between files work immediately, but for symlinks between directories, git requires configuration of submodules. So far, so good.

It's good that you've come to the same conclusion as I did: merging all of these repositories into a single one is the only solution that makes sense. Anyway, let's overview the situation with the POSIX symlink. Basically, hardlinks, symlinks, and NTFS junctions (reparse points) are associations between the file system objects (files or directories, it does not matter) that make that object graph an arbitrary graph, not necessarily a tree (a tree is a graph without loops). From the revision control system standpoint, it makes no essential difference between the pure-tree set of file system objects and arbitrary graphs. Why?

This is because a git repository always works with a set of file system objects of the same file system, and this is quite a reasonable approach. Only then the repository clone can be placed at any point of a local file system without losing any dependencies. Yes, git can refer to foreign repositories, but this is done through URL references, because different repositories, strictly speaking, are technically independent. POSIX links between different repositories would not make any sense, because cloning two different repositories at arbitrary locations would break those links anyway. All the above is a schematic proof of this fact, but it can be proven in a much stricter way.

You also need to take into account your practical reason: you require the possibility to modify files on all the repositories of the solution.

So, let's talk about merging the repositories into one. The first important key (#1) is to create a brand-new git repository. Another key (#2) is that you need to plan the migration and the development in two or more stages, it is important to keep the possibility to retreat your steps and rethink the general design. Let's assume you have four already functioning repositories, A, B, C, and D. The tentative steps may look like this:

• Clone the repositories to a local file system A, B, C, and D side by side. Let's say, you have the common local directory U with A, B, C, and D inside it.
• Remove all .git directories.
• Create a brand-new git repository on the server.
• Clone the newly created empty (or almost empty) repository to the local file system, say, N (new).
• Move A, B, C, and D from U into N.
• Commit changes in N, and push the result to the server.
• Make a first tag for N (a new repository) and push it to the server with the tag.
• The first stage of the development is complete. In the general case, the new directory contains some code, which is not yet operational.
• Here is the next key (#3): you did not modify 4 existing repositories.
• Make the code operational by adding missing links between A, B, C, and D. Here is another key (#4): better use relative paths, not POSIX links, no matter how well they are supported by git.
  A short note on POSIX links: people usually used them in the programming system where working with relative links looked cumbersome. I saw them only in pretty old C/C++ projects, mostly on *NIX. There is nothing wrong with this approach, but it would be some unnecessary complication. No matter — you decide, the key #4 is just advice.
• After the final commit of the operational code, the second stage is complete. Make a tag at this point.
• Continue development on N until the software is stable. Change all parts of the new repository arbitrarily.
• At a certain stable point, decide that this third stage is complete, and tag it.
• Now you have options for the fourth stage. Based on your new understanding of the state of the new project, you can get to the next step of development in two ways: 1) You can continue development in the new repository (cloned to N) and forget A, B, C, D. 2) You can decide that the former A, B, C, D parts are stable enough, or some subset of this 4 repositories is stable enough, so you can refactor the code so to make them independent again; at this point, you can start submitting the changes into some or all of these independent repositories.

This is a quick sketch. Please comment if I missed something, then I would be able to modify or add something.