I'm currently making a shell in C++ and I've been able to output the absolute directory as the prompt with getcwd(), however when the user reaches the home directory I would like to output a tilde instead of the home directory.
So for example instead of
[bb@bbmachine bb]
I would like to output
[bb@bbmachine ~]
How would I do something like that?
Read first about globbing, notably glob(7). For the expansion of path starting with tilde ~ consider wordexp(3) and glob(3).
For showing a path starting with the home directory, you need to get that home directory. You could use getenv(3) like getenv("HOME") (there are rare pathological cases where that could fail) or getpwuid(3) (or even getpw(3)) on the result of getuid(2). The home directory (when getpwuid succeeds) is given by the pw_dir field. See also credentials(7) (and execve(2)...)
There are rare pathological cases where getenv("HOME") and getpwuid(getuid()) fail or give unconsistent results (about the user's home directory). I believe you should detect failure, but you might not care about inconsistency. See also environ(7). You could even consider the GNU specific secure_getenv(3).
BTW, you could cache the home directory, i.e. compute it once in your shell. It is unlikely to change often.
You could also decide to canonicalize a path using realpath(3).
You certainly have corner cases: do you want to display /home/john/../mary/ as ~/../mary, as /home/mary/, as ~mary, (or even leave it as /home/john/../mary) assuming your process is running under john uid? (and both john and mary exist as user names in passwd(5) with home directories /home/john/ and /home/mary/). And what about /home/john/../../proc/$$/maps (see proc(5))?
You need to decide how you'll handle these corner or pathological cases, and document the behavior of your shell.
(the functions mentioned have a C API, you might want to use extern "C" in your C++ code)