What happens if I miss facebook's callback, because my app is down, or had an error or something?
Will facebook retry later?
You mean for an "post authorize" or "post remove" event? In either case: no, Facebook won't retry. Both of those calls are renowned for being less than 100% reliable, even if your app is up all the time. (There have even been past bugs that caused the "remove" callback to be called for current users. This hasn't happened for quite a while, but that's not to say it won't again.)
You can check if a user has authorised your app with the Users.isAppUser API call if you really need to be sure. Following on from the above, it can even be worth making that check in your "remove" callback just in case Facebook calls it by mistake.