As far as I understand, there is no way to dynamically change application icon in Android. There are a lot of posts in SO about it. Nevertheless, on my stock Nexus 4 with vanilla Android, I have Go launcher installed and it can set badges above the app icon. So can Nova Launcher: https://play.google.com/store/apps/details?id=com.teslacoilsw.launcher
I'm interested how can this be achieved in Android? Some hacks? Private Api?
As anyone who has installed those apps knows, those are replacement home screens. A home screen can do whatever the home screen wants:
If those home screens want to display badges over icons for well-known apps using their own code (e.g., Gmail), they can do so
If those home screens want to expose some API that app developers can use to enable badges for their apps, they can do so
For example, Nova Launcher does the former, as people who read the Play Store listing can tell:
Unread count badges for SMS, Missed calls, Gmail, Email, Google Voice and more. Counts show in the dock, desktop, drawer and folders.
Beyond that, the only way I know of for an app to try to affect its launcher icon is by changing the enabled LAUNCHER component, disabling the old one and enabling a different one with a different icon. This is going to be unreliable, though, as not all home screens will necessarily change the icon dynamically, perhaps not doing so until a device reboot.