I want to secure the communication of a TCP-based program using a shared passphrase/key. The easiest way to do that without having to deal with block size, padding, ... is to directly use a stream cipher. Doing that way, the amount of data is not changed between clear and encrypted data and the modification is trivial.
Using only a stream cipher means that there is no authentication and I have always considered/heard that encryption without authentication is not secure enough and should not be used.
If adding authentication to a stream cipher is mandatory, we lose the simplicity that stream cipher has added because we must add an HMAC or use an authenticated encryption method (like crypto_secretbox from NaCl), there is a minimum message length, we must handle padding, ...
What would you recommend? Is it safe to only use stream cipher without authentication in some particular cases?
Using some kind of message authenticator is particularly important with stream ciphers, because the relationship between changes to the ciphertext and changes to the plaintext is so simple.
You can't just blindly go and apply the stream cipher without adding any extra information to the stream, anyway - remember the most important rule of stream ciphers:
So unless you are only ever going to encrypt a single connection, and throw the passphrase away afterwards, you will need to generate a session key for each connection from the shared secret. This implies that you will need to send some extra information at the start of the connection, and since you're sending that anyway, sending a HMAC after each message should be no big deal.
Using a stream cipher because it seems simpler is usually a mistake, anyway. You mentioned crypto_secretbox from NaCl - I recommend using that, it will take care of the authentication and padding issues for you.