is there any ability to create custom email templates in dnn 8, I didn't found anything in the official documentation related about this topic. Thanks !
The emails that DNN sends out are templated via language resource (.resx) files, which you can edit through the Languages module under the Admin menu. From that interface, you can edit the site's language, and then edit the GlobalResources file in the Global Resources section of the editor. (In DNN 9, this interface is now under the Languages tab within Site Settings, and you click the Translate button for the site's language).
If you want to add arbitrary new emails that you send to groups, there is not a built-in mechanism for that. There's an old "core" module, Newsletters, which can be used for that, or you can search on the DNN Store for a more full-featured module. However, the typical recommendation is to export your user list to a service like Mail Chimp or Constant Contact, rather than sending emails directly through the DNN site.
Finally, if you're building a custom extension which needs to send email, you can use the Mail.SendEmail or Mail.SendMail methods (SendEmail is more simple/streamlined, SendMail has more options).