I'm (re)designing large-scale application, we use multi-layer architecture based on DDD.
We have MVC with data layer (implementation of repositories), domain layer (definition of domain model and interfaces - repositories, services, unit of work), service layer (implementation of services). So far, we use domain models (mostly entities) across all layers, and we use DTOs only as view models (in controller, service returns domain model(s) and controller creates view model, which is passed to the view).
I'v read countless articles about using, not using, mapping and passing DTOs. I understand that there's no any definitive answer, but I'm not sure if it's ok or not returning domain models from services to controllers. If I return domain model, it's still never passed to the view, since controller always creates view-specific view model - in this case, it seem legit. On the other hand, it doesn't feel right when domain model leaves business layer (service layer). Sometimes service needs to return data object that wasn't defined in the domain and then we either have to add new object to the domain that isn't mapped, or create POCO object (this is ugly, since some services return domain models, some effectively return DTOs).
The question is - if we strictly use view models, is it ok to return domain models all the way to controllers, or should we always use DTOs for communication with service layer? If so, is it ok to adjust domain models based on what services need? (Frankly I don't think so, since services should consume what domain has.) If we should strictly stick to DTOs, should they be defined in service layer? (I think so.) Sometimes it's clear that we should use DTOs (e.g., when service performs lot of business logic and creates new objects), sometimes it's clear that we should use just domain models (e.g., when Membership service returns anemic User(s) - it seems it wouldn't make much sense to create DTO that is the same as domain model) - but I prefer consistency and good practices.
Article Domain vs DTO vs ViewModel - How and When to use them? (and also some other articles) is very similar to my problem, but it doesn't answer this question(s). Article Should I implement DTOs in repository pattern with EF? is also similar, but it doesn't deal with DDD.
Disclaimer: I don't intend to use any design pattern only because it exists and is fancy, on the other hand, I'd like to use good design patterns and practices also because it helps designing the application as a whole, helps with separation of concerns, even though using particular pattern isn't "necessary", at least at the moment.
it doesn't feel right when domain model leaves business layer (service layer)
Makes you feel like you are pulling the guts out, right? According to Martin Fowler: the Service Layer defines the application's boundary; it encapsulates the domain. In other words, it protects the domain.
Sometimes service needs to return data object that wasn't defined in the domain
Can you provide an example of this data object?
If we should strictly stick to DTOs, should they be defined in service layer?
Yes, because the response is part of your service layer. If it is defined "somewhere else" then the service layer needs to reference that "somewhere else", adding a new layer to your lasagna.
is it ok to return domain models all the way to controllers, or should we always use DTOs for communication with service layer?
A DTO is a response/request object, it makes sense if you use it for communication. If you use domain models in your presentation layer (MVC-Controllers/View, WebForms, ConsoleApp), then the presentation layer is tightly coupled to your domain, any changes in the domain requires you to change your controllers.
it seems it wouldn't make much sense to create DTO that is the same as domain model)
This is one of the disadvantages of DTO to new eyes. Right now, you are thinking duplication of code, but as your project expands, then it would make much more sense, especially in a team environment where different teams are assigned to different layers. See WET principle.
DTO might add additional complexity to your application, but so are your layers. DTO is an expensive feature of your system, they don't come free.
This article provides both advantage and disadvantage of using a DTO: http://guntherpopp.blogspot.com/2010/09/to-dto-or-not-to-dto.html
Summary as follows:
When to Use
When not to Use
Arguments Against DTO
Arguments For DTO