Let's say I have a website example.com which I bought via a common domain registry nomcheap.com.
I want all traffic to a specific subdomain app.example.com to go to name server ns1.appserver.com so I can serve a specific user app.
I also want all other traffic (www.example.com, hello.example.com, *.example.com, etc.) to go to a different name server from a different provider ns1.squaresites.com so I can serve a general website (think something like a commerce Wordpress site).
None of the name servers are provided by the original domain registry nomcheap.com.
Is this possible? If so, any suggestions on how?
To point a subdomain to a name servers you need to create an NS record for the subdomain:
app.example.com NS ns1.appserver.com
This will make all queries go to ns1.appserver.com
*.example.com NS ns1.squaresites.com
The second record should catch all subdomains that don't have their own records (of any kind).