How to Delegate Credentials through double hop to SQL Server?
What I am trying to do:
We have a Task Scheduler that kicks off an EXE, which in the course of its runtime, will connect to SQL Server.
So that would be:
- taskServer.myDomain triggers the Task Scheduler action
- taskServer.myDomain exe runs locally
- taskServer.myDomain initiates a connection to sqlServer.myDomain
The scheduled task is associated with a service account (svc_user) that is set to run with highest privilege, run whether the user is logged in or not, and store credentials for access to non-local resources.
The actual behavior
What we are seeing is the Task Scheduler is indeed running as svc_user. It triggers the EXE as expected, and the EXE is also running as svc_user. When the EXE initiates a connection to SQL Server, it errors on authentication.
Looking at the Event Viewer we can see the failure trying to initialize the connection to SQL
Exception Info: System.Data.SqlClient.SqlException
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data.SqlClient.SqlConnectionString, System.Data.SqlClient.SqlCredential, System.Object, System.String, System.Security.SecureString, Boolean, System.Data.SqlClient.SqlConnectionString, System.Data.SqlClient.SessionData, System.Data.ProviderBase.DbConnectionPool, System.String, Boolean, System.Data.SqlClient.SqlAuthenticationProviderManager)
And then looking at the SQL Server logs we can see the root of the issue
Logon,Unknown,Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'. Reason: Could not find a login matching the name provided.
The connection initialized by the EXE to SQL Server is trying to authenticate as ANONYMOUS LOGON.
What I have tried
Background
This issue popped up when our IT team started deploying a GPO lockdown in our environments. So in order to get to this point, we first had to add some GPO exceptions to allow the svc_user to:
- log on locally
- log on as batch job
Progress?
This is where we started being able to capture the ANONYMOUS LOGON error in SQL Server. From there we tried a handful of other GPO exceptions including
- Allow Credential Save
- Enable computer and user accounts to be trusted for delegation
The actual issue?
So it would appear that this is a double hop delegation issue. Which eventually led me here and then via the answer, here and here.
So I tried adding GPO policies to allow delegating fresh credentials using the WSMAN/* protocol + wildcard.
Two issues with this:
- the
Freshcredentials refer to prompted credentials while the EXE is running as a service during off-hours and inheriting the credentials from the TaskScheduler - the
WSMANprotocol appears to be used for remote PowerShell sessions (via the original question in the serverfault post) and not SQL Service connections.
So, I added the protocol MSSQLSvc/* to the enabled delegation and tried all permutations of Fresh, Saved and Default delegation. (This was all done in Local Computer Policy -> Computer Configuration -> Administrative Templates -> system -> Credentials Delegation)
Where it gets weird
We have another server, otherServer.myDomain, which we setup with the same TaskSchedule. It is setup with the same GPO memberships, but seems to be able to successfully connect to SQL Server. AFAIK, the servers are identical as far as setup and configuration.
The Present
I have done a bit more digging into anywhere I could think that might offer clues as to how I can feed the credentials through or where they might be falling through. Including watching the traffic between the taskServer and the sqlServer as well as otherServer and sqlServer.
I was able to see NTLM challenges coming from the sqlServer to the taskServer/otherServer.
- In the case of taskServer, the NTLM response only has a
workstationString=taskServer - On otherServer, the NTLM response has
workstationString=otherServer,domainString=myDomain, anduserString=svc_user.
Question
What is the disconnect between hop 1 (task scheduler to EXE) and hop 2 (EXE to SQL on sqlServer)? And why does this behavior not match between taskServer and otherServer?
So I finally have an update/solution for this post.
The crux of the issue was a missing SPN. The short answer:
- Add an SPN for sqlServer associated with the service account SQL services are running as (not the svc_user)
- example:
SetSPN -S MSSQLSvc/sqlServer.myDomain myDomain\svc_sql_user
- example:
- Add another SPN like above but w/ the sql service port
- example:
SetSPN -S MSSQLSvc/sqlServer.myDomain:1433 myDomain\svc_sql_user
- example:
- Set the SQL service user account to allow delegation like so
- Which of definitions of database schema is correct?
- How to get isolation level in EF for MSSQL?
- Generate SQL Create Scripts for existing tables with query
- Insert multiple rows WITHOUT repeating the "INSERT INTO ..." part of the statement?
- Index for two columns that will be used in an operation on where clause?
- How do you update a datetime field in a table every time another field in the same table gets updated?
- Pros & Cons of TRUNCATE vs DELETE FROM
- comparing and contrasting Sybase and SQL Server
- How does sql server choose values in an update statement where there are multiple options?
- How to disconnect all users in sql server except sa?
- Select Consecutive Numbers in SQL
- SQL Find consecutive numbers in groups
- Spring Quartz configuration for connecting MS SQL server
- The most elegant way to generate permutations in SQL server
- Check column has 'letter' or '-' then bring value from another column
- Reading child XML node from column
- Combine JSON objects from multiple rows into a single object
- Find all stored procedures that reference a specific column in some table
- Laravel 10 stopped working with SQL Server Database
- Using TransactionScope around a stored procedure with transaction in SQL Server 2014
- Entity storing it's status as a FK to a status table. Can I update the FK once the status changes?
- How can I convert a Sql Server 2008 DateTimeOffset to a DateTime
- I need to replace old path's images to new path with the same image
- Format date as yyyy-mm-dd hh:mm:ss.000
- Does a foreign key automatically create an index?
- How do I select only the latest entry in a table for each group so each 'latest' is a single row?
- "The certificate chain was issued by an authority that is not trusted" when connecting DB in VM Role from Azure website
- Kafka Connect SqlServer Configuration (SSL)
- SQL-Server: Is there a SQL script that I can use to determine the progress of a SQL Server backup or restore process?
- TSQL Email Validation (without regex)