How to call SharePoint 2007 web services from Silverlight on FBA site

I have a Silverlight 5 app that runs on a SharePoint 2007 site. The Silverlight app gets a bunch of data from SharePoint lists using the Lists.asmx service. The main site is secured using NTLM security and Silverlight is able to successfully call the web services without having to eplicitly set anything to do with authentication. We extended the SharePoint site to a second domain that uses asp.net FormsBasedAuthentication. On this version of the site, the first web service call fails with a 403 Forbidden response and the exception:

System.ServiceModel.CommunicationException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound.

As I understand it, this is a generic message and not the true exception. Well, obviously we need the real exception to have any idea what the problem is. This MSDN page (Creating and Handling Faults in Silverlight) offers two solutions to getting the real exception.

The first is to modify your WCF service to return an alternate HTTP status code. This doesn't work because for one, it is not a WCF service, it's an .asmx web service so I can't add the suggested WCF behavior to modify the status code. Second, it's SharePoint's service so I can't do much to modify it anyway. Could I modify IIS to achieve an equivalent solution somehow?

The second solution is to register an alternative HTTP stack in the Silverlight application. I tried this out and found out the "real" exception was an authentication exception. So I went down dead ends trying to figure out out to get authenticated for about 24 hours only to finally find out that normally, the ASP.NET authentication cookie is passed with the service request, unless you are using the Client HTTP stack. So registering the client HTTP stack allowed me to see real exceptions, but it created its own exception which seems to only be fixed by not using the Client HTTP stack...

I believe have verified with Fiddler that the authentication cookie is being sent when using the default HTTP stack. I don't know if the Lists.asmx service is unable to use it and is giving an authentication error anyway, or if there is some other exception. How can I determine the cause of the "The remote server returned an error: NotFound." exception?

Well, this isn't a good answer to the quesion "How can I determine the cause..." but it's what worked for me. What I did was open up STSSOAP.dll, the assembly containing the Lists service implemention, in reflector. I copied the relevant code to implment my own GetListItems method in my own service. Luckily, the actual Lists.GetListItems method code was minimal and just called other SharePoint methods and even luckier, those methods & members are all declared public so I was able to do this. I then replaced Lists.asmx with my version on the server and attached the debugger to get some info. What I found was my authentication cookie was being used and HttpContext.Current.Session.User.Identity.IsAuthenticated was true. So it knew I was logged in. But I was still getting an authentication error which I could see farther down in SharePoint's code, was converted to a 403.

So I know asp.net considers me authenticated but SharePoint says I don't have permissions. But the service account the app pool is running under is a SP admin so why don't I have permissions? So next I inspected the SPContext.Current.Web.User; SharePoint's current user. It was null! In other words, as far as SharePoint is concerned, I'm still logged in as an anonymous user because I haven't explicitly logged in with a domain user, and an anonymous user doesn't have permissions on the list I'm accessing or most of the rest of the API. The answer by Sean McDonough to this quesion lays it out.

Basically I need to run the involved code with elevated privileges to get the code to run under the asp.net service account that I originally thought it was running under. If using the API, you can use the RunWithElevatedPrivileges delegate. But if you're calling the web services, you can't do that. The few options I could think of were:

1. Call the services on the base site that uses Windows Authentication. This would require me to embed the credentials in the client-side Silverlight application which is a security threat so I nixed this one.
2. Implement my own web service to get the data for me.
   • The web service could access the list using the SharePoint API and RunWithElevatedPrivileges.
   • The web service could call the Windows Authentication site's services passing the current credentials or other embedded credentials.

For implementation ease, all I've done so far is the second option under number 2 and it's working. However, it is clunky and I may change to option 1 which I expect will also work.