Shredding XML column in SQL Server into new tables
I have a source table, each row of which has an XML field that contains an unknown number of survey responses that I need to shred and put into output tables. I've tried nodes and OPENXML and I can't seem to get it working. From my research, I'm thinking I need to use CROSS APPLY, but I can't seem to put it all together in my head in a way that I understand and get it to work. I apologize if I'm missing something simple, but I really am stuck and would appreciate any help offered for how to deal with this.
Here's the source table containing imported data (the format of which I have no control over) that looks like this:
ImportId SurveyId DateImported ResponseData
-------- -------- ------------ ------------
1 11223344 2017-05-21 18:00:00.00 <survey><data><result>...
2 55667788 2017-05-21 18:01:00.00 <survey><data><result>...
3 99009988 2017-05-21 18:02:00.00 <survey><data><result>...
Here's a simplified example of the XML in the ResponseData column:
<survey>
<data>
<result>
<id>12345</id>
<date_submitted>2017-05-10 09:30:15</date_submitted>
<url_variables>
<respondent_id>
<key>respondent_id</key>
<value>987654</value>
<type>url</type>
</respondent_id>
<respondent_level>
<key>respondent_level</key>
<value>5</value>
<type>url</type>
</respondent_level>
</url_variables>
<survey_data>
<question>
<id>1</id>
<answer>Yes</answer>
</question>
<question>
<id>2</id>
<answer>No</answer>
</question>
<question>
<id>3</id>
<subquestions>
<subquestion>
<id>4</id>
<answer>Maybe</answer>
</subquestion>
<subquestion>
<id>5</id>
<answer>I don't know</answer>
</subquestion>
</subquestions>
</question>
... more questions ...
</survey_data>
</result>
<result>
<id>67890</id>
<date_submitted>2017-05-11 10:00:00</date_submitted>
<url_variables>
<respondent_id>
<key>respondent_id</key>
<value>34567</value>
<type>url</type>
</respondent_id>
<respondent_level>
<key>respondent_level</key>
<value>10</value>
<type>url</type>
</respondent_level>
</url_variables>
<survey_data>
<question>
<id>1</id>
<answer>No</answer>
</question>
<question>
<id>2</id>
<answer>Yes</answer>
</question>
<question>
<id>3</id>
<subquestions>
<subquestion>
<id>4</id>
<answer>Definitely not</answer>
</subquestion>
<subquestion>
<id>5</id>
<answer>I object</answer>
</subquestion>
</subquestions>
</question>
... more questions ...
</survey_data>
</result>
... more results ...
<data>
</survey>
I need to take this data, shredding the XML to get multiple results in each ResponseData field and put it in two tables like these:
ResultId SurveyId RespondentId RespondentLevel DateSubmitted
-------- -------- ------------ --------------- -------------
12345 11223344 987654 5 2017-05-10 09:30:15
67890 11223344 34567 10 2017-05-11 10:00:00
...
(Data extracted from the rest of ImportId 1 followed by ImportId 2, 3, etc)
ResultId QuestionId SubquestionId Answer
---------- ---------- ------------- ------
12345 1 0 Yes
12345 2 0 No
12345 3 4 Maybe
12345 3 5 I don't know
67890 1 0 No
67890 2 0 Yes
67890 3 4 Definitely not
67890 3 5 I object
...
(Data extracted from the rest of ImportId 1 followed by ImportId 2, 3, etc)
Solution
For Two Tables, you would need two queries
Example (I used only 1 record but will apply to multiple)
Declare @YourTable Table ([ImportId] int,[SurveyId] int,[DateImported] datetime,[ResponseData] xml)
Insert Into @YourTable Values
(1,11223344,'2017-05-21 18:00:00.00','<survey><data><result><id>12345</id><date_submitted>2017-05-10 09:30:15</date_submitted><url_variables><respondent_id><key>respondent_id</key><value>987654</value><type>url</type></respondent_id><respondent_level><key>respondent_level</key><value>5</value><type>url</type></respondent_level></url_variables><survey_data><question><id>1</id><answer>Yes</answer></question><question><id>2</id><answer>No</answer></question><question><id>3</id><subquestions><subquestion><id>4</id><answer>Maybe</answer></subquestion><subquestion><id>5</id><answer>I don''t know</answer></subquestion></subquestions></question></survey_data></result><result><id>67890</id><date_submitted>2017-05-11 10:00:00</date_submitted><url_variables><respondent_id><key>respondent_id</key><value>34567</value><type>url</type></respondent_id><respondent_level><key>respondent_level</key><value>10</value><type>url</type></respondent_level></url_variables><survey_data><question><id>1</id><answer>No</answer></question><question><id>2</id><answer>Yes</answer></question><question><id>3</id><subquestions><subquestion><id>4</id><answer>Definitely not</answer></subquestion><subquestion><id>5</id><answer>I object</answer></subquestion></subquestions></question></survey_data></result></data></survey>')
Select B.*
From @YourTable A
Cross Apply (
Select [ResultID] = r.n.value('(id)[1]','int')
,A.[SurveyId]
,[RespondentId] = r.n.value('(url_variables/respondent_id/value)[1]','int')
,[RespondentLevel] = r.n.value('(url_variables/respondent_level/value)[1]','int')
,[DateSubmitted] = r.n.value('(date_submitted)[1]','datetime')
From A.[ResponseData].nodes('survey/data/result') r(n)
) B
Select B.*
From @YourTable A
Cross Apply (
Select [ResultID] = r.n.value('(id)[1]','int')
,[QuestionId] = IsNull(q.n.value('(id)[1]','int'),0)
,[SubquestionId] = IsNull(s.n.value('(id)[1]','int'),0)
,[answer] = concat(q.n.value('(answer)[1]','varchar(50)')
,s.n.value('(answer)[1]','varchar(50)')
)
From A.[ResponseData].nodes('survey/data/result') r(n)
Cross Apply r.n.nodes('survey_data/question') q(n)
Outer Apply q.n.nodes('subquestions/subquestion') s(n)
) B
Returns
- Updating and join on multiple rows, which row's value is used?
- What changed about the Statement.execute() between sqljdbc4 and mssql-jdbc?
- Varchar datetime with EDT in it convert into datetime
- LINQ: Why does a union of a union throw error message "IQueryable<...> does not contain a definition for 'Union'"?
- SQL Logic doesn't work
- Aggregation: Category missing in SQL subquery
- Error copying data from SQL Server to Redshift via a linked server connection
- Backup SQL Schema Only?
- Row wise average in SQL Server
- a *FUTUREPROOF* Project reference, to connect to SQL Server?
- Error for Invalid length parameter passed to SUBSTRING function
- Why is Query Optimizer Underestimating and Causing Data Spill to tempdb?
- SQL Server Management Studio Schema View
- Drop User from SQL Server Database?
- Create failed for Database
- SQL Server Express localdb.msi offline installer
- Loading data into SQL Server using Pandas dataframe
- Data has always changed(SQL Server Management Studio)
- SQL Server : generate random numeric string 27 characters long
- SQL Server Restore Error: Directory lookup for the file "db.mdf" failed
- Use dynamic database name in a stored procedure
- Why Does SQL Update Top apparently reduce locking even when no records are updated?
- "Type character does not declare data type" error
- Stopping repeated XML Headers
- C# filling a dataset from the database
- Microsoft SQL join on longitudinal data
- Add SSIS to existing SQL Server instance
- Cannot start LocalDB
- Remove all spaces from a string in SQL Server
- How to check the existence of related records in group by query?