I have a vb.net application which calls a large quantity of data (about 65,000 rows) from two tables in a SQL Server CE database.
The code is pretty straightforward:
cmd.CommandText = "SELECT [Table1Col1], [Table1Col2], ... [Table1Col8], " & _
"[Table2Col1], [Table2Col2] FROM [Table1] LEFT JOIN [Table2] " & _
"ON [Table1].[LinkCol] = [Table2].[LinkCol] WHERE [Table1Col8] = 0 " & _
"ORDER BY [Table1].[LinkCol]"
reader = cmd.ExecuteReader()
Do While reader.read
[read data, format it, etc]
Loop
The reader.read
statement by far takes the most time to execute. It takes about 2 hours to read and process the 65,000 rows, and I estimate that about 60-70% of that is from the lag in the reader.read
statement.
Is there a way to speed this up?
EDIT:
I did some time measurements, and the reader.read
statement takes 150 times longer (on average) than the block of code where I read, format, etc.
The issue appears to be because of the LEFT JOIN in the sql statement. I gained massive speed up by executing two readers, one for each table, and syncing them in the code. Something like the following:
cmd1.CommandText = "SELECT [LinkCol], [Table1Col1], [Table1Col2], ... [Table1Col8] " & _
"FROM [Table1] WHERE [Table1Col8] = 0 ORDER BY [Table1].[LinkCol]"
cmd2.CommandText = "SELECT [LinkCol], [Table2Col1], [Table2Col2] FROM [Table2] " & _
"ORDER BY [LinkCol]"
reader1 = cmd1.ExecuteReader()
reader2 = cmd2.ExecuteReader()
Do While reader1.read
tbl1ID = cint(reader1("LinkCol"))
do while tbl1ID <> tbl2ID
reader2.read()
tbl2ID = cint(reader2("LinkCol"))
loop
[read data, format it, etc]
Loop
This is a simplified version without all the checks to avoid errors and ensure the readers stay in-sync, but it may be helpful to someone who encounters a similar issue where using one reader on two joined tables results in performance issues.