[SOLVED] SQLite queries much slower using JDBC than in Firefox SqliteManager plugin

I'm having a problem with a query that I tested in the SQLiteManager plugin in Firefox.

This particular query executes in about 60 ms in the Firefox plugin, but it takes a whopping 3.7 seconds to execute when I use the same query and the same database in Java using the latest Sqlite JDBC driver.

The Firefox plugin is often a little bit faster for my other queries that work well (50ms faster at the top, sometimes JDBC is faster), but that's probably the overhead of creating the connection and adding the results to a list, but this particular query's performance difference is just ridiculous.

Here is the query:

SELECT p1.Id, p1.FirstName || ' ' || p1.LastName AS PlayerName, sch1.LaneNum, l1.Name AS LeagueName, l1.Season, SUM(s1.Score) AS Series, e1.Date FROM Scores s1
JOIN SchedulePlayers sp1 ON  s1.SchedulePlayerId = sp1.Id
JOIN Schedules sch1 ON sp1.ScheduleId = sch1.Id
JOIN Players p1 ON sp1.PlayerId = p1.Id
JOIN TeamEncounters te1 ON sch1.TeamEncounterId = te1.Id
JOIN Encounters e1 ON te1.EncounterId = e1.Id
JOIN Leagues l1 ON e1.LeagueId = l1.Id

WHERE s1.GameNum < 4 AND l1.Name LIKE 'Juniors%' AND l1.Season = 2013 AND (sch1.LaneNum = 1 OR  sch1.LaneNum = 2) AND s1.IsBowlout = 0
GROUP BY p1.Id, l1.Id, e1.Id
ORDER BY Series DESC LIMIT 0,20

Obviously, the slow part is the "LIKE 'Juniors%'", but that doesn't explain why it is slow in Java and not in the plugin.

If I execute EXPLAIN QUERY PLAN, I see that the firefox plugin uses the following index for the Leagues table: Columns: "Season, Name, RealName" (RealName is not used in this query yet).

If I execute EXPLAIN QUERY PLAN in Java, the index that is used for the Leagues table is the INTEGER PRIMARY KEY index, which is where I think the problem is.

In java, I run the above query, and then using the same connection I run the same query two more times, but replacing the l1.Name LIKE 'Juniors% part by p1.Sex = 1 and p1.Sex = 2 the second time. Those last two queries are fast in both cases, which further proves that the problem comes from the l1.Name LIKE 'Juniors%'

I have primary keys on all tables and foreign keys on all columns that require it. I also have many other indexes because I am redesigning the old database from scratch because there were many repeated fields and I decided to add indexes to make it even faster, but in this particular case, I'm stuck, especially since it works in one case but not another. Is it possible that I indexed the tables too aggressively and it makes it more difficult for the Manager to pick the correct indexes?

Feel free to ask for more information about the tables, columns, queries, etc.

EDIT

The Firefox plugin uses SQLite 3.7.17 and the JDBC driver uses SQLite 3.8.0. I tried using the 3.7.20 JDBC driver (couldn't find a download link to the 3.7.17 driver) and I get the same performance issues, and some other queries get a worse performance with this one, so I switched back to 3.8.0.

I edited the performance times because I had made a mistake when benchmarking: the previous times were for running the queries multiple times. So in Firefox, it takes about 60 ms to execute the query once while in Java, it takes 3600 ms, so that's 60 times more, which is not acceptable for my application.

Here is the detailed EXPLAIN QUERY PLAN from the Java query execution, where the columns are, in order: SelectId, Order, From, Detail:

0 0 0 SEARCH TABLE Scores AS s1 USING INDEX idxScoresGameNumScore (GameNum<?)
0 1 1 SEARCH TABLE SchedulePlayers AS sp1 USING INTEGER PRIMARY KEY (rowid=?)
0 2 3 SEARCH TABLE Players AS p1 USING INTEGER PRIMARY KEY (rowid=?)
0 3 2 SEARCH TABLE Schedules AS sch1 USING INTEGER PRIMARY KEY (rowid=?)
0 0 0 EXECUTE LIST SUBQUERY 1
0 4 4 SEARCH TABLE TeamEncounters AS te1 USING INTEGER PRIMARY KEY (rowid=?)
0 5 5 SEARCH TABLE Encounters AS e1 USING INTEGER PRIMARY KEY (rowid=?)
0 6 6 SEARCH TABLE Leagues AS l1 USING INTEGER PRIMARY KEY (rowid=?)
0 0 0 USE TEMP B-TREE FOR GROUP BY
0 0 0 USE TEMP B-TREE FOR ORDER BY

As you can see, Leagues uses the integer primary key, so it completely ignores indices that have 'Name' in it.

The EXPLAIN QUERY PLAN for the Firefox plugin is:

0 0 6 SEARCH TABLE Leagues AS l1 USING COVERING INDEX idxLeaguesRealName (Season=?) (~19 rows)
0 1 5 SEARCH TABLE Encounters AS e1 USING INDEX idxEncounters (LeagueId=?) (~16 rows)
0 2 4 SEARCH TABLE TeamEncounters AS te1 USING AUTOMATIC COVERING INDEX (EncounterId=?) (~6 rows)
0 3 2 SEARCH TABLE Schedules AS sch1 USING INDEX sqlite_autoindex_Schedules_1 (TeamEncounterId=?) (~1 rows)
0 4 1 SEARCH TABLE SchedulePlayers AS sp1 USING COVERING INDEX idxSchedulePlayers (ScheduleId=?) (~6 rows)
0 5 3 SEARCH TABLE Players AS p1 USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
0 6 0 SEARCH TABLE Scores AS s1 USING INDEX sqlite_autoindex_Scores_1 (SchedulePlayerId=? AND GameNum<?) (~1 rows)
0 0 0 USE TEMP B-TREE FOR GROUP BY
0 0 0 USE TEMP B-TREE FOR ORDER BY

As you can see, the order of the tables isn't the same either, and actually, all tables use the PRIMARY KEY index in Java, while they use what seem to be 'better' indices in Java, which I think is weird.

*I tried using INDEXED BY idxLeaguesRealName * after JOIN Leagues l1, but the performance stays the same (probably because the SEARCH TABLE Leagues is at the bottom in Java instead of the first table).

idxLeaguesRealName is an index on Season, Name, RealName, which, according to 5.3 in this link posted by @CL. is a low quality index because season only takes about 4 different values for 230 different leagues. I have run the ANALYZE command before running the queries , though, so according to that link, it should fix the problem of using a low quality index.

Another thing I tried is creating a new index that also uses the primary key field (for instance: Id, Season, Name), but the Query Planner doesn't use it. I don't even know if it's a good idea to put the primary key as one of the fields in an user-created index. I'm just trying everything I can think of, because I'm at lost here since I don't understand the performance difference between the two ways of running the queries.

EXTRA INFORMATION ON OTHER QUERIES THAT ARE ALMOST THE SAME

As I mentioned earlier, I run other queries that are almost the same, except that l1.Name LIKE 'Juniors%' is replaced by either p1.Sex = 1 or p1.Sex = 2. Those queries execute in about 62ms each in Firefox and in 52 ms in Java, which means the query planner does a good job on this similar query.

In JDBC, the EXPLAIN QUERY PLAN gives this output:

0 0 4 SCAN TABLE TeamEncounters AS te1 USING COVERING INDEX idxTeamEncounters
0 1 5 SEARCH TABLE Encounters AS e1 USING INTEGER PRIMARY KEY (rowid=?)
0 2 2 SEARCH TABLE Schedules AS sch1 USING INDEX sqlite_autoindex_Schedules_1 (TeamEncounterId=?)
0 0 0 EXECUTE LIST SUBQUERY 1
0 3 6 SEARCH TABLE Leagues AS l1 USING INTEGER PRIMARY KEY (rowid=?)
0 4 1 SEARCH TABLE SchedulePlayers AS sp1 USING COVERING INDEX idxSchedulePlayers (ScheduleId=?)
0 5 3 SEARCH TABLE Players AS p1 USING INTEGER PRIMARY KEY (rowid=?)
0 6 0 SEARCH TABLE Scores AS s1 USING INDEX sqlite_autoindex_Scores_1 (SchedulePlayerId=? AND GameNum<?)
0 0 0 USE TEMP B-TREE FOR GROUP BY
0 0 0 USE TEMP B-TREE FOR ORDER BY

Which is vastly different from the original query's plan, since this one uses indices that seem to make more sense than only using PRIMARY KEYs indices like in the other case.

I just checked, and there are other queries in my application that execute slowly. All the slow queries are the ones that have the 'l1.Name LIKE 'Juniors%', everything else runs really fast.

I have read that queries that use LIKE run slowly, which would make me switch the way I designed some tables, like adding a field 'IsJuniorLeague' and comparing to that instead, which would probably fix the issue, but since I have seen that it's possible to make those queries fast enough, like in the Firefox plugin, I really want to understand what's happening behind the scenes since I usually test my queries in firefox first before trying them in my application since it's faster that way.