I installed Redmine version 3.4.2.stable onto Ruby 2.3.3, and hit rake test.
Only 3 tests fail, which is pretty good for the first run of a batch of thousands of tests on a new installation.
Then I add a bunch of plugins. Let's just say they are:
google_analytics_plugin 1.0.0
progressive_projects_list 1.0.0
redmine_agile 1.4.5
redmine_agreement 0.1.1
redmine_base_deface 0.0.1
redmine_cms 1.0.1
redmine_contacts 4.1.1
Now a lot more tests fail. 215 failures and 1530 errors out of 4225 test cases, to be exact. I'm going to assume that rake test calls rake redmine:plugins:test, but this is still an incredible number of failures.
As an example, one of the errors is:
MemberTest#test_validate_member_role:
ActiveRecord::StatementInvalid: Mysql2::Error: Incorrect string value: '\xD0\xAD\xD1\x82\xD0\xB8...' for column 'title' at row 1: INSERT INTO `wiki_pages` (`created_on`, `title`, `id`, `wiki_id`, `protected`, `parent_id`) VALUES ('2007-03-07 15:18:07', 'Этика_менеджмента', 10, 1, 0, NULL)
It seems someone has switched the language to Russian and not switched it back.
Now my actual question: Does everyone in Redmine-land accept this level of test fragility? I certainly can't be expected to fix every test, submit patches to their maintainers, and then level up my client to use the latest versions of these plugins.
How am I supposed to do TDD with Redmine?
The Redmine project itself maintains its core tests which assumes that its functional integrity is in place, i.e. that there are no plugins present which change functionality. As such, a plain Redmine without any plugins is expected to pass all core tests.
Once you introduce plugins however, things change. Since many plugins hook into or change a lot of internal Redmine functionality (often via monkey patches) in order to provide their functionality, installing them results in various tests of the core Redmine breaking as their assumptions no longer hold true. Since Remdine has only few internal APIs for plugins to hook into, monkey-patches are often the only way for plugins to hook into the core.
Unfortunately though, it is hard (if not impossible) to monkey-patch the core-tests along with the functionality of the plugins to adapt them to the new/changed behavior.
This leads to the current unfortunate situation that almost all Redmine plugins are barely (if at all) tested. Sometimes, you'll have unit tests of the plugin's functionality, but fully-functional integration tests are very rare. When using plugins, you thus can't generally rely on working tests but have to revert to code reviews and manual click-through tests.
I found that when reviewing plugins, the ones that work best have only few monkey patches, try to restrict themselves to stable interfaces or provides hooks where possible, and avoid overriding core functionality of Redmine.