[SOLVED] Character Set Issues when Upgrading from Symfony 2.0.* to Symfony 2.1.*?

Character Set Issues when Upgrading from Symfony 2.0.* to Symfony 2.1.*?

I have recently upgraded my staging test site to the latest version of Symfony and updated all the vendors using composer as instructed in the upgrade document that comes with the download.

Everything has all updated fine, but I have noticed now that some bits of HTML are not displaying in the Twig templates.

I did a comparison with the current live site and it appears to be a character set issue. As an example I had a drop down list that had the following value in:

Kitchen Ducting > Ducting Kits > Ducting Kit 4” / 100mm

In the updated site the drop-down list item just appeared blank. When I used Twig's raw function it then displayed the item again, but with the dreaded question mark in a black diamond.

Kitchen Ducting > Ducting Kits > Ducting Kit 4� / 100mm

Things that you should know that may help:

The staging test site and live site are both on the same server.
In my httpd.conf file I have 'AddDefaultCharset utf-8'.
In my php.ini file I have 'default_charset = "utf-8"'.
The HTML file served has the Content-Type meta tag 'content="text/html; charset=utf-8"'
My database is InnoDB and uses 'utf8' as the default character set and 'utf8_general_ci' as default collation. All tables in the database also use the defaults.

I looked into BOM with UTF8, but could not work out if that was a problem or not?

Solution

I managed to resolve the issue by editing the MySql config file on the server (/etc/my.cnf) and adding:

skip-character-set-client-handshake
collation-server=utf8_general_ci
character-set-server=utf8

That sorted it out for me. I'm not sure if it was something that had changed in Twig or Doctrine due to the upgrade to Symfony, but something in the upgrade caused the default charset to not work. The changes I made to the MySQL configuration meant that it will always fall back to UTF-8.