After profiling drupal7.34 instance using XHProf I see that the opendir function called by file_scan_dir which recurses to 15 depth is taking long execution time. file_scan_dir is in turn called by drupal_system_listing (3 calls)
Function Calls Ex Wall Wall %
opendir 1608 500ms 27%
ob_flush 1 270ms 17%
is_dir 10,872 107ms 7%
preg_match 20,597 85ms 5%
Key Settings
**php.ini (PHP5.3.3)**
memory_limit = 512MB
realpath_cache_size = 1024k
max_execution_time = 90
**apc.ini**
apc.shm_size=192M (free 100MB after several loads)
apc.stat=1
apc.stat_ctime=0
The drupal files are on a NFS filesystem so I expect it to be bit slow. When I change apc.stat=0 and restart apache. I expected opendir overhead to disappear as most of the files will be read from opcache, but these numbers don't change at all after many reloads of the same page.
I did check that I have no missing modules which is a known issue with drupal 7. Still it looks like drupal is searching for some file which is missing.
Any ideas what may be causing this issue?
I found the problem by temporarily adding dpm(func_get_args()); in bootstrap.inc as discussed here https://drupal.stackexchange.com/a/138345
Turned out that AT-commerce version 7.x-3.0 theme which is not properly checking other optional modules causing severe performance degradation.
The fix saved over 900ms on every page load, which has a big impact
I have provided a suggested fix for the drupal module https://www.drupal.org/node/2406089 Listing it here for completeness
/**
* Implements hook_css_alter().
*/
function at_commerce_css_alter(&$css) {
// Replace all Commerce module CSS files with our own for total control over all styles.
// Commerce module uses the BAT CSS file naming convention (base, admin, theme).
$path = drupal_get_path('theme', 'at_commerce');
// cart
if (module_exists('commerce_cart')){
$cart_theme = drupal_get_path('module', 'commerce_cart') . '/theme/commerce_cart. theme.css';
if (isset($css[$cart_theme])) {
$css[$cart_theme]['data'] = $path . '/css/commerce/commerce_cart.theme.css';
$css[$cart_theme]['group'] = 1;
}
}
// checkout
if (module_exists('commerce_checkout')){
$checkout_base = drupal_get_path('module', 'commerce_checkout') . '/theme/ commerce_checkout.base.css';
$checkout_admin = drupal_get_path('module', 'commerce_checkout') . '/theme/ commerce_checkout.admin.css';
$checkout_theme = drupal_get_path('module', 'commerce_checkout') . '/theme/ commerce_checkout.theme.css';
if (isset($css[$checkout_base])) {
$css[$checkout_base]['data'] = $path . '/css/commerce/commerce_checkout.base. css';
$css[$checkout_base]['group'] = 1;
}
if (isset($css[$checkout_admin])) {
$css[$checkout_admin]['data'] = $path . '/css/commerce/commerce_checkout.admin. css';
$css[$checkout_admin]['group'] = 1;
}
if (isset($css[$checkout_theme])) {
$css[$checkout_theme]['data'] = $path . '/css/commerce/commerce_checkout.theme. css';
$css[$checkout_theme]['group'] = 1;
}
}
// customer
if (module_exists('commerce_customer')){
$customer_admin = drupal_get_path('module', 'commerce_customer') . '/theme/ commerce_customer.admin.css';
if (isset($css[$customer_admin])) {
$css[$customer_admin]['data'] = $path . '/css/commerce/commerce_customer.admin. css';
$css[$customer_admin]['group'] = 1;
}
}
// file (contrib)
if (module_exists('commerce_file')){
$file_forms = drupal_get_path('module', 'commerce_file') . '/theme/commerce_file. forms.css';
if (isset($css[$file_forms])) {
$css[$file_forms]['data'] = $path . '/css/commerce/commerce_file.forms.css';
$css[$file_forms]['group'] = 1;
}
}
// line items
if (module_exists('commerce_line_item')){
$line_item_admin = drupal_get_path('module', 'commerce_line_item') . '/theme/ commerce_line_item.admin.css';
$line_item_theme = drupal_get_path('module', 'commerce_line_item') . '/theme/ commerce_line_item.theme.css';
if (isset($css[$line_item_admin])) {
$css[$line_item_admin]['data'] = $path . '/css/commerce/commerce_line_item.admin. css';
$css[$line_item_admin]['group'] = 1;
}
if (isset($css[$line_item_theme])) {
$css[$line_item_theme]['data'] = $path . '/css/commerce/commerce_line_item.theme. css';
$css[$line_item_theme]['group'] = 1;
}
}
// order
if (module_exists('commerce_order')){
$order_admin = drupal_get_path('module', 'commerce_order') . '/theme/ commerce_order.admin.css';
$order_theme = drupal_get_path('module', 'commerce_order') . '/theme/ commerce_order.theme.css';
if (isset($css[$order_admin])) {
$css[$order_admin]['data'] = $path . '/css/commerce/commerce_order.admin.css';
$css[$order_admin]['group'] = 1;
}
if (isset($css[$order_theme])) {
$css[$order_theme]['data'] = $path . '/css/commerce/commerce_order.theme.css';
$css[$order_theme]['group'] = 1;
}
}
// payment
if (module_exists('commerce_payment')){
$payment_admin = drupal_get_path('module', 'commerce_payment') . '/theme/ commerce_payment.admin.css';
$payment_theme = drupal_get_path('module', 'commerce_payment') . '/theme/ commerce_payment.theme.css';
if (isset($css[$payment_admin])) {
$css[$payment_admin]['data'] = $path . '/css/commerce/commerce_payment.admin. css';
$css[$payment_admin]['group'] = 1;
}
if (isset($css[$payment_theme])) {
$css[$payment_theme]['data'] = $path . '/css/commerce/commerce_payment.theme. css';
$css[$payment_theme]['group'] = 1;
}
}
// price
if (module_exists('commerce_price')){
$price_theme = drupal_get_path('module', 'commerce_price') . '/theme/ commerce_price.theme.css';
if (isset($css[$price_theme])) {
$css[$price_theme]['data'] = $path . '/css/commerce/commerce_price.theme.css';
$css[$price_theme]['group'] = 1;
}
}
// product
if (module_exists('commerce_product')){
$product_admin = drupal_get_path('module', 'commerce_product') . '/theme/ commerce_product.admin.css';
$product_theme = drupal_get_path('module', 'commerce_product') . '/theme/ commerce_product.theme.css';
if (isset($css[$product_admin])) {
$css[$product_admin]['data'] = $path . '/css/commerce/commerce_product.admin. css';
$css[$product_admin]['group'] = 1;
}
if (isset($css[$product_theme])) {
$css[$product_theme]['data'] = $path . '/css/commerce/commerce_product.theme. css';
$css[$product_theme]['group'] = 1;
}
}
// tax
if (module_exists('commerce_tax')){
$tax_admin = drupal_get_path('module', 'commerce_tax') . '/theme/commerce_tax. admin.css';
$tax_theme = drupal_get_path('module', 'commerce_tax') . '/theme/commerce_tax. theme.css';
if (isset($css[$tax_admin])) {
$css[$tax_admin]['data'] = $path . '/css/commerce/commerce_tax.admin.css';
$css[$tax_admin]['group'] = 1;
}
if (isset($css[$tax_theme])) {
$css[$tax_theme]['data'] = $path . '/css/commerce/commerce_tax.theme.css';
$css[$tax_theme]['group'] = 1;
}
}
}