How to Filter directories in a QFileDialog then set filter for filenames in selected directory

I'm trying to create a QFileDialog that will eventually select a file based on a filter, but I want to limit the user to only be able to select from certain directories from a common directory and I haven't had any luck.
I've tried Filtering directories displayed in a QFileDialog, qfiledialog - Filtering Folders? and How to set filter for directories in qfiledialog and haven't had any luck. I've tried creating a QSortFilterProxyModel but it isn't working as I'm expecting.
Here's what I currently have:

class FileFilterProxyModel : public QSortFilterProxModel
{
   protected:
      virtual bool filterAcceptsRow (int row, const QModelIndex &parent) const;
}

bool FileFilterProxyModel::filterAcceptsRow (int row, const QModelIndex &parent) const
{
   QModelIndex index0 = sourceModel()->index (row, 0, parent);
   QFileSystemModel *fileModel = qobject_cast<QFileSystemModel*> (sourceModel());

   if ((fileModel != NULL) and (fileModel->isDir (index0))
   {
      if (fileModel->fileName(index0).startsWith ("di_"))
      {
         return true;
      }
      else
      {
         return false;
      }
   }
   else
   {
      return false;
   }
}

Which is called by:

QFileDialog dialog;
FileFilterProxyModel *proxyModel = new FileFilterProxyModel;

dialog.setProxyModel (proxyModel);
dialog.setOption (QFileDialog::DontUseNativeDialog);
dialog.setDirectory (directoryName);
dialog.setFileMode (QFileDialog::ExistingFile);
dialog.exec ();

The result is really confusing me. When I run the QFileDialog with the FileFilterProxyModel it doesn't show directories (even though there are directories that match the filter). If I add a qDebug() statement before the check for the fileName it only shows the entries for the path to the specified directory.
What's really strange is that if I make the section where it checks if the directory name starts with "di_" to false and change the other case to true it shows everything but the directories that start with "di_", which I would expect.

I can't figure out how changing the result of the check on the start of the directory name would completely change the directories that are being checked.
Once I get the directories filtered and displayed, I'll then need to filter the subsequent filenames based on a different filter. Can I use the FileFilterProxyModel for that or do I need to do something different?

UPDATE
Thanks to @C137 I was able to get only the directories that I wanted displayed and I was able to get the files filtered by adding

dialog.SetNameFilter ("<enter filter here>");

I think I knew where the problem is, first I changed the FileFilterProxyModel to become the default filter, like this:

bool FileFilterProxyModel::filterAcceptsRow(int sourceRow, const QModelIndex &sourceParent) const
{
    QModelIndex index0 = sourceModel()->index(sourceRow, 0, sourceParent);
    if (!index0.isValid()) return false;
    QFileSystemModel *fileModel = qobject_cast<QFileSystemModel*>(sourceModel());
    auto fname = fileModel->fileName(index0);
    bool valid = QSortFilterProxyModel::filterAcceptsRow(sourceRow, sourceParent);
    if(fname == "A")valid = false;
    qDebug() << fname << " " << valid;
    return valid;
}

And setting dialog directory to "E:/A/B". As directory A is invalid, directories A and B won't be displayed along with all their content.

Now, when I set the filter to this implementation:

bool FileFilterProxyModel::filterAcceptsRow(int sourceRow, const QModelIndex &sourceParent) const
{
    QModelIndex index0 = sourceModel()->index(sourceRow, 0, sourceParent);
    if (!index0.isValid()) return false;
    QFileSystemModel *fileModel = qobject_cast<QFileSystemModel*>(sourceModel());
    auto fname = fileModel->fileName(index0);
    auto fpath = fileModel->filePath(index0);
    bool valid = fname.startsWith("di_") || fpath == "E:/" || fname == "A" || fname == "B";
    return valid;
}

Then directory B along side all it's directories that start with di_ are shown. I think this because all directories in the path to these directories are valid.

What's really strange is that if I make the section where it checks if the directory name starts with "di_" to false and change the other case to true it shows everything but the directories that start with "di_", which I would expect.

Now this is happening because everything in the path to these files/directories is valid, except for directories starting with di_ that's why they aren't shown.

Finally a complete clean implementation would be:

class FileFilterProxyModel : public QSortFilterProxyModel
{
public:
    FileFilterProxyModel(const QString &prefix, const QStringList &path)
        : m_prefix(prefix),
          m_path(path){}
private:
    QString m_prefix;
    QStringList m_path;
protected:
    virtual bool filterAcceptsRow(int source_row, const QModelIndex& source_parent) const;
};

bool FileFilterProxyModel::filterAcceptsRow(int sourceRow, const QModelIndex &sourceParent) const
{
    QModelIndex index = sourceModel()->index(sourceRow, 0, sourceParent);
    if (!index.isValid())
    {
        return false;
    }
    QFileSystemModel *fileModel = qobject_cast<QFileSystemModel*>(sourceModel());
    auto fname = fileModel->fileName(index);
    auto fpath = fileModel->filePath(index);
    if(fileModel->isDir(index))
    {
        // Directory
        if(fpath == m_path[0] || m_path.contains(fname))
        {
            // In path
            return true;
        }
        // Inside the target directory, validate by prefix.
        return fname.startsWith(m_prefix);
    }
    else
    {
        // Filter files the way you want
        return true;
    }
}

Which is called by:

QFileDialog dialog;
QStringList path;
QString prefix = "di_";
QString directoryName = "E:/A/B";
path << "E:/" << "A" << "B";
dialog.setOption (QFileDialog::DontUseNativeDialog);
dialog.setProxyModel(new FileFilterProxyModel("di_", path));
dialog.setDirectory (directoryName);
dialog.setFileMode (QFileDialog::ExistingFile);
dialog.exec ();

And now I can sleep in piece :D