RowVirtualization cause incorrect background color for rows

I have a WPF application and there is a Datagrid in some pages. This datagrid needs to load 5000 rows at once (Pagination is not an option for me) and this takes ages. I set EnableRowVirtualization=True and the performance is acceptable now, but there is a problem here. In my datagrid I need to set different background colors to different rows depending on a column value (say STATUS), changing EnableRowVirtualization from False to True, caused incorrect coloring when I scroll.

----Edit----

Here is my XAML code:

<my:DataGrid Name="dgDataGrid" DockPanel.Dock="Top" AutoGenerateColumns="False"  ClipboardCopyMode="ExcludeHeader" 
                     CanUserDeleteRows="True" RowHeight="20" SelectionMode="Extended"  SelectionUnit="FullRow" FontFamily="Tahoma" 
                     ItemsSource="{Binding}"  VirtualizingStackPanel.VirtualizationMode="Recycling" VirtualizingStackPanel.IsVirtualizing="True" 
                     EnableRowVirtualization="True" EnableColumnVirtualization="False" IsSynchronizedWithCurrentItem="True" BorderBrush="Blue"  
                     RowBackground="White" HorizontalGridLinesBrush="Blue"  GridLinesVisibility="Horizontal" VerticalGridLinesBrush="Blue"  
                     IsTextSearchEnabled="False" IsTabStop="True" HeadersVisibility="All" Loaded="dgDataGrid_Loaded"  
                     ContextMenuOpening="dgDataGrid_ContextMenuOpening" LoadingRow="dgDataGrid_LoadingRow" 
                     ScrollViewer.IsDeferredScrollingEnabled ="True">
            <my:DataGrid.Resources>

            </my:DataGrid.Resources>

            <my:DataGrid.RowHeaderTemplate>
                <DataTemplate>
                    <TextBlock Text="{Binding RelativeSource={RelativeSource Mode=FindAncestor, AncestorType={x:Type my:DataGridRow}}, Path=Header}"></TextBlock>
                </DataTemplate>
            </my:DataGrid.RowHeaderTemplate>
            <my:DataGrid.ColumnHeaderStyle>
                <Style TargetType="my:DataGridColumnHeader">
                    <Setter Property="ContentTemplate">
                        <Setter.Value>
                            <DataTemplate>
                                <TextBlock Text="{Binding}" Foreground="Blue"/>
                            </DataTemplate>
                        </Setter.Value>
                    </Setter>
                </Style>
            </my:DataGrid.ColumnHeaderStyle>

            <my:DataGrid.ContextMenu>
                <ContextMenu Name="cmDataGrid" StaysOpen="True">
                    <MenuItem Name="mnuView" Header="نمایش">
                        <MenuItem Name="mnuHideColumn"  Header="Hide Column" Click="mnuHideColumn_Click"/>
                        <MenuItem Name="mnuShowColumn" Header="Show Column"/>
                        <Separator/>
                        <MenuItem Name="mnuGroupByColumn" 
                          Header="Group by this column" Click="mnuGroupColumn_Click" />
                        <MenuItem Name="mnuClearGroups" 
                          Header="Clear grouping" Click="mnuGroupColumn_Click" />
                        <Separator/>
                        <MenuItem Header="Header Alignment">
                            <MenuItem Name="mnuHeaderCenter" Header="Center"/>
                            <MenuItem Name="mnuHeaderLeft" Header="Left"/>
                            <MenuItem Name="mnuHeaderRight" Header="Right"/>
                        </MenuItem>
                        <MenuItem Header="Content Alignment">
                            <MenuItem Name="mnuContentCenter" Header="Center"/>
                            <MenuItem Name="mnuContentLeft" Header="Left"/>
                            <MenuItem Name="mnuContentRight" Header="Right"/>
                        </MenuItem>
                    </MenuItem>
                </ContextMenu>
            </my:DataGrid.ContextMenu>
        </my:DataGrid>

and following codes do the bindings:

Note: all my columns are getting generated on the fly depends on the object requested to be loaded:

public static DataGridColumn CreateTextBoxWithBackgroudColumn(DataColumn dataCol, string columnName)
        {
            DataGridTemplateColumn dgtc = new DataGridTemplateColumn();
            dgtc.Header = columnName;
            dgtc.HeaderStyle = (Style)(App.Current as App).FindResource("ColumnHeaderStyle");

            FrameworkElementFactory cellTemplateFactory = new FrameworkElementFactory(typeof(TextBlock));
            Binding dataBinding = new Binding(dataCol.ColumnName);
            dataBinding.Mode = BindingMode.TwoWay;
            dataBinding.UpdateSourceTrigger = UpdateSourceTrigger.PropertyChanged;

            dataBinding.Converter = new BackGroundConverter();

            cellTemplateFactory.SetBinding(TextBlock.BackgroundProperty, dataBinding);
            DataTemplate cellTemplate = new DataTemplate();
            cellTemplate.VisualTree = cellTemplateFactory;
            cellTemplate.Seal();
            dgtc.CellTemplate = cellTemplate;

            return dgtc;
        }

The returned DataGridColumn will be added to my DataGrid columns.

Is that possible to have both RowVirtualization and coloring at the same time?

Thanks.

As RowVirtualization is not working well for row background coloring, I added a column and colored it instead of whole row.