SCHED_FIFO thread is preempted by SCHED_OTHER thread in Linux

I have written a test program to test out SCHED_FIFO. I have learnt that SCHED_FIFO cannot be preempted by SCHED_OTHER threads. But I couldn't explain the results obtained when same program is run multiple times.

/* Includes */
#include <unistd.h>     /* Symbolic Constants */
#include <sys/types.h>  /* Primitive System Data Types */ 
#include <errno.h>      /* Errors */
#include <stdio.h>      /* Input/Output */
#include <stdlib.h>     /* General Utilities */
#include <pthread.h>    /* POSIX Threads */
#include <string.h>     /* String handling */
#include <sched.h>

/* prototype for thread routine */
void print_message_function ( void *ptr );
void print_message_function1 ( void *ptr );

/* struct to hold data to be passed to a thread
 * this shows how multiple data items can be passed to a thread
 */
typedef struct str_thdata
{
    int thread_no;
    int thread_value;
    char message[100];
} thdata;

int main()
{
    pthread_t thread1, thread2;  /* thread variables */
    thdata data1, data2;         /* structs to be passed to threads */

    /* initialize data to pass to thread 1 */
    data1.thread_no = 1;
    data1.thread_value = 0;
    strcpy(data1.message, "Hello!");

    /* initialize data to pass to thread 2 */
    data2.thread_no = 2;
    data2.thread_value = 10000;
    strcpy(data2.message, "Hi!");

    /* create threads 1 and 2 */    
    pthread_create (&thread1, NULL, (void *) &print_message_function, (void *) &data1);
    pthread_create (&thread2, NULL, (void *) &print_message_function1, (void *) &data2);

    /* Main block now waits for both threads to terminate, before it exits
     * If main block exits, both threads exit, even if the threads 
     * have not finished their work 
     */ 
    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);
          
    /* exit */  
    exit(0);
} /* main() */

/** 
* print_message_function is used as the start routine for the threads used
* it accepts a void pointer 
**/
void print_message_function ( void *ptr )
{
    thdata *data;            
    data = (thdata *) ptr;  /* type cast to a pointer to thdata */

    struct sched_param param;
    //int priority=10;
    /* sched_priority will be the priority of the thread */
    //param.sched_priority = priority;
    /* only supported policy, others will result in ENOTSUP */

    int policy = SCHED_OTHER;
    /* scheduling parameters of target thread */
    pthread_setschedparam(pthread_self(), policy, &param);
    printf("Thread %d says sched policy  %d \n", data->thread_no, SCHED_OTHER);
    pthread_getschedparam(pthread_self(),&policy,&param);

    printf("Thread %d says %s  %d \n", data->thread_no, data->message,policy);

    int i=0;
    /* do the work */
    printf("Thread %d says %s %d \n", data->thread_no, data->message,(int)pthread_self());
    for(i=0;i<100;i++) 
        printf("Thread %d says %d \n", data->thread_no,data->thread_value++);
    pthread_exit(0); /* exit */
} /* print_message_function ( void *ptr ) */

void print_message_function1 ( void *ptr )
{
    thdata *data;            
    data = (thdata *) ptr;  /* type cast to a pointer to thdata */

    struct sched_param param;
    int priority=10;
    /* sched_priority will be the priority of the thread */
    param.sched_priority = priority;
    /* only supported policy, others will result in ENOTSUP */

    int policy = SCHED_FIFO;
    /* scheduling parameters of target thread */
    pthread_setschedparam(pthread_self(), policy, &param);
    printf("Thread %d says sched policy %d \n", data->thread_no, SCHED_FIFO);

    pthread_getschedparam(pthread_self(),&policy,&param);

    printf("Thread %d says %s  %d \n", data->thread_no, data->message,policy);

    int i=0;
    /* do the work */
    printf("Thread %d says %s  %d \n", data->thread_no, data->message,(int)pthread_self());
    for(i=0;i<100;i++)
        printf("Thread %d says %d \n", data->thread_no,data->thread_value++);
    pthread_exit(0); /* exit */
} /* print_message_function ( void *ptr ) */

I have got unexpected results in multiple runs where I have seen SCHED_FIFO is preempted by SCHED_OTHER thread, i.e. as per program, thread 2 is in FIFO mode, while thread 1 is SCHED_OTHER mode. I have seen multiple times where thread2 is preempted by thread1.

Can someone help me out in finding the issue?

You probably have these sysctl settings in effect, which are default values:

kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000

This means that real time threads are allowed to hog only 95% of every 1 second period.

Also see: Can't provoke Priority Inversion in C++