Jenkins PollScm for specific repository

I want to build a project using two Git repositories. One of them contains the source code, while the other has the build and deployment scripts.

I am using Jenkins pipeline for building my project. The pipeline scripts are in Jenkins-pipeline repository and the Source code in middleware repository. As my pipeline scripts are in Jenkins-pipeline repository I am configuring my pipeline with the Jenkinsfile from Jenkins-pipeline repository.

Here is the Jenkins file I am using:

pipeline {
agent any
parameters {
    string(name: 'repo_branch', defaultValue: 'development', description: 'The branch to be checked out')
    string(name: 'git_repo',  defaultValue: 'ssh://git@my-server.com/middleware.git' description: 'Git repository from where we are going to checkout the code')

}
options {

    buildDiscarder(logRotator(numToKeepStr: '5'))
    disableConcurrentBuilds()
    timeout(time: 10, unit: 'MINUTES')
}
triggers {
    pollSCM('* * * * *')
}
stages {
    stage('Checkout git repo') {
        steps {
            git branch: "${params.repo_branch}", url: "${params.git_repo}", credentialsId: 'git-credentials'
        }
    }
    stage('Build binary') {
        steps {
            sh "./gradlew clean build -x test"
        }
    }
}
}

Now, as we can see, the repository I am cloning in the Jenkinsfile is different and the repository where I am keeping my jenkinsfile is different.

Now the issue here is:
In Jenkins file, I am using pollScm to trigger my Jenkins job whenever there is a new commit in the repository but I have two repositories that I have configured for the job so

1. Source Code Repository (middleware.git)
2. Pipeline Script Repository (Jenkins-pipeline.git)

So whenever there is a commit in either of these repositories my Jenkins job is getting triggered, which I don't want! I want to trigger My Jenkins build only when there is new commit in my Source code repository and should NOT trigger the build when there is commit in Jenkins-pipeline repositories. How do I do that?

Update

I am using shared libraries in the Jenkinsfile and these libraries are also in another repository. When I am committing something in shared library repository then also Jenkins jobs are getting triggered and I am using a shared library in multiple jobs so because of that all the jobs are getting triggered which I don't want.

P.S. I can't use the webhooks to trigger my Jenkins build because of my Jenkins is in private network!

For the library not to be included in the change-check, uncheck the 'Include @Library changes in job recent changes' checkbox in the 'system configuration' page for the relevant library, or set the new possible argument 'changelog=false' in the Pipeline '@Library' directive.

For the Pipeline itself to not be included in the 'pull scm' it is a bit more tricky. One option is to include the Jenkinsfile in the code-repository, but if the Jenkinsfile itself will be changed, a build will be triggered as well, and it is not really needed.

Another option is to include, in the start of the Pipeline a check whether the change triggering the job is in the source-repository or in the pipeline-repository, and if it wasn't in the source-repository stop the build.