apache minify apache-nifi hortonworks-data-platform hortonworks-dataflow

Why the ExpressionLanguageScope in DBCPConnectionPool service is limited to only 'VARIABLE_REGISTRY' and not ' FLOWFILE_ATTRIBUTES'?

The DBCPConnectionPool Service requires 5 connection parameters to establish connection to a database as shown in the picture below [Marked Yellow]

I used UpdateAttribute Processor to manually add these 5 connection parameters and gave them their respective values as shown in the picture below [Marked Yellow]

Now, when I was trying to read the values for the connection parameters in DBCPConnectionPool Service through these attributes (Shown in picture below) , I was unable to read them.

To know the reason why the DBCPConnectionPool Service was unable to read the Flowfile attributes, I went ahead to check the source code for both DBCPConnectionPool Service and UpdateAttribute Processor.

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java

Souce code for DBCPConnectionPool Service :

Souce code for UpdateAttribute Processor :

Thus, I came to know the reason why it was unable to read the values from FlowFile attributes. This is because the ExpressionLanguageScope is limited to VARIABLE_REGISTRY and not FLOWFILE_ATTRIBUTES.

Now, My Question is that why the ExpressionLanguageScope for DBCPConnectionPool Service is limited to VARIABLE_REGISTRY. What is the reason for this limitation? The reason why I am asking this question is because I want to read the values for the connection parameters through FlowFile attributes.

Solution

For the same question that has been asked in the NiFi dev mailing list, Andy had answered it in the best way possible. The reason why DBCPConnectionPool services or any controller services for that matter, uses ExpressionLanguageScope.VARIABLE_REGISTRY is that, the controller services have no access to the flowfiles so it won't read the flowfiles' attributes. And for the question, why it only supports VARIABLE_REGISTRY is:

Just because it doesn't read flowfile attributes doesn't mean that it should not use attributes from elsewhere.
One of the major reasons why VARIABLE_REGISTRY was introduced was to avoid exposing the sensitive values which is the case when we pass around such values as flowfile attributes. The controller services fit this case, because many of them use sensitive properties like Password.

And if you're assuming that you can make it work just changing the the scope for those properties to ExpressionLanguageScope.FLOWFILE_ATTRIBUTES, you're wrong. Changing them makes no sense and doesn't work, the reason is again the controller services never get to access the flowfiles.

If there is a specific requirement for you wherein you need to use different property values for different flowfiles, Andy in the original dev thread had shared some links which I'm posting again: