SFTP Server not allowing a Logic App- with routed traffic to one IP- to connect

So I've routed traffic from my Logic App (Standard) to one IP using this method(Through VNet and NAT Gateway setup)

Although when I've whitelisted that IP to an SFTP server (I've set one up with blob storage) and run the workflow, I get a BadGateway error. I even whitelisted or added the VNet that my Logic App was using to see if that would help- but no luck. When I allow all IP's to connect- the workflow succeeds.

Is there something I'm overlooking? Any way to get this to succeed with one IP?

EDIT: This didn't seem possible to do with a logic app at this time. I tried creating a Function App instead with this functionality, the comment section on the answer below solved my issue.

I have tried this several times, I do not think there is a way of doing this with the native SFTP connector in Logic Apps (if I am wrong about this I would love to be shown so). As I understand it the native SFTP connector has its traffic outbound from the pooled IPs of the service even if you vNet-integrate your Logic App, the connector is not running in your Logic App, in other words.

Docs:

1. SFTP-SSH is a managed connector: https://learn.microsoft.com/en-us/connectors/sftpwithssh/
2. Managed connectors are outbound from these IPs: https://learn.microsoft.com/en-us/connectors/common/outbound-ip-addresses#azure-logic-apps