Connection to FTP server sometimes works and others not

I have a ubuntu server (on Azure) running proftpd, when I try to connect to that server using FileZilla sometimes it works, sometimes it doesn't (usually it doesn't work at first... and I need to keep trying several random times before it works... and once it does it works for good...), now this is the error I receive it FileZilla logs:

Status: Resolving address of ftp.myserver.com
Status: Connecting to xx.xx.xx.xx:21...
Status: Connection established, waiting for welcome message...
Status: Insecure server, it does not support FTP over TLS.
Command:    USER my_user
Response:   331 Password required for my_user
Command:    PASS *******
Error:  Connection timed out after 20 seconds of inactivity
Error:  Could not connect to server
Status: Waiting to retry...
Status: Resolving address of ftp.myserver.com
Status: Connecting to xx.xx.xx.xx:21...
Status: Connection established, waiting for welcome message...
Response:   220 ProFTPD 1.3.5a Server (Debian) [xx.xx.xx.xx]
Command:    AUTH TLS
Response:   500 AUTH not understood
Command:    AUTH SSL
Response:   500 AUTH not understood
Status: Insecure server, it does not support FTP over TLS.
Command:    USER my_user
Response:   331 Password required for my_user
Command:    PASS *******
Error:  Connection timed out after 20 seconds of inactivity
Error:  Could not connect to server

and this is what I see in proftpd logs:

2016-08-09 10:26:37,263 FTP proftpd[33961] 10.0.0.6 (yy.yy.yy.yy[yy.yy.yy.yy]): USER my_user: Login successful.
2016-08-09 10:26:37,264 FTP proftpd[33961] 10.0.0.6 (yy.yy.yy.yy[yy.yy.yy.yy]): FTP session closed.
2016-08-09 10:26:37,468 FTP proftpd[33970] 10.0.0.6 (yy.yy.yy.yy[yy.yy.yy.yy]): FTP session opened.

I don't know why the server closes and reopens the connection after the login but I am no FTP expert...

Any thoughts on how to fix this?

Edit:

This is the content of proftpd.conf file

Solution

There are multiple possible causes for a delay at login time with ProFTPD. The most common causes are the mod_delay module (see its FAQ), or IdentLookups or UseReverseDNS.

However, since your delay happens after the PASS command has been sent, that rules out the IdentLookups or UseReverseDNS directives, as those pertain to the initial connection establishment, before any commands are sent.

Per discussion with the reporter, any latency added by mod_delay was ruled out. That leaves PAM, which, depending on the configuration (e.g. in /etc/pam.d/ftp) and the modules used, can add its own latency (over which ProFTPD has little control). To disable ProFTPD's use of PAM, you would use the following in the config:

<IfModule mod_auth_pam.c>
  AuthPAM off
</IfModule>

The reporter mentioned that disabling the use of PAM did indeed remove the delay -- thus pointing out that one of the PAM modules was the root cause.

Hope this helps!