I am customizing the node/add and node/edit forms of a content type with a form_alter. In my content type, there is a file field that permits to upload files to the content.
What I would like to do is to customize the file box by changing the link to the file that is composed at runtime with Ajax. How can I do it without modifying Drupal core?
Your Private Files directory should not be in the docroot. Hiding it with a .htaccess rule will not work, as you point out in a comment.
Say you have Drupal in /var/www/sites/example.com/, then you should not store your private files under that directory; /var/www/sites/example.com/sites/default/private/files/ is just plain wrong.
You should, instead store the files where apache will not serve them, but can read them. E.g. in /var/www/files/example.com/. Then change the setting in Drupal to use that absolute path.
If you are running a large(r) site, you will probably want to store your files on a dedicated mount (drive, NFS etc.), say /media/nfs-example-com/.