ionCube stores php files in encrypted format and it is installed as php extension but what i want to know is when I request the encrypted php file from non-encrypted php file how does php compiler executes it.
Does it send the encrypted file to ionCube server and get the original file and compile that or there is something else.
Means how the communication is going on between our server and ionCube. I guess it is through curl but i want to know how it works.
As you may have picked up on now, original code is never obtained, and processing is based on bytecode.
Here's some high level information that may help.
PHP Extensions
PHP has two types of extensions, module extensions such as CURL that typically wrap external APIs and expose their functionality via new PHP functions, and PHP engine extensions. Though the distinction isn't set in stone, engine extensions tend to interact with PHP's compiler and execution engine, though they may add new PHP functions too. ionCube is an engine extension that also adds PHP functions for its API and also to support ionCube24, though used also to be installable as a module extension using dl(). Both kinds of modules are shared libraries, and a single line to the php.ini file is used to add an extension to PHP, with PHP making use of OS functions to dynamically link the library into the running process.
Hooks
PHP has internal hooks that allow an extension to intercept the compile and execute stages of source file processing. An extension might use these simply to perform additional steps before or after regular processing, or replace the usual processing entirely. The ionCube Loader uses the compile hook to examine a file before the PHP engine compiles it, and takes over the task of processing the file if it is an ionCube file. The result of either reading an ionCube file or normal compilation is ultimately bytecode, however ionCube bytecode is non-standard, and with version 9 it may still be encrypted or unavailable for other reasons after initial processing of a file. As the standard execution engine cannot process ionCube bytecode, the Loader also uses the execution hook to take over execution of the compiled code if it was read from an ionCube encoded file. A further task of the Loader is to allow files produced for certain older version of PHP to run on newer versions, and where necessary the Loader performs on the fly transformations of the compiled code to make it usable on whatever version of PHP is running. PHP internals change significantly from time to time, most recently and most significantly between PHP 5 and 7, making this a challenging but important task for end user experience.
Processing of ionCube files does not require communication with outside servers, however since version 9, code can be protected with encryption keys that only exist when created at runtime by the PHP application itself, and an application developer may write PHP code that makes external calls to obtain data for constructing the decryption keys when required.
Encoded files
In terms of the files themselves, early PHP encoding tools of this type in essence compiled to bytecode and serialised this form directly to files. There was little knowledge and interest in PHP internals among developers in general, and this approach gave good protection and excellent performance. When interest first emerged in producing bytecode decompilers from a hacker group in China called the "Blue Wind" around 2006 ish, simply compiling to bytecode was clearly no longer acceptable. To varying degrees, tools such as ionCube then added more protection around the bytecode to hamper the task of successful reverse engineering. Though steps can be taken to limit the effectiveness of decompilation even if bytecode is recovered, the success at code protection still depends fundamentally on the ability to hide the necessary decoding key(s) though, and all encoding tools of this type store such a key in the encoded file itself.
In evolving code protection for ionCube version 9, a challenge was to address the limitation of stored keys, and the ability to encrypt code without storing the necessary decryption key statically anywhere was the obvious and necessary next step. This was added as a feature called "Dynamic Keys".
Hopefully that gives some insight into how ionCube and in some respects similar tools work. For more detailed knowledge of engine extension implementation, I'd recommend looking at the source code for the PHP OpCache and also Derick Rethans Xdebug.
Disclosure: I am associated with ionCube.