Start a service application using StartService in C++

I created a windows service application in C++ using Visual Studio 2013. In the main() function, call another function whose body contains the following code: :

SERVICE_TABLE_ENTRY SerTable[] =
    {
        { const_cast<char *>(ServiceName.c_str()), (LPSERVICE_MAIN_FUNCTION)ServiceMain },
        { NULL, NULL }
    };
    int res = StartServiceCtrlDispatcher(SerTable);
    if (res == 0)
    {
        DEBUG_LOG(ServiceName+":StartServiceCtrlDispatcher failed", GetLastError());
        return QERROR;
    }

My main() function also contains system("start notepad");

I can successfully build the program and executable is generated. Now I try to open this exe from another program using OpenSCManager(), and CreateService(). By this the service is created and listed under Services.msc. I right-clicked and started the service and it shows status as started. But nothing happens..

Now if I double click on my exe it shows the message: error 1063: StartServiceControldispatcher Failed and then opens notepad.

Why notepad is not opened when the service is started under Services.msc?

You must launch notepad from your ServiceMain not main. ServiceMain is the function that is invoked when the service is started by the OS, so move your system("start notepad") call to your ServiceMain.

MORE CLARIFICATIONS

As a response to the comment by the OP - the error that happens now is 1063 or ERROR_FAILED_SERVICE_CONTROLLER_CONNECT. Documentation of StartServiceCtrlDispatcher explains that:

This error is returned if the program is being run as a console application rather than as a service.

That is why you get the error when launching the executable with a double click - you are launching it as a console app. When you will launch the service properly (either from Control Panel, command prompt or WinAPI), this function will succeed.

When the service control manager starts a service process, it waits for the process to call the StartServiceCtrlDispatcher function. The main thread of a service process should make this call as soon as possible after it starts up (within 30 seconds). If StartServiceCtrlDispatcher succeeds, it connects the calling thread to the service control manager and does not return until all running services in the process have entered the SERVICE_STOPPED state. The service control manager uses this connection to send control and service start requests to the main thread of the service process. The main thread acts as a dispatcher by invoking the appropriate HandlerEx function to handle control requests, or by creating a new thread to execute the appropriate ServiceMain function when a new service is started.

So yes, after a proper launch, ServiceMain will be called.