In a directory where source code resides, there are also Visual Studio related files, like *.user, *.vcxproj. And Debug and Release folders.
I would like to change project directory layout so it looks like this:
For example, the project is named ProjectX.
In the ProjectX main folder, there would be only two subfolders:
ProjectX/Source
ProjectX/Build
In ProjectX/Source there would be all source code files, but nothing else.
In ProjectX/Build there would be all Visual Studio-related files.
How can I do it?
To move the vcxproj into the build directory, move the vcxproj into the build directory.
To move your source files under a source directory, move your source files under a source directory.
After either or both of the above steps, you'll need to re-add the files to Visual Studio, but I'm sure you knew that. Visual Studio isn't psychic and can't guess where your files are if you move everything around.
The SDF file can be moved by asking Stack Overflow how to move the SDF file: How to change ipch path in Visual Studio 2010
Visual Studio 2015 should not be creating .user files any more. Perhaps you're not fully updated?
At this point you have everything you wanted, even though it's weird. Except for portable projects, people aren't generally moving their vcxproj files. The vcxproj file is typically considered the root of the project, just like a makefile would be; under it is build and source.
Now let me give you advice. Don't do all of that. Do this instead:
Move your source files in a source directory. Now in Visual Studio, go to project properties and change Output Directory and Intermediate Directory to $(ProjectDir)Build\$(Platform)\$(Configuration)\
Now you have one Build directory instead of a Debug and Release directory. That means a lot to some people. It means more to people with more build configurations.