I am having trouble understanding the purpose of the -xhost flag used with icc.
On the intel website, it states:
xHost, QxHost
Tells the compiler to generate instructions for the highest instruction set available on the compilation host processor.
I am not sure what is meant by "highest instruction set".
Also, I see something about SIMD here. If -xhost can speed up your code, why would someone choose not to use this flag?
The -xhost flag generates the most optimal code possible, based on the capabilities of your current CPU (that is, the one in the computer you're using to do the compilation).
By "highest instruction set", it means that the compiler will automatically turn on the code-generation flags corresponding to the highest instruction set supported by your CPU. So, if your CPU only supports SSE2, then that's all that will be turned on. If it supports AVX2, then that option will be turned on. Whatever the highest instruction set extension that your CPU supports, the compiler will generate code targeting that instruction set extension.
This option is generally used when you want to build code to run on the same computer where you're building it. For example, when building a scientific algorithm that you'll run on the same computer, or when compiling your own Linux kernel.
Technically speaking, the generated binaries will run on any computer that supports at least the same instruction set extensions as the build computer, which is why the documentation talks about "the highest instruction set available on the compilation host processor".
As Peter Cordes already noted in a comment, ICC's -xhost flag is essentially equivalent to GCC and Clang's -march=native flag. Both of them tell the compiler to automatically turn on all options that match what the host CPU is capable of, generating the most optimal binary possible for the host CPU, but which will run on other CPUs, as long as they have equal or higher capabilities.
You can do exactly the same thing that -xhost is going to do by looking up the specifications for your computer's CPU and adding the corresponding code-gen options to the compiler command line. -xhost just does it for you, looking up what your host CPU supports and enabling those flags automatically, without you having to do the legwork. So, it is a convenience feature; nothing more, nothing less.
The -xhost flag can, indeed, speed up your code by taking advantage of certain instruction set extensions, but it can also result in a binary that won't work at all (on a different computer that doesn't support the same instruction set extensions as your build computer). Maybe that's not a problem for you; in that case, you'd definitely turn on the -host flag. But, in many cases, we software developers are building binaries for other people to run, and in that case, we have to be a bit more careful about exactly which CPUs we want to exclude.
It is also worth noting that Intel's compiler can actually generate a single executable with dynamic dispatching support that allows you to support two different architectures. See Sergey L.'s answer to a related question for more details.