Does there have to be a mode switch ie. from user to kernel mode in order for the action to be called a context switch?
No. Not all processors even have privilege levels.
A context switch usually means restoring an execution state from a Process Control Block. However, the notion of a context is implementation-defined. From Wikipedia:
What constitutes the context is determined by the processor and the operating system.
...
When a transition between user mode and kernel mode is required in an operating system, a context switch is not necessary; a mode transition is not by itself a context switch. However, depending on the operating system, a context switch may also take place at this time.