Where are the rules for annotation classes written for java

I am trying to understand how the annotation understands what to do when we use the annotation. I am not talking about the behaviors like when to execute that is covered by Retention, values etc. I want to understand how annotations understand the rules for that annotation. For example how does @Override annotation knows how to check if the function overrides a method in super class and so on. I tried digging a lot and I reached here but I don't get where the rules for the annotations are written. It feels like magic to me.

As already commented, processors (e.g. the compiler) must interpret the annotations, but the running program can also read/use the annotations (e.g by reflection).

For example the @Override annotation is used by the compiler, see it's documentation:

Indicates that a method declaration is intended to override a method declaration in a supertype. If a method is annotated with this annotation type compilers are required to generate an error message unless at least one of the following conditions hold:

• The method does override or implement a method declared in a supertype.
• The method has a signature that is override-equivalent to that of any public method declared in Object.

The @Override annotation is part of the standard API, other annotations may be part of some framework (e.g .JUnit @Test) or additional annotation processors, see Annotation Processing in javac. The developer can also declare annotations, see Annotation Interfaces.

In other words:
an Annotation is just that, an annotation. It is like a tag or mark that can be added to some elements (e.g. to a method, class, ...). There is no rule or anything similar directly attached to it. But some tools, like the compiler, or even normal Java code (some framework/library or written by you) can read and handle that annotation as desired.

There are a couple of annotations in the Java Language Specification (JLS) which compilers are required to handle. The action for the @Override (as example) is coded in the compiler, to do as specified in the JLS. Same for @Deprecated.