Cadence workflow not executing activities after introducing versioning

I introduced cadence versioning in to cadence workflow and after that the workflow has stopped exeucting at the point of versioning introduction. I am receiving the following error :

2020-10-29T07:23:49.587Z DEBUG internal/internal_event_handlers.go:465 ExecuteActivity {"Domain": "domain_1", "TaskList": "tasklist_1", "WorkerID": "6@cdnc-5ddb9ccbb5-5dt5j@tasklist", "WorkflowType": "do_work_workflow", "WorkflowID": "CREATE", "RunID": "cab97b65-9892-48c5-b842-3f8b462d8602", "ActivityID": "4", "ActivityType": "do_Task_D"} 2020-10-29T07:23:49.620Z DEBUG internal/internal_task_handlers.go:1077 Cached state staled, new task has unexpected events {"Domain": "domain_1", "TaskList": "tasklist_1", "WorkerID": "6@cdnc-5ddb9ccbb5-5dt5j@tasklist1", "WorkflowID": "CREATE", "RunID": "cab97b65-9892-48c5-b842-3f8b462d8602", "CachedPreviousStartedEventID": 30, "TaskFirstEventID": 22, "TaskStartedEventID": 30, "TaskPreviousStartedEventID": 21}

My workflow code will look like this:

func doWorkflow(ctx workflow.Context, input string) error {
    err := doTaskA(input)
    if err != nil {
        return err
    }
    err = doTaskB(input)
    if err != nil {
        return err
    }
    versionTaskC := workflow.GetVersion(ctx, "ChangeID", workflow.DefaultVersion, 1)
    if versionTaskC == workflow.DefaultVersion {
        err = doTaskC(input)
        if err != nil {
            return err
        }
    } else {
        err = doTaskD(input)
        if err != nil {
            return err
        }
    }
    err = doTaskD2(input)
    if err != nil {
        return err
    }
    err = doTaskD3(input)
    if err != nil {
        return err
    }
    return nil
}

At the ChangeID , version returned is 1, and the workflow tries to execute TaskD but, it is not executing it, It is stuck in an infinite loop, trying to execute the TaskD.

The error message I get is

Cached state staled, new task has unexpected events

and

BadRequestError{Message: CadenceChangeVersion is not valid search attribute}

Can you please help me with this issue ?

there are two possibles:

1. You are running a Cadence server with version lower than 0.11.
2. [very likely]Your Cadence server upgraded from a lower version but didn't make the elasticSearch schema change:

cadence admin cluster add-search-attr --search_attr_key CadenceChangeVersion --search_attr_type 1

And you probably also need to add this:

cadence admin cluster add-search-attr --search_attr_key BinaryChecksums --search_attr_type 1

Background:

CadenceChangeVersion is introduced in https://github.com/uber/cadence/releases/tag/v0.11.0

It's to help searching for workflow change versions.

Let me know if this is not correct.