I am having difficulty understanding the nomenclature for this scenario:
Say I have one web server, Server A, in an ALB target group, and users hitting that server.
I would like to take that server offline, and replace it with Server B, without too much interruption to the existing user sessions.
So, I would plan to add Server B to the target group, and hopefully route all NEW sessions to Server B. All existing sessions (and no new sessions) would continue to hit Server A. I could then decide an appropriate time to remove Server A, once old user activity has slowed or ceased on Server A.
It doesn't seem that deregistration is used for this purpose. I don't see settings for sticky-sessions that refer only to NEW sessions.
What would be the best approach for this scenario?
All sessions would have to be sticky, not just NEW sessions. In your description, the old sessions are "stuck" to the old server, and the new sessions are "stuck" to the new server. The closest you can get with ALB settings is to enable sticky sessions, and set an appropriate Deregistration Delay setting.
To have more control of this switch over you will need to use other AWS services besides ALB, such as AWS Global Accelerator.