Cocos2D Sprite Relative Movement

I have a problem that I can't seem to solve. I would like to move a sprite on the screen based on finger movements, but instead of making the sprite move towards the touch position I would like the sprite to move in the same way the finger moves on the screen.

Example: User puts his/her finger on the screen (any position) and then drags the finger on the screen by 200 pixel to the left, the sprite should also move of 200 pixels to the left.

My approach was to store the position of the player at the start and then calculating the difference between the start position and the current position of the finger to add to the starting position of the sprite itself.

Here is my code

-(void)ccTouchesBegan:(NSSet *)touches withEvent:(UIEvent *)event{
if([touches count] == 1){
    UITouch *touch = [touches anyObject];
    CGPoint startPos = [touch locationInView:[touch view]];
    startPosition = startPos;
}

}

-(void)ccTouchesMoved:(NSSet *)touches withEvent:(UIEvent *)event{
    if([touches count] == 1){
        UITouch *touch = [touches anyObject];
        CGPoint touchPos = [touch locationInView:[touch view]];
        CGPoint playerPos = player.position;

        float yDifference = startPosition.y - touchPos.y;
        playerPos.y = playerPos.y - yDifference;

        // just to change to GL coordinates instead of the ones used by Cocos2D
        // still doesn't work even if I remove this...
        CGPoint convertedPoint = [[CCDirector sharedDirector] convertToGL:playerPos];
        [player setPosition:convertedPoint];
    }

}

I hope the problem I have is clear but if it is not don't hesitate do ask questions or further explaining. I've been stuck on this for quite a while :'(

Also, if you want it to move "relative" to where it started as opposed to exactly where the touch is:

-(void)ccTouchesBegan:(NSSet *)touches withEvent:(UIEvent *)event{
    if([touches count] == 1){
      UITouch *touch = [touches anyObject];
      CGPoint startPos = [touch locationInView:[touch view]];
      startPosition = startPos;
      playerStartPos = playerPos;
    }
 }

-(void)ccTouchesMoved:(NSSet *)touches withEvent:(UIEvent *)event{
    if([touches count] == 1){
      UITouch *touch = [touches anyObject];
      CGPoint touchPos = [touch locationInView:[touch view]];

      CGPoint newPlayerPos;
      newPlayerPos.x = playerStartPos.x + (touchPos.x - startPosition.x);
      newPlayerPos.y = playerStartPos.y + (touchPos.y - startPosition.y);

      // just to change to GL coordinates instead of the ones used by Cocos2D
      // still doesn't work even if I remove this...
      CGPoint convertedPoint = [[CCDirector sharedDirector] convertToGL:newPlayerPos];
      [player setPosition:convertedPoint];
}

Your code is looking like it's "feeding back" onto itself by continually adding the difference to the playerPos.y