select() doesn't use what is set for timeval

I have a snippet below - I've adjusted my tv_usec in several ways, but I keep sitting in a select() loop for almost exactly 10 seconds, regardless of what tv_usec is set at.

    char buffer[512];
    fd_set readfds;
    struct timeval tv;
    tv.tv_usec = 50;    

    int rv = 1;

    // clear the set ahead of time
    FD_ZERO(&readfds);

    // add our descriptors to the set
    FD_SET(mySocket, &readfds);

    // the n param for select()
    int n = mySocket + 1;

    while(rv != 0)
    {
        rv = select(n, &readfds, NULL, NULL, &tv);

        if (rv == -1)
            perror("select"); // error occurred in select()

        bzero(buffer,512);
        int n = recvfrom(mySocket,buffer,512,0,(struct sockaddr *)&server, &sockLen);

        // do stuff...
    }

tv.tv_sec needs to be initialized to something. Setting tv.tv_sec = 0 solved the issue.