Crossword puzzle focus won't move across the inputs with a number, Tried putting the wrapper and the number in the jQuery function

I am making a crossword and trying to get the focus to move right and left with the arrow keys and after a letter is typed, but it gets stuck on the squares with the numbers in them.

This is what my html looks like for the div with the numbers in them.

enter co<div><input class="letter" type="text" disabled /> </div>
    <div><input class="letter" type="text" disabled /> </div>
      <div class="wrapper">
      <input class="letter hideletter" type="text" maxlength="1" value="" placeholder="D" />
      <span class="num">1</span>
      </div>   
    <div><input class="letter" type="text" disabled /> </div>
    <div><input class="letter" type="text" disabled /> </div>

Jquery:

 //makes arrow keys work, but only works right to left and left to right at this point
$(document).keydown(
    function (e) {

        //makes focus jump to next input when a letter is typed
        $(".hideletter:focus").on('input', function() {
            $(".hideletter:focus").next().focus();


        });

        switch (e.which) {


            case 39: //right
               $(".hideletter:focus").next().focus();
                $(".num:focus").next().focus();
                break;
            case 40: //down doesn't work
                $(".hideletter:focus").next().focus();
                $(".num:focus").next().focus();
                break;

            case 37: //left
                $(".hideletter:focus").prev().focus();
                $(".num:focus").prev().focus();
                break; 

            case 38: //up doesn't work
                $(".hideletter:focus").next().focus();
                $(".num:focus").next().focus();
                break;



                default: return; // exit this handler for other keys  

        }
        e.preventDefault(); // prevent the default action (scroll / move caret)
    });

Here is a link to the crossword puzzle. https://jenniferlang1921.github.io/Crossword2/

Here is a link to my code: https://github.com/JenniferLang1921/Crossword2

Also, if you have any ideas on how to move the arrows up and down, that would be great.

Thanks!

Your code will break when the input element with a parent div , because the focus will always goes to next element , means the next element will be always <span class="num">4</span>

In order to fix it , you have to check a condition whether the current element has any parent , if has parent change the focus next to the parent

switch (e.which) {
                case 39:
              var focusedElement= $(document.activeElement);

               if(focusedElement.parent().hasClass('wrapper')){
                focusedElement.parent().next().focus();
               }
else{
                   $(".hideletter:focus").next().focus();
                   $(".num:focus").next().focus();
}

You can apply the same logic to rest of the key strokes you need