bashcsvawkwc

Word count command returning incorrect row count for CSV files


I am running the following command to count the number of rows in a CSV file:

wc -l filename.csv

Sometimes the row count is correct and other times it is 1 less than the actual count. All of the files are in the same format with the same header row. My first suspicion is the header row but it doesn't seem like the command would differentiate between the header and other rows. The files are saved with the same encoding utf-8.

Is this an issue with the formatting of the CSV files and/or a nuance of the wc command?


Solution

  • Assumptions:

    Setup some test files:

    $ printf "a\nb\nc"   > file1            # no \n on the last line
    $ printf "a\nb\nc\n" > file2
    
    $ od -c file1
    0000000   a  \n   b  \n   c  
    0000005                      ^^-- terimination is missing
    
    $ od -c file2
    0000000   a  \n   b  \n   c  \n
    0000006                      ^^-- terminate last line
    

    We can see that wc -l 'misses' the last line from file1:

    $ wc -l file1
    2 file1
    
    $ wc -l file2
    3 file2
    

    One awk approach that works for both files:

    $ awk 'END {print NR}' file1
    3
    
    $ awk 'END {print NR}' file2
    3
    

    An alternative approach would have the OP go through and append a \n to the last line of files where the \n is missing.

    A web search on bash insure last line has linefeed would be a good place to start looking for solutions ...