linuxbashcommand-linescripting

Total number of lines in a directory

I have a directory with thousands of files (100K for now). When I use wc -l ./*, I'll get: 

 c1            ./test1.txt
 c2            ./test2.txt
 ...
 cn            ./testn.txt
 c1+c2+...+cn  total

Because there are a lot of files in the directory, I just want to see the total count and not the details. Is there any way to do so?

I tried several ways and I got following error:
Argument list too long

Solution

  • If what you want is the total number of lines and nothing else, then I would suggest the following command:

    cat * | wc -l

    This concatenates the contents of all of the files in the current working directory and pipes the resulting blob of text through wc -l.

    I find this to be quite elegant. Note that the command produces no extraneous output.

    UPDATE:

    I didn't realize your directory contained so many files. In light of this information, you should try this command:

    for file in *; do cat "$file"; done | wc -l

    Most people don't know that you can pipe the output of a for loop directly into another command.

    Beware that this could be very slow. If you have 100,000 or so files, my guess would be around 10 minutes. This is a wild guess because it depends on several parameters that I'm not able to check.

    If you need something faster, you should write your own utility in C. You could make it surprisingly fast if you use pthreads.

    Hope that helps.

    LAST NOTE:

    If you're interested in building a custom utility, I could help you code one up. It would be a good exercise, and others might find it useful.