linuxbashsedawkvi

Delete empty lines in csv file


I have a file with 4 million of lines, every line ends with the char $, but I mistakenly add a new line after the the line delimiter while scraping a website, so right now it is looking like this:

fist name, last name, phone, address, postal code, city, region,$

$

fist name, last name, phone, address, postal code, city, region,$

$

the new line '$' only shows up of course if I use :set list, but I'm trying to use this file for a bulk insert in mysql and I'm having problems with it now.

I would like to change the file to:

fist name, last name, phone, address, postal code, city, region,$

fist name, last name, phone, address, postal code, city, region,$

How can I do this? with sed or awk or even vi ? looked up around and what I found is not really applying to this case.

please don't take in consideration the extra empty line shown above.

Thanks in advance


Solution

  • To remove blank lines with sed:

    sed -i '/^$/d' yourfile.csv
    

    To remove lines consisting of a single $:

    sed -i '/^$$/d' yourfile.csv
    

    Most versions of sed support the -i switch; if yours does not you will need e.g. sed '/^$$/d' yourfile.csv > newfile.csv.

    Removing blank lines with white space is more complicated. This usually works:

    sed '/^ *$/d' yourfile.csv
    

    If this is not sufficient, try checking also for tabs. For older sed's, this will work:

    sed '/^[ X]*$/d' yourfile.csv
    

    where X here a tab, entered via Control-V Tab.

    Newer sed's will take a [ \t\r]* or \s* or [[:space:]]*, sometimes requiring a -E switch.