I have a large (>10GB) file which is an InfluxDB line protocol export. Line protocol format is roughly
measurement,tag1=value1,tag2=value2,... value=XXX timestamp
for example
deconz.0.Sensors.10.pressure value=998 1622621407241000000
deconz.0.Sensors.10.pressure value=999 1622621970836000000
deconz.0.Sensors.10.pressure value=999 1622624177180000000
deconz.0.Sensors.10.pressure value=999 1622625419255000000
deconz.0.Sensors.10.pressure value=998 1622625975843000000
deconz.0.Sensors.11.battery value=85 1622621407241000000
deconz.0.Sensors.11.battery value=88 1622623616070000000
deconz.0.Sensors.11.battery value=88 1622660536826000000
deconz.0.Sensors.11.battery value=85 1622663594301000000
deconz.0.Sensors.11.battery value=88 1622666692089000000
deconz.0.Sensors.11.temperature value=21.44 1622621407241000000
deconz.0.Sensors.11.temperature value=21.61 1622646781032000000
deconz.0.Sensors.11.temperature value=21.64 1622650221200000000
deconz.0.Sensors.12.humidity value=55.54 1622621407242000000
deconz.0.Sensors.12.humidity value=55.7 1622633302339000000
deconz.0.Sensors.12.humidity value=55.73 1622636722283000000
deconz.0.Sensors.12.humidity value=55.89 1622640061715000000
deconz.0.Sensors.12.humidity value=55.96 1622643481822000000
deconz.0.Sensors.13.battery value=85 1622621407242000000
deconz.0.Sensors.13.battery value=85 1622908043752000000
deconz.0.Sensors.13.temperature value=24.01 1622621407242000000
deconz.0.Sensors.13.temperature value=24.13 1622626969228000000
deconz.0.Sensors.13.temperature value=24.21 1622630216027000000
deconz.0.Sensors.13.temperature value=24.33 1622630974954000000
deconz.0.Sensors.14.humidity value=47.72 1622632937200000000
deconz.0.Sensors.14.humidity value=47.8 1622633311833000000
deconz.0.Sensors.14.humidity value=46.7 1622636659393000000
deconz.0.Sensors.15.pressure value=1002 1622673441206000000
deconz.0.Sensors.15.pressure value=1002 1622685777307000000
deconz.0.Sensors.15.pressure value=1003 1622686242842000000
deconz.0.Sensors.16.temperature value=23.47 1622654455194000000
deconz.0.Sensors.16.temperature value=23.55 1622655939005000000
deconz.0.Sensors.16.temperature value=23.57 1622655959670000000
energymeter_total,uuid=c4695262-624c-11ea-b2f7-374e5ccddc43 value=30436.6 1622594844107000000
energymeter_total,uuid=c4695262-624c-11ea-b2f7-374e5ccddc43 value=30436.6 1622594908800000000
energymeter_total,uuid=c4695262-624c-11ea-b2f7-374e5ccddc43 value=30436.6 1622594973493000000
energymeter_total,uuid=c4695262-624c-11ea-b2f7-374e5ccddc43 value=30436.6 1622595158917000000
energymeter_total,manual=true value=26984.9 1592641140000000000
I want to split this file by measurement, ie. up to the first comma OR space, using the measurement name as the target filename.
This does the job (except for the comma as separator) but is dreadfully slow (runs for 8h on an Intel i5 with SSD storage):
cat ../influx_export | while read FILE VAL TS ; do echo "$FILE $VAL $TS" >> "$FILE" ; done
I'm sure there is a scripted solution (no compiled code) that is at least 10x faster. However, the source file is too big to fit entirely into RAM.
Are there any more efficient approaches using awk, perl, sed, ruby, whatever?
bash is notoriously slow for iterating over a file (because read only reads one character at a time to ensure it doesn't consume anything after a newline that may be intended for a following command to read).
Use awk instead:
awk -F'[, ]' '{
print $0 >> $1
}' ../influx_export
It's possible that, if there are many unique values for $1, you may wind up with a "too many files open" error. In that case, a simple (if inefficient) solution will be to explicitly close each file immediately after writing to it. Even if awk needs to open a file for each line, this should still be faster than using pure bash.
awk -F'[, ]' '{
print $0 >> $1; close($1)
}' ../influx_export