Lua Script for Reading CSV Produces Incorrect Output Format on Last Line

Issue Description

I am writing a Lua script to read a CSV file and parse each line into a table. While processing the CSV file, I noticed that the output format of the last line is incorrect.

Here is the critical part of my code:

function split(s, delimiter)
    local result = {};
    for match in (s..delimiter):gmatch("(.-)"..delimiter) do
        table.insert(result, match);
    end
    return result;
end



function readCSV(filename)
    local file = io.open(filename, "r")
    if not file then
        error("Failed to open file: " .. filename)
    end

    local header = file:read()
    local headerArr = split(header, ",")
    print("length of headerArr: " .. #headerArr)
    for i, value in ipairs(headerArr) do
        print(i, value)
    end

    local data = {}

    for line in file:lines() do
        line = line:gsub("\n", ""):gsub("\r", "")
        print("line: " .. line)
        local valuesArr = split(line, ",")
        print("length of valuesArr: " .. #valuesArr)

        for i, value in ipairs(valuesArr) do
            -- print(i, value)
        end

        local entry = {}

        for i = 1, #valuesArr do
            print("headerArr[" .. i .. "]:" .. headerArr[i])
            print("valuesArr[" .. i .. "]:" .. valuesArr[i])
            -- print("headerArr[" .. i .. "] key = '" .. headerArr[i] .. "' , value = " .. valuesArr[i])
            print(string.format("headerArr[%d] key = '%s' , value = '%s'", i, headerArr[i], valuesArr[i]))
            print("-------------------------------------------------")
        end

        for key, value in pairs(entry) do
            -- print(">>> Entry[" .. key .. "]: " .. value)
        end

        -- table.insert(data, entry)
    end

    file:close()

    return data
end

-- Usage example
local filename = "/Users/weijialiu/Downloads/FC 24 CT v24.1.1.4/FC_24_LE_ICONS.csv"
local data = readCSV(filename)

-- Print the data
for i, entry in ipairs(data) do
    for header, value in pairs(entry) do
        print(header .. ": " .. value)
    end
    print("--------------------")
end

Problem Details

While processing the CSV file, the output format of the last line is incorrect. Specifically, the issue appears as:

...
headerArr[1021]:runningcode2
valuesArr[1021]:0
headerArr[1021] key = 'runningcode2' , value = '0'
-------------------------------------------------
headerArr[1022]:modifier
valuesArr[1022]:2
headerArr[1022] key = 'modifier' , value = '2'
-------------------------------------------------
headerArr[1023]:gkhandling
valuesArr[1023]:9
headerArr[1023] key = 'gkhandling' , value = '9'
-------------------------------------------------
headerArr[1024]:eyecolorcode
valuesArr[1024]:2
' , value = '2' key = 'eyecolorcode
-------------------------------------------------

As shown above, the output format is disrupted, which seems to be an issue with string concatenation or data processing.

What I've Tried

1. Ensured correct string concatenation before printing.
2. Checked the CSV file format to ensure there are no extra delimiters or newlines.
3. Added debugging information to check each step of data processing.

Despite these efforts, I still cannot pinpoint the cause of this issue. I would appreciate any help to resolve this problem. Thank you!

The strangest thing is that this issue only occurs with the last line in each loop. We have obtained the correct result, but when it is printed, there is a problem. This is very strange.

This is likely to be because you were reading a file with the CRLF as the line terminator, but you forgot to trim the CR character from the header line:

local header = file:read():gsub("\r", "")