I have a slice with ~2.1 million log strings in it, and I would like to create a slice of slices with the strings being as evenly distributed as possible.
Here is what I have so far:
// logs is a slice with ~2.1 million strings in it.
var divided = make([][]string, 0)
NumCPU := runtime.NumCPU()
ChunkSize := len(logs) / NumCPU
for i := 0; i < NumCPU; i++ {
temp := make([]string, 0)
idx := i * ChunkSize
end := i * ChunkSize + ChunkSize
for x := range logs[idx:end] {
temp = append(temp, logs[x])
}
if i == NumCPU {
for x := range logs[idx:] {
temp = append(temp, logs[x])
}
}
divided = append(divided, temp)
}
The idx := i * ChunkSize
will give me the current "chunk start" for the logs
index, and end := i * ChunkSize + ChunkSize
will give me the "chunk end", or the end of the range of that chunk. I couldn't find any documentation or examples on how to chunk/split a slice or iterate over a limited range in Go, so this is what I came up with. However, it only copies the first chunk multiple times, so it doesn't work.
How do I (as evenly as possible) chunk an slice in Go?
You don't need to make new slices, just append slices of logs
to the divided
slice.
http://play.golang.org/p/vyihJZlDVy
var divided [][]string
chunkSize := (len(logs) + numCPU - 1) / numCPU
for i := 0; i < len(logs); i += chunkSize {
end := i + chunkSize
if end > len(logs) {
end = len(logs)
}
divided = append(divided, logs[i:end])
}
fmt.Printf("%#v\n", divided)