As a Java dev, I'm currently looking at Go because I think it's an interesting language.
To start with it, I decided to take a simple Java project I wrote months ago, and re-write it in Go to compare performances and (mainly, actually) compare the code readability/complexity.
The Java code sample is the following:
public static void main(String[] args) {
long start = System.currentTimeMillis();
Stream<Container> s = Stream.from(new Iterator<Container>() {
int i = 0;
@Override
public boolean hasNext() {
return i < 10000000;
}
@Override
public Container next() {
return new Container(i++);
}
});
s = s.map((Container _source) -> new Container(_source.value * 2));
int j = 0;
while (s.hasNext()) {
s.next();
j++;
}
System.out.println(System.currentTimeMillis() - start);
System.out.println("j:" + j);
}
public static class Container {
int value;
public Container(int v) {
value = v;
}
}
Where the map function is:
return new Stream<R>() {
@Override
public boolean hasNext() {
return Stream.this.hasNext();
}
@Override
public R next() {
return _f.apply(Stream.this.next());
}
};
And the Stream class is just an extension to java.util.Iterator to add custom methods to it. Other methods than map differs from standard Java Stream API.
Anyway, to reproduce this, I wrote the following Go code:
package main
import (
"fmt"
)
type Iterator interface {
HasNext() bool
Next() interface{}
}
type Stream interface {
HasNext() bool
Next() interface{}
Map(transformer func(interface{}) interface{}) Stream
}
///////////////////////////////////////
type incremetingIterator struct {
i int
}
type SampleEntry struct {
value int
}
func (s *SampleEntry) Value() int {
return s.value
}
func (s *incremetingIterator) HasNext() bool {
return s.i < 10000000
}
func (s *incremetingIterator) Next() interface{} {
s.i = s.i + 1
return &SampleEntry{
value: s.i,
}
}
func CreateIterator() Iterator {
return &incremetingIterator{
i: 0,
}
}
///////////////////////////////////////
type stream struct {
source Iterator
}
func (s *stream) HasNext() bool {
return s.source.HasNext()
}
func (s *stream) Next() interface{} {
return s.source.Next()
}
func (s *stream) Map(tr func(interface{}) interface{}) Stream {
return &stream{
source: &mapIterator{
source: s,
transformer: tr,
},
}
}
func FromIterator(it Iterator) Stream {
return &stream{
source: it,
}
}
///////////////////////////////////////
type mapIterator struct {
source Iterator
transformer func(interface{}) interface{}
}
func (s *mapIterator) HasNext() bool {
return s.source.HasNext()
}
func (s *mapIterator) Next() interface{} {
return s.transformer(s.source.Next())
}
///////////////////////////////////////
func main() {
it := CreateIterator()
ss := FromIterator(it)
ss = ss.Map(func(in interface{}) interface{} {
return &SampleEntry{
value: 2 * in.(*SampleEntry).value,
}
})
fmt.Println("Start")
for ss.HasNext() {
ss.Next()
}
fmt.Println("Over")
}
Both producing the same result but when Java takes about 20ms, Go takes 1050ms (with 10M items, test ran several times).
I'm very new to Go (started couple of hours ago) so please be indulgent if I did something really bad :-)
Thank you!
The other answer changed the original task quite "dramatically", and reverted to a simple loop. I consider it to be different code, and as such, it cannot be used to compare execution times (that loop could be written in Java as well, which would give smaller execution time).
Now let's try to keep the "streaming manner" of the problem at hand.
Note beforehand:
One thing to note beforehand. In Java, the granularity of System.currentTimeMillis() could be around 10 ms (!!) which is in the same order of magnitude of the result! This means the error rate could be huge in Java's 20 ms! So instead you should use System.nanoTime() to measure code execution times! For details, see Measuring time differences using System.currentTimeMillis().
Also this is not the correct way to measure execution times, as running things for the first time might run several times slower. For details, see Order of the code and performance.
Your original Go proposal runs on my computer roughly for 1.1 seconds, which is about the same as yours.
interface{} item typeGo doesn't have generics, trying to mimic this behavior with interface{} is not the same and have serious performance impact if the value you want to work with is a primitive type (e.g. int) or some simple structs (like the Go equivalent of your Java Container type). See: The Laws of Reflection #The representation of an interface. Wrapping an int (or any other concrete type) in an interface requires creating a (type;value) pair holding the dynamic type and value to be wrapped (creation of this pair also involves copying the value being wrapped; see an analysis of this in the answer How can a slice contain itself?). Moreover when you want to access the value, you have to use a type assertion which is a runtime check, so the compiler can't be of any help optimizing that (and the check will add to the code execution time)!
So let's not use interface{} for our items, but instead use a concrete type for our case:
type Container struct {
value int
}
We will use this in the iterator's and stream's next method: Next() Container, and in the mapper function:
type Mapper func(Container) Container
Also we may utilize embedding, as the method set of Iterator is a subset of that of Stream.
Without further ado, here is the complete, runnable example:
package main
import (
"fmt"
"time"
)
type Container struct {
value int
}
type Iterator interface {
HasNext() bool
Next() Container
}
type incIter struct {
i int
}
func (it *incIter) HasNext() bool {
return it.i < 10000000
}
func (it *incIter) Next() Container {
it.i++
return Container{value: it.i}
}
type Mapper func(Container) Container
type Stream interface {
Iterator
Map(Mapper) Stream
}
type iterStream struct {
Iterator
}
func NewStreamFromIter(it Iterator) Stream {
return iterStream{Iterator: it}
}
func (is iterStream) Map(f Mapper) Stream {
return mapperStream{Stream: is, f: f}
}
type mapperStream struct {
Stream
f Mapper
}
func (ms mapperStream) Next() Container {
return ms.f(ms.Stream.Next())
}
func (ms mapperStream) Map(f Mapper) Stream {
return nil // Not implemented / needed
}
func main() {
s := NewStreamFromIter(&incIter{})
s = s.Map(func(in Container) Container {
return Container{value: in.value * 2}
})
fmt.Println("Start")
start := time.Now()
j := 0
for s.HasNext() {
s.Next()
j++
}
fmt.Println(time.Since(start))
fmt.Println("j:", j)
}
Execution time: 210 ms. Nice, we're already sped it up 5 times, yet we're far from Java's Stream performance.
Iterator and Stream typesSince we can't use generics, the interface types Iterator and Stream doesn't really need to be interfaces, since we would need new types of them if we'd wanted to use them to define iterators and streams of another types.
So the next thing we do is we remove Stream and Iterator, and we use their concrete types, their implementations above. This will not hurt readability at all, in fact the solution is shorter:
package main
import (
"fmt"
"time"
)
type Container struct {
value int
}
type incIter struct {
i int
}
func (it *incIter) HasNext() bool {
return it.i < 10000000
}
func (it *incIter) Next() Container {
it.i++
return Container{value: it.i}
}
type Mapper func(Container) Container
type iterStream struct {
*incIter
}
func NewStreamFromIter(it *incIter) iterStream {
return iterStream{incIter: it}
}
func (is iterStream) Map(f Mapper) mapperStream {
return mapperStream{iterStream: is, f: f}
}
type mapperStream struct {
iterStream
f Mapper
}
func (ms mapperStream) Next() Container {
return ms.f(ms.iterStream.Next())
}
func main() {
s0 := NewStreamFromIter(&incIter{})
s := s0.Map(func(in Container) Container {
return Container{value: in.value * 2}
})
fmt.Println("Start")
start := time.Now()
j := 0
for s.HasNext() {
s.Next()
j++
}
fmt.Println(time.Since(start))
fmt.Println("j:", j)
}
Execution time: 50 ms, we've again sped it up 4 times compared to our previous solution! Now that's the same order of magnitude of the Java's solution, and we've lost nothing from the "streaming manner". Overall gain from the asker's proposal: 22 times faster.
Given the fact that in Java you used System.currentTimeMillis() to measure execution, this may even be the same as Java's performance. Asker confirmed: it's the same!
Now we're talking about roughly the "same" code which does pretty simple, basic tasks, in different languages. If they're doing basic tasks, there is not much one language could do better than the other.
Also keep in mind that Java is a mature adult (over 21 years old), and had an enormous time to evolve and be optimized; actually Java's JIT (just-in-time compilation) is doing a pretty good job for long running processes, such as yours. Go is much younger, still just a kid (will be 5 years old 11 days from now), and probably will have better performance improvements in the foreseeable future than Java.
This "streamy" way may not be the "Go" way to approach the problem you're trying to solve. This is merely the "mirror" code of your Java's solution, using more idiomatic constructs of Go.
Instead you should take advantage of Go's excellent support for concurrency, namely goroutines (see go statement) which are much more efficient than Java's threads, and other language constructs such as channels (see answer What are golang channels used for?) and select statement.
Properly chunking / partitioning your originally big task to smaller ones, a goroutine worker pool might be quite powerful to process big amount of data. See Is this an idiomatic worker thread pool in Go?
Also you claimed in your comment that "I don't have 10M items to process but more 10G which won't fit in memory". If this is the case, think about IO time and the delay of the external system you're fetching the data from to process. If that takes significant time, it might out-weight the processing time in the app, and app's execution time might not matter (at all).
Go is not about squeezing every nanosecond out of execution time, but rather providing you a simple, minimalist language and tools, by which you can easily (by writing simple code) take control of and utilize your available resources (e.g. goroutines and multi-core CPU).
(Try to compare the Go language spec and the Java language spec. Personally I've read Go's lang spec multiple times, but could never get to the end of Java's.)