gocyclic-referencegob

Is there a way to serialize cyclic data structures with encoding/gob?


I'm working on porting a neural network library to Go. I want to be able to save and restore a trained network, so I'm attempting to serialize it directly. The problem is, the network struct contains cycles in its field (Neuron A has a connection to Neuron B, which has a connection to Neuron A). Whenever I try to serialize the entire network with encoding/gob, it fails with a stackoverflow.

Here's a very simple example of code that breaks in the same way:

package main

import (
    "bytes"
    "encoding/gob"
    "fmt"
    "log"
)

type P struct {
    Name    string
    Q *Q
}

type Q struct {
    Name string
    P *P
}

func main() {
    var network bytes.Buffer        // Stand-in for a network connection
    enc := gob.NewEncoder(&network) // Will write to network.
    dec := gob.NewDecoder(&network) // Will read from network.

    p := &P{ "P", nil }
    q := &Q{ "Q", p }
    p.Q = q

    err := enc.Encode(p)
    if err != nil {
        log.Fatal("encode error:", err)
    }
    // Decode (receive) the value.
    var p2 *P
    err = dec.Decode(&p2)
    if err != nil {
        log.Fatal("decode error:", err)
    }
    fmt.Printf("%#v", p2)
}

http://play.golang.org/p/LrO0VlLnX4

Barring rewriting the entire structure of the library to avoid cycles, is there a straightforward way to get around this problem?

Thanks


Solution

  • You can't use gob directly, but fear not brave citizen of the world!

    You can implement the BinaryMarshaler/BinaryUnmarshaler interfaces on your type as a work around and gob will happily use them instead when it's encoding/decoding your type.

    func (p *P) MarshalBinary() (_ []byte, err error) {
        var buf bytes.Buffer
        enc := gob.NewEncoder(&buf)
        enc.Encode(p.Name)
        if p.Q == nil {
            return buf.Bytes(), nil
        }
        isCyclic := p.Q != nil && p.Q.P == p
        enc.Encode(isCyclic)
        if isCyclic {
            p.Q.P = nil
            err = enc.Encode(p.Q)
            p.Q.P = p
        } else {
            err = enc.Encode(p.Q)
        }
        //buf.Encode
        return buf.Bytes(), err
    }
    
    func (p *P) UnmarshalBinary(data []byte) (err error) {
        dec := gob.NewDecoder(bytes.NewReader(data))
        if err = dec.Decode(&p.Name); err != nil {
            return
        }
        var isCyclic bool
        if err = dec.Decode(&isCyclic); err != nil {
            return
        }
        err = dec.Decode(&p.Q)
        if isCyclic {
            p.Q.P = p
        }
        return
    }
    

    playground

    warning creating a new decoder/encoder every time is extremely inefficient, you might want to look into using binary.*.