rubyfibers

How can I use Fiber in a way that's compatible with Enumerator?


The following class is designed to handle data that becomes available in chunks, while being able to write the processing function imperatively. I'm using Fiber to suspend execution when necessary to wait for more input:

class Filter
  def initialize
    @fiber = Fiber.new do
      run
    end
    @fiber.resume
  end

  def <<(chunk)
    @fiber.resume(chunk)
  end

  def each_two
    loop do
      a = Fiber.yield
      b = Fiber.yield
      yield a + b
    end
  end

  def run
    each_two do |chunk|
      puts chunk.inspect
    end
  end
end

filter = Filter.new
filter << "Hello"
filter << ", "
filter << "world"
filter << "!\n"
$ ruby filter.rb
"Hello, "
"world!\n"

I can even wrap each_two in an Enumerator:

  def run
    pairs = enum_for(:each_two)
    pairs.each do |chunk|
      puts chunk.inspect
    end
  end

But if I use external iteration, it breaks:

  def run
    pairs = enum_for(:each_two)
    loop do
      puts pairs.next.inspect
    end
  end
$ ruby filter.rb
nil
nil
filter.rb:20:in 'block in Filter#each_two': undefined method '+' for an instance of Fiber (NoMethodError)

      yield a + b
              ^
    from <internal:kernel>:168:in 'Kernel#loop'
    from filter.rb:17:in 'Filter#each_two'
    from filter.rb:in 'Enumerator#each'

I suspect that this is due to Enumerator using Fiber as an implementation detail. It's unfortunate that this is a leaky abstraction. Is there a way to make them work together nicely?

(I suspect this question is related to External iteration is broken but internal iteration works, but I'm not sure.)


Solution

  • With external iteration each_two runs in a different fiber which is where Fiber.yield returns control (instead of the main fiber) where iterator does its thing and resumes.

    You can use transfer instead to navigate between fibers. Transfer to main fiber where it waits and then transfer back to enum fiber:

    class Filter
      def initialize
        @main_fiber = Fiber.current
        @fiber = Fiber.new do
          run
        end
        @fiber.transfer
      end
    
      def <<(chunk)
        @enum_fiber.transfer(chunk)
      end
    
      def each_two
        @enum_fiber = Fiber.current
        loop do
          a = @main_fiber.transfer
          b = @main_fiber.transfer
          yield a + b
        end
      end
    
      def run
        pairs = enum_for(:each_two)
        loop do
          p pairs.next
        end
      end
    end
    
    f = Filter.new
    f << "h"
    f << "i"
    # => "hi"