scalascalding

Convert Seq to Pipe in Scalding

Context: I'm reading in a file where multiple fields are a list of IDs. I need to convert these fields into a Pipe to join them with other Pipes.

What I have tried:

val otherPipe = pipe
     .project('fieldIwant)
     .map { p: Pipe => p.toString.split(",") } // converts pipe -> array
     .unique
Solution

  • You can get a TypedPipe from a collection using:

    TypedPipe.from(Seq(1, 2, 3, 4, 555, 3))

    If you need to fall back to FieldsAPI (which is deprecated), you can do it using

    TypedPipe.from(Seq(1, 2, 3, 4, 555, 3)).toPipe('fieldName)