I have a problem.
I created a SCIO (Apache Beam) project via a sbt archetype : sbt new spotify/scio.g8
The goal of this Job is to Read a parquet file, from GS
When i use the ParquetIO provided by Apache Beam directly in SCIO, this work (GenericRecord) :
val SCHEMA: Schema =
new Schema.Parser()
.parse(
"{\n"
+ " \"namespace\": \"ioitavro\",\n"
+ " \"type\": \"record\",\n"
+ " \"name\": \"TestObject\",\n"
+ " \"fields\": [\n"
+ " {\"name\": \"field1\", \"type\": \"int\"}\n,"
+ " {\"name\": \"field2\", \"type\": \"int\"}\n"
+ " ]\n"
+ "}")
val inputIO = ParquetIO
.read(SCHEMA)
.from(options.getInputParquet)
sc.customInput("input", inputIO)
.map(file => file.toString)
.saveAsTextFile(args(outputTopic))
But i wanted to use the scio-parquet : https://spotify.github.io/scio/io/Parquet.html#read-avro-parquet-files
I have the following dependencies in the build.sbt file :
val scioVersion = "0.8.0-beta2"
val beamVersion = "2.16.0"
val scalaMacrosVersion = "2.1.1"
"com.spotify" %% "scio-core" % scioVersion,
"com.spotify" %% "scio-extra" % scioVersion,
"com.spotify" %% "scio-parquet" % scioVersion,
"com.spotify" %% "scio-avro" % scioVersion,
I use a schema
object TestModel {
@AvroType.fromSchema(
"""{
| "type":"record",
| "name":"TestObject",
| "namespace":"com.spotify.scio.avro",
| "doc":"Record for an account",
| "fields":[
| {"name":"field1","type":"int"},
| {"name":"field2","type":"int"}
| ]}
""".stripMargin)
class TestSchema
}
Then in the Job i use :
import com.spotify.scio.avro.TestObject
val projection = Projection[TestObject](_.getIntField, _.getIntField)
sc.parquetAvroFile[TestObject](args(inputParquet), projection)
.map(file => file.toString)
.saveAsTextFile(args(outputTopic))
But i have the following error :
object TestObject is not a member of package com.spotify.scio.avro
[error] import com.spotify.scio.avro.TestObject
[error] ^
[error] /mypath/my-job/src/main/scala/com/test/MyJob.scala:41:33: not found: type TestObject
[error] val projection = Projection[TestObject](_.getIntField, _.getIntField)
[error] ^
[error] /my-path/my-job/src/main/scala/test/MyJob.scala:59:26: not found: type TestObject
[error] sc.parquetAvroFile[TestObject](args(inputParquet), projection)
[error] ^
I don't understand why but maybe the case class TestObject is not generated correctly by the schema @AvroType.fromSchema.
Or maybe i don't use the api correctly, but i follow the link : https://spotify.github.io/scio/io/Parquet.html#read-avro-parquet-files
Thanks for your help.
With the sbt-avro plugin, this work.