first time posting here! I am trying to save my Logistic Regression model via pyspark2pmml. However I keep getting the error stated in the title. I will post my pipeline and model code.
from pyspark.ml.feature import Binarizer
binarizer = Binarizer(threshold=10000, inputCol="traffic_count", outputCol="label")
stages = []
stages = [binarizer]
from pyspark.ml import Pipeline
from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler
SI_roadname = StringIndexer(inputCol='road_name',outputCol='road_Index')
SI_suburb = StringIndexer(inputCol='suburb',outputCol='suburb_Index')
SI_cardinal = StringIndexer(inputCol='cardinal_direction_name',outputCol='cardinal_Index')
SI_period = StringIndexer(inputCol='period',outputCol='period_Index')
SI_label = StringIndexer(inputCol='label',outputCol='label_index')
stages = []
stages += [SI_roadname, SI_suburb, SI_cardinal, SI_period, SI_label]
OHE = OneHotEncoder(inputCols['road_Index','suburb_Index','cardinal_Index','period_Index','label_index'],outputCols=['road_OHE','suburb_OHE','cardinal_OHE','period_OHE','label_OHE'])
stages += [OHE]
assembler = VectorAssembler(inputCols=['wgs84_latitude','wgs84_longitude'],outputCol='features')
stages += [assembler]
pipeline = Pipeline(stages=stages)
pipelineModel = pipeline.fit(df)
model = pipelineModel.transform(df)
from pyspark.ml.linalg import DenseVector
input_data = model.rdd.map(lambda x: (x["label"], DenseVector(x["features"])))
df_train = sqlContext.createDataFrame(input_data, ["label", "features"])
train, test = df_train.randomSplit([0.7,0.3])
lr = LogisticRegression(labelCol='label')
lr_model = lr.fit(train)
pred_labels = lr_model.evaluate(test)
pred_labels.predictions.show()
So the particular error I got comes from this line
from pyspark2pmml import PMMLBuilder
PMMLBuilder(spark, df, pipelineModel)
PMMLBuilder.buildFile("lr_model.pmml","path")
I am very new to using using Pyspark in general so I hope someone can give me a helping hand. I will post some screen shots too for context.
You're overwriting self argument of the object with the string "lr_model.pmml". That's why you're receiving the error AttributeError: 'str' object has no attribute 'sc' Pyspark PMML.
You have to call buildFile passing a path as argument, see.
def buildFile(self, path):
javaFile = self.sc._jvm.java.io.File(path)
javaFile = self.javaPmmlBuilder.buildFile(javaFile)
return javaFile.getAbsolutePath()
From the README of the library:
from pyspark2pmml import PMMLBuilder
pmmlBuilder = PMMLBuilder(sc, df, pipelineModel)
pmmlBuilder.buildFile("DecisionTreeIris.pmml")