apache-sparkgeomesa

GeoMesa Spark can't use UDF functions


I wanted to use GeoMesa UDF functions in Java, but I can't seem to use any of the functions, I have these imports related to GeoMesa:

import org.locationtech.jts.geom.*;
import org.apache.spark.sql.types.*;
import org.locationtech.geomesa.spark.jts.*;

but I can not use any of the UDF functions in it:

dataset.withColumn("column", st_makePoint(...));

I have these Maven dependencies:

    <dependency>
        <dependency>
        <groupId>org.locationtech.geomesa</groupId>
        <artifactId>geomesa-spark-jts_2.11</artifactId>
        <version>2.4.1</version>
    </dependency>
    <dependency>
        <groupId>org.locationtech.geomesa</groupId>
        <artifactId>geomesa-spark-sql_2.11</artifactId>
        <version>2.4.1</version>
    </dependency>
    <dependency>
        <groupId>org.locationtech.geomesa</groupId>
        <artifactId>geomesa-spark-core_2.11</artifactId>
        <version>2.4.1</version>
    </dependency>

it doesn't recognize st_makePoint at all, what can I do about this?


Solution

  • In order to add the geospatial UDF and UDTs to a Spark Session, one needs to call one of two pathways. For just JTS support, one can follow the steps here: https://www.geomesa.org/documentation/stable/user/spark/sparksql_functions.html (basically, to call .withJTS on the Spark Session).

    Alternatively, creating a dataframe from a GeoMesa data source using Spark SQL should register the UDTs and UDFs as well. (https://www.geomesa.org/documentation/stable/user/spark/sparksql.html#usage)

    A full list of the supported geospatial functions is here: https://www.geomesa.org/documentation/stable/user/spark/sparksql_functions.html

    As an edit, the Spark SQL API, DataFrame API, and PySpark APIs each require separate bindings in GeoMesa. In the event that I'm wrong, then the failure to be able to use a function from the Spark SQL Functions documentation in one of the other APIs is a bug and should be filled at the GeoMesa JIRA here: https://geomesa.atlassian.net.