sql-serverbcptediousnode-mssql

In nodeJS / SQL Server to bulk insert geometry with SRID?


I need to do a bulk insert (via bcp) into a SQL Server table with geometry data. I figured out how to get the operation working generally, however, there are a few criteria/issues that I have been having trouble with:

  1. The data needs to have an SRID. I am working with geometry columns, but I need to have SRID = 4326. I am ok with either a solution that lets me set them per row, or something that would allow me to configure the default SRID for a column.
  2. I would prefer not to need to write my own serializer to serialize the data to SQL Server's geometry UDT binary, but I am willing to if needed. However, part of the spec for the geometry UDT binary is to set the VALID bit and
  3. I do NOT want to write something to check the validity of the geometry. Writing something to ensure OGC compliance would already be beyond the complexity of what I want to get into, and the SQL Server validity conditions do not seem to be a perfect 1-to-1 match for the OGC standard (for instance, SQL Server also considers a LineString invalid if it overlaps itself). I want either SQL Server or a trusted utility to make the validity determination.
  4. I don't want to make multiple passes. For instance: using bcp to insert the rows and then making an update pass to set SRIDs on all of them.

I have seen examples of .NET applications being able to do this, but I am working from NodeJS. I'm not even sure if there is some SQL Server API that would let me do this, or if the .NET classes were using the UDT binary and had internal code to make the validity determination.

So my questions are:

  1. Is this even possible (from NodeJS)?
  2. If so, how?

I have tried many things.

  1. Doing the bulk insert using a varchar column and wkt. The SRID was set to 0 (SQL Server's default for geometry values)
  2. Seeing if there was a way to change the default SRID for a column (I haven't found anything)
  3. Looking for ways to call commands or other transformations on columns in a bulk insert, so that I could call geometry::STGeomFromText with an SRID. I have found nothing.
  4. Serializing the geometry to SQL Server's geometry UDT binary. Feels like node-mssql/tedious should do this already. Otherwise, doable, but I realized that the valid bit was going to be a problem if I didn't want to fake it
  5. Looking for code available on NodeJS that could validate whether a geometry was ogc compliant. I found one thing: @turf/boolean-valid. A simple reading of the code revealed some ogc rules that don't seem to be being checked. Plus, I later realized that SQL Server has some validity rules that don't perfectly match ogc anyway.
  6. Format files, mostly as an extension of 3. Again, I found nothing.
  7. Computed columns. This would probably technically work, but would require an additional two columns for every desired geometry column, one if we are fixing the SRID or just using the computed column for the validity setting.
  8. Triggers: this would probably also technically work, but it feels a bit too close to making a second pass and work would need to be taken to avoid recursion.

Solution

  • Posting an answer for the sake of those who may come looking later:

    The original question "Is this possible with SQL Server bcp from NodeJS?" remains unanswered.

    However, @siggemannen suggested serializing my recordset into JSON, which can then be sent to SQL Server in a VARCHAR(MAX) and turned back into a table value using the OPENJSON function. The columns of this table value can then be processed in SQL as you would when SELECTing from any other table.

    This allows efficiently processing datasets at a size that meets my use case.