spark submit - An existing connection was forcibly closed by the remote host [on master node ]
I have setup a spark cluster on my windows 7 machine locally. It has a master and a worker node. I have created a simple jar using sbt compile + sbt package and trying to submit it to the spark master node using spark-submit. Currently both the master and worker are on the same machine and if it works then the plan is to deploy a local cluster across multiple machines. Eventually all of this will be executed on Azure.
Master node
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Windows\system32>spark-class org.apache.spark.deploy.master.Master
2018-12-26 20:00:45 INFO Master:2612 - Started daemon with process name: 13968@ws-amalhotra
2018-12-26 20:00:45 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-12-26 20:00:45 INFO SecurityManager:54 - Changing view acls to: admin
2018-12-26 20:00:45 INFO SecurityManager:54 - Changing modify acls to: admin
2018-12-26 20:00:45 INFO SecurityManager:54 - Changing view acls groups to:
2018-12-26 20:00:45 INFO SecurityManager:54 - Changing modify acls groups to:
2018-12-26 20:00:45 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(admin); groups with view permissions: Set(); user
s with modify permissions: Set(admin); groups with modify permissions: Set()
2018-12-26 20:00:46 INFO Utils:54 - Successfully started service 'sparkMaster' on port 7077.
2018-12-26 20:00:46 INFO Master:54 - Starting Spark master at spark://192.168.8.101:7077
2018-12-26 20:00:46 INFO Master:54 - Running Spark version 2.3.2
2018-12-26 20:00:46 INFO log:192 - Logging initialized @1268ms
2018-12-26 20:00:46 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2018-12-26 20:00:46 INFO Server:419 - Started @1334ms
2018-12-26 20:00:46 INFO AbstractConnector:278 - Started ServerConnector@16391414{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-26 20:00:46 INFO Utils:54 - Successfully started service 'MasterUI' on port 8080.
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@204e3825{/app,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@748394e8{/app/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@19b99890{/,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c0f561c{/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3443bda1{/static,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54541f46{/app/kill,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e8c3d12{/driver/kill,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO MasterWebUI:54 - Bound MasterWebUI to 0.0.0.0, and started at http://ws-amalhotra.ivp.co.in:8080
2018-12-26 20:00:46 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22eb9260{/,null,AVAILABLE}
2018-12-26 20:00:46 INFO AbstractConnector:278 - Started ServerConnector@7e6d0324{HTTP/1.1,[http/1.1]}{192.168.8.101:6066}
2018-12-26 20:00:46 INFO Server:419 - Started @1394ms
2018-12-26 20:00:46 INFO Utils:54 - Successfully started service on port 6066.
2018-12-26 20:00:46 INFO StandaloneRestServer:54 - Started REST server for submitting applications on port 6066
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a4c3e84{/metrics/master/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5a3b4746{/metrics/applications/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO Master:54 - I have been elected leader! New state: ALIVE
2018-12-26 20:00:54 INFO Master:54 - Registering worker 192.168.8.101:8089 with 8 cores, 14.9 GB RAM
2018-12-26 20:01:20 INFO Master:54 - Driver submitted org.apache.spark.deploy.worker.DriverWrapper
2018-12-26 20:01:20 INFO Master:54 - Launching driver driver-20181226200120-0000 on worker worker-20181226200053-192.168.8.101-8089
2018-12-26 20:01:22 INFO Master:54 - Removing driver: driver-20181226200120-0000
2018-12-26 20:01:25 WARN TransportChannelHandler:78 - Exception in connection from /192.168.8.101:63501
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
2018-12-26 20:01:25 WARN TransportChannelHandler:78 - Exception in connection from /192.168.8.101:63557
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
2018-12-26 20:01:25 INFO Master:54 - 192.168.8.101:63501 got disassociated, removing it.
2018-12-26 20:01:25 INFO Master:54 - 192.168.8.101:63557 got disassociated, removing it.
2018-12-26 20:01:25 INFO Master:54 - 192.168.8.101:63556 got disassociated, removing it.
Worker node
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Windows\system32>spark-class org.apache.spark.deploy.worker.Worker spark://192.168.8.101:7077 -p 8089
2018-12-26 20:00:53 INFO Worker:2612 - Started daemon with process name: 13960@ws-amalhotra
2018-12-26 20:00:53 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-12-26 20:00:53 INFO SecurityManager:54 - Changing view acls to: admin
2018-12-26 20:00:53 INFO SecurityManager:54 - Changing modify acls to: admin
2018-12-26 20:00:53 INFO SecurityManager:54 - Changing view acls groups to:
2018-12-26 20:00:53 INFO SecurityManager:54 - Changing modify acls groups to:
2018-12-26 20:00:53 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(admin); groups with view permissions: Set(); user
s with modify permissions: Set(admin); groups with modify permissions: Set()
2018-12-26 20:00:53 INFO Utils:54 - Successfully started service 'sparkWorker' on port 8089.
2018-12-26 20:00:54 INFO Worker:54 - Starting Spark worker 192.168.8.101:8089 with 8 cores, 14.9 GB RAM
2018-12-26 20:00:54 INFO Worker:54 - Running Spark version 2.3.2
2018-12-26 20:00:54 INFO Worker:54 - Spark home: C:\spark
2018-12-26 20:00:54 INFO log:192 - Logging initialized @1367ms
2018-12-26 20:00:54 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2018-12-26 20:00:54 INFO Server:419 - Started @1411ms
2018-12-26 20:00:54 INFO AbstractConnector:278 - Started ServerConnector@319b7858{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
2018-12-26 20:00:54 INFO Utils:54 - Successfully started service 'WorkerUI' on port 8081.
2018-12-26 20:00:54 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@32b3c348{/logPage,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3db48501{/logPage/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2606a6f4{/,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6cf2a898{/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@75f7a48a{/static,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@563f35a1{/log,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO WorkerWebUI:54 - Bound WorkerWebUI to 0.0.0.0, and started at http://ws-amalhotra.ivp.co.in:8081
2018-12-26 20:00:54 INFO Worker:54 - Connecting to master 192.168.8.101:7077...
2018-12-26 20:00:54 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b2bed3e{/metrics/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO TransportClientFactory:267 - Successfully created connection to /192.168.8.101:7077 after 28 ms (0 ms spent in bootstraps)
2018-12-26 20:00:54 INFO Worker:54 - Successfully registered with master spark://192.168.8.101:7077
2018-12-26 20:01:20 INFO Worker:54 - Asked to launch driver driver-20181226200120-0000
2018-12-26 20:01:20 INFO DriverRunner:54 - Copying user jar file:/D:/_Work/azurepoc/sbtexample/target/scala-2.12/sbtexample_2.12-0.1.0-SNAPSHOT.jar to C:\spark\work\driver-20181226200120-00
00\sbtexample_2.12-0.1.0-SNAPSHOT.jar
2018-12-26 20:01:20 INFO Utils:54 - Copying D:\_Work\azurepoc\sbtexample\target\scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar to C:\spark\work\driver-20181226200120-0000\sbtexample_2.12-0.1
.0-SNAPSHOT.jar
2018-12-26 20:01:20 INFO DriverRunner:54 - Launch Command: "C:\Program Files\Java\jdk1.8.0_181\bin\java" "-cp" "C:\spark\bin\..\conf\;C:\spark\jars\*" "-Xmx1024M" "-Dspark.master=spark://19
2.168.8.101:7077" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/D:/_Work/azurepoc/sbtexample/target/scala-2.12/sbtexample_2.12-0.1.0-SNAPSHOT.jar"
"-Dspark.rpc.askTimeout=10s" "-Dspark.app.name=example1.HelloWorld" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.8.101:8089" "C:\spark\work\driver-20181226200120-00
00\sbtexample_2.12-0.1.0-SNAPSHOT.jar" "example1.HelloWorld"
2018-12-26 20:01:22 INFO Worker:54 - Driver driver-20181226200120-0000 exited successfully
Spark-submit code
C:\Users\amalhotra>spark-submit --deploy-mode cluster --master spark://192.168.
8.101:7077 --class "example1.HelloWorld" "D:\_Work\azurepoc\sbtexample\target\
scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar"
Running Spark using the REST application submission protocol.
2018-12-26 20:01:09 INFO RestSubmissionClient:54 - Submitting a request to laun
ch an application in spark://192.168.8.101:7077.
2018-12-26 20:01:19 WARN RestSubmissionClient:66 - Unable to connect to server
spark://192.168.8.101:7077.
Warning: Master endpoint spark://192.168.8.101:7077 was not a REST server. Falli
ng back to legacy submission gateway instead.
2018-12-26 20:01:19 WARN NativeCodeLoader:62 - Unable to load native-hadoop lib
rary for your platform... using builtin-java classes where applicable
C:\Users\amalhotra>spark-submit --deploy-mode cluster --master spark://192.168.
8.101:7077 --class "example1.HelloWorld" "D:\_Work\azurepoc\sbtexample\target\
scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar"
Jar code
package example1
import java.io._
object HelloWorld {
def main(args: Array[String]): Unit = {
println("===============================================")
println("===============================================")
println("Hello, world!")
println("===============================================")
println("===============================================")
}
}
I am getting the error on the master node:
2018-12-26 20:01:25 WARN TransportChannelHandler:78 - Exception in connection from /192.168.8.101:63501
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
2018-12-26 20:01:25 WARN TransportChannelHandler:78 - Exception in connection from /192.168.8.101:63557
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Things I have ensured :
- Its not a port related issue as I have completely disabled the antivirus and the firewall rules.
- Also I have replicated the entire thing on my personal machine (to ensure that its not an infrastructure related issue)
- Nothing in the event viewer
EDIT : Thanks a lot Sc0rpion for that hint. That was real silly of me. I was trying to submit the job to the spark master URL. Where as according to the link shared by Sc0rpion, it is to be submitted to the REST endpoint. I must have missed this while going through the documentation.
Also, I have a small concern now. I have ensured that my job is running by writing to a text file from my program, but I am unable to see anything getting printed on the console of the master or the worker when I am submitting my job to the spark master.
Code to submit job to master :
C:\Users\amalhotra>spark-submit --deploy-mode cluster --master spark://192.168.
8.101:6066 --class "example1.HelloWorld" "D:\_Work\azurepoc\sbtexample\target\
scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar"
Master reacts like below :
2018-12-28 13:54:25 INFO Master:54 - Driver submitted org.apache.spark.deploy.worker.DriverWrapper
2018-12-28 13:54:25 INFO Master:54 - Launching driver driver-20181228135425-0002 on worker worker-20181228134824-192.168.8.101-8089
2018-12-28 13:54:27 INFO Master:54 - Removing driver: driver-20181228135425-0002
Worker reacts :
2018-12-28 13:54:25 INFO Worker:54 - Asked to launch driver driver-20181228135425-0002
2018-12-28 13:54:25 INFO DriverRunner:54 - Copying user jar file:/D:/_Work/azurepoc/sbtexample/target/scala-2.12/sbtexample_2.12-0.1.0-SNAPSHOT.jar to C:\spark\work\driver-20181228135425-00
02\sbtexample_2.12-0.1.0-SNAPSHOT.jar
2018-12-28 13:54:25 INFO Utils:54 - Copying D:\_Work\azurepoc\sbtexample\target\scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar to C:\spark\work\driver-20181228135425-0002\sbtexample_2.12-0.1
.0-SNAPSHOT.jar
2018-12-28 13:54:25 INFO DriverRunner:54 - Launch Command: "C:\Program Files\Java\jdk1.8.0_181\bin\java" "-cp" "C:\spark\bin\..\conf\;C:\spark\jars\*" "-Xmx1024M" "-Dspark.master=spark://19
2.168.8.101:7077" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/D:/_Work/azurepoc/sbtexample/target/scala-2.12/sbtexample_2.12-0.1.0-SNAPSHOT.jar"
"-Dspark.app.name=example1.HelloWorld" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.8.101:8089" "C:\spark\work\driver-20181228135425-0002\sbtexample_2.12-0.1.0-SNAP
SHOT.jar" "example1.HelloWorld"
2018-12-28 13:54:27 INFO Worker:54 - Driver driver-20181228135425-0002 exited successfully
But nowhere I see the output of my application. It should have outputted like below :
Solution
As pointed by link - Unable to submit jobs to spark cluster (cluster-mode)
spark-submit --deploy-mode cluster --master spark://192.168.8.101:6066 --class "example1.HelloWorld" "D:_Work\azurepoc\sbtexample\target\scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar"
instead of port 7077
- Where is the documentation for scala.language.postfixOps?
- Chronomorphisms in Scala
- Remove all records which are duplicate in spark dataframe
- Running Scala from a script using terminal
- Karate-Gatling: Requests with query parameter in reports are not getting aggregated
- Ternary conditional operator in Scala
- Compiler as a service in Java
- Quill can't find case class property
- forming list of columns after groupByKey or groupBy
- Why is a `val` inside an `object` not automatically final?
- convert Rdd to dataframe
- Equivalent of Scala View in JavaScript
- How do I remove trailing elements in a Scala collection?
- Java compile speed vs Scala compile speed
- Adaptation of argument list by inserting () has been deprecated
- Reading Avro File in Spark
- Shuffle a list of integers with Java 8 Streams API
- What is the === (triple-equals) operator in Scala Koans?
- Expanding dataframe column into multiple columns
- Reactive streams (monix) operator combination for buffering for a timespan with overlapping elements
- Scala logging line numbers don't match using slf4s/slf4j?
- What are the available sbt new templates?
- getting the value of a key in a map in Scala
- Scala Type Mismatch Error when creating a Map
- How to split parquet files into many partitions in Spark?
- ExecutionContext issue in Akka Http Server
- Get field from Apache Spark Row which is Wrapped array as a Seq into a List using Scala
- What is the order of the scala compiler phases?
- Scala Akka HTTP Actor Class protocol Type Mismatch Error
- Convert Easting Northing Coordinates to Latitude and Longitude in Scala/Spark with EPSG Coordinate Transformation