[SOLVED] Kinesis GetShardIterator... invalid because it did not come from this stream

Kinesis GetShardIterator... invalid because it did not come from this stream

I have build a KCL plus spark base on https://spark.apache.org/docs/1.2.0/streaming-kinesis-integration.html

I am running this on EMR (spark installed via bootstrap). I have created on stream sparkTest and did my testing its working fine . I have observed no DynamoDB was created . I have deleted stream and cluster . Next day again i have created Kinesis Steam with same name and deployed my code with newly launched cluster. Now i am getting

5/06/12 08:17:28 ERROR worker.InitializeTask: Caught exception:
com.amazonaws.services.kinesis.model.InvalidArgumentException: StartingSequenceNumber 49551532098093284204238000035066183240246145871536717826 used in GetShardIterator on shard shardId-000000000000 in stream sparkTest under account 618673372431 is invalid because it did not come from this stream. (Service: AmazonKinesis; Status Code: 400; Error Code: InvalidArgumentException; Request ID: 770ef875-10db-11e5-b24b-af6f372168ae)
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1078)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:726)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:461)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClie

I am not able to understand why this is coming. if i create new kinesis stream and then do work it works again. is it issue with Kinesis ?

One more thread is going into this is with https://github.com/awslabs/amazon-kinesis-connectors/issues/8 however i am not using kinesis app name and creating stream with

KinesisUtils.createStream(
                        jssc, streamName, endpointUrl, kinesisCheckpointInterval, InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2())

Solution

SparkConf sparkConfig = new SparkConf().setAppName("arbitraryName").setMaster("local[2]");

KinesisUtils.createStream(
                        jssc, streamName, endpointUrl, kinesisCheckpointInterval, InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2()));

If i change the name "arbitraryName". it works fine . i found this hint from the https://spark.apache.org/docs/1.2.0/streaming-kinesis-integration.html

key points:
The application name used in the streaming context becomes the Kinesis application name
The application name must be unique for a given account and region.