I have a jar deployed to GitLab Package Registry, which I'm trying to add to my Spark Application. However, it just can't get authorized to the password-protected registry.
I've tried setting spark.jars.packages & spark.jars.repositories, but even with the hardcoded token it's not able to authorize correctly. Although the URL it tried to use to fetch the pom/jar works just fine when copied and used directly with curl.
Looking for a solution, I came across this blog post: https://xebia.com/blog/spark-packages-from-a-password-protected-repository/.
Came up with following Ivy settings:
<ivysettings>
<settings defaultResolver="default" />
<credentials
host="private.git.domain.com"
username="read_only"
passwd="<TOKEN>"/>
<resolvers>
<ibiblio name="maven2" m2compatible="true" />
<ibiblio name="gitlab" m2compatible="true"
root="https://private.git.domain.com/api/v4/projects/<ID>/packages/maven" />
<chain name="default">
<resolver ref="maven2" />
<resolver ref="gitlab" />
</chain>
</resolvers>
</ivysettings>
But still I keep getting the HTTP response status: 401 CLIENT ERROR: Unauthorized, as it seems it's not able to add the credentials correctly. The realm is being set automatically to GitLab Packages Registry.
Did anyone manage to achieve it?
After a few other trials & errors it worked with this setting:
<ivysettings>
<settings defaultResolver="default" />
<credentials
host="private.git.domain.com"
realm="GitLab Packages Registry"
username="read_only"
passwd="<TOKEN>"/>
<resolvers>
<ibiblio name="maven2" m2compatible="true" />
<ibiblio name="gitlab" m2compatible="true"
root="https://private.git.domain.com/api/v4/projects/<ID>/packages/maven" />
<chain name="default">
<resolver ref="maven2" />
<resolver ref="gitlab" />
</chain>
</resolvers>
</ivysettings>
The problem was that although the realm was set automatically, it was also used for retriving the credentials. So if the realm is missing in the credentials block, it couldn't match it the host (with realm added automatically).