I am trying to train GloVe embeddings. In the GloVe implementation from stanfordnlp there are 4 scripts to run. However, running the second script, coocur, results in an Illegal Hardware Instruction-Error. I don't understand how this error is produced.
With the input file 3.txt my commands look like this:
$ ./vocab_count -min-count 1 -verbose 2 < 3.txt > vocab.txt
BUILDING VOCABULARY
Processed 8354 tokens.
Counted 3367 unique words.
Using vocabulary of size 3367.
$ ./cooccur -memory 4.0 -vocab-file vocab.txt -verbose 2 -window-size 15 < 3.txt > cooccurrence.bin
zsh: illegal hardware instruction ./cooccur -memory 4.0 -vocab-file vocab.txt -verbose 2 -window-size 15 < 3.tx
I am running these commands on a remote server (Debian GNU/Linux 9 (stretch)). When I run the same commands on the same data locally (18.04.2 LTS (Bionic Beaver)), there is no problem. What could be the cause of this?
I've hit the same issue in recent days.
The Docker image was built on a server using Jenkins. It has been running fine until the underlying cluster host orchestration software and physical hardware was upgraded.
My solution has been to remove the build of GloVe from the Dockerfile and instead put the build/make inside a script which runs when the container starts.
The actual cause of the error may be caused by the CFLAGS: -march=native set in the Glove Makefile: https://github.com/stanfordnlp/GloVe/blob/07d59d5e6584e27ec758080bba8b51fce30f69d8/Makefile#L4 This will cause the GloVe build to rely on the underlying CPU instruction set on which the Docker image is built.
There's a discussion of this further here: mtune and march when compiling in a docker image