javahadoopthread-safetydistributed-filesystem

is hadoop DistributedFileSystem thread safe?

I am using hadoop for writing data I scrape.

I have a spring service that is called from multiple threads to write some content to the HDFS.

@Service
public class WriteService
{
    public void write(String path, String content)
    {
        FileSystem fs = FileSystem.get(conf);
    }
}

I am not sure whether the FileSystem object can be a member of the WriteService and I don't find whether it is thread safe or not. I am using the DistributedFileSystem object.

Do you know if it is thread-safe and I can use it as a member to my service?

Thank you

Solution

  • Hadoop DFS uses a so-called WORM-Model. This makes it more robust when it comes to concurrency issues.

    But, to answer the question, it is not safe in general. You still need to think about concurrency control requirements.