ArrayList of HashSets iterating over indexes I don't specify?

isPrime() checks if a number is prime, and getPrimes(int upper) gets all primes up to and including upper. I want sievePrimeFactorSets to make a HashSet of all the prime factors (no repeats) of each number, and to store that HashSet at the given value, e.g. the HashSet at primeFactors.get(20) = [2,5].

Right now it adds every prime to every value, so primeFactors.get(20) = [2,3,5,7,11,13,etc]. Why is this happening?

public ArrayList<HashSet<Integer>> sievePrimeFactorSets(int upper)
{
    ArrayList<HashSet<Integer>> primeFactors = new ArrayList<HashSet<Integer>>();
    HashSet<Integer> empty = new HashSet<Integer>();
    for (int i = 0; i <= upper; i++)
    {
        primeFactors.add(empty);
    }
    ArrayList<Integer> primes = getPrimes(upper);
    for (Integer p : primes)
    {
        for (int j = p; j <= upper; j+=p)
        {
            primeFactors.get(j).add(p);
        }
    }
    return primeFactors;
}

public ArrayList<Integer> getPrimes (int upper)
{
    ArrayList<Integer> primes = new ArrayList<Integer>();
    primes.add(2);
    for (int i = 3; i <= upper; i++)
    {
        if (isPrime(i))
        {
            primes.add(i);
        }
    }
    return primes;
}

This line:

primeFactors.add(empty);

adds the same empty hash set to each element of the array. So every element shares the same hash set and the changes you think you are making to one are actually made to all elements.

Just replace with:

primeFactors.add(new HashSet<>());