dbscan
or hdbscan
algorithms for this clustering task.When I try to use one of those algorithms, I'm getting memory error.
If the number of features in your dataset is not too much (below 20-25), you can consider using BIRCH. It's an iterative method that can be used for large datasets. In each iteration it builds a tree with only a small sample of data and put each instance into clusters.