k-mean clustering - inertia only gets larger
I am trying to use the KMeans clustering from faiss on a human pose dataset of body joints. I have 16 body parts so a dimension of 32. The joints are scaled in a range between 0 and 1. My dataset consists of ~ 900.000 instances. As mentioned by faiss (faiss_FAQ):
As a rule of thumb there is no consistent improvement of the k-means quantizer beyond 20 iterations and 1000 * k training points
Applying this to my problem I randomly select 50000 instances for training. As I want to check for a number of clusters k between 1 and 30.
Now to my "problem":
The inertia is increasing directly as the number of cluster increases (n_cluster on the x-axis):
I tried varying the number of iterations, the number of redos, verbose and spherical, but the results stay the same or get worse. I do not think that it is a problem of my implementation; I tested it on a small example with 2D data and very clear clusters and it worked.
Is it that the data is just bad clustered or is there another problem/mistake I have missed? Maybe the scaling of the values between 0 and 1? Should I try another approach?
I found my mistake. I had to increase the parameter max_points_per_centroid. As I have so many data points it sampled a sub-batch for the fit. For a larger number of clusters this sub-batch is larger. See FAQ of faiss:
max_points_per_centroid * k: there are too many points, making k-means unnecessarily slow. Then the training set is sampled
The larger subbatch of course has a larger inertia as there are more points in total.
- Problems with creating a mathematical clustering model with an additive criterion in CPLEX OPL Studio
- How do I reduce a specific colour range to a single colour?
- How to use Machine Learning to find the pattern customer profile?
- matplotlib detect and isolate in circles different groups of points
- How does use MASK R_CNN to extract exact region in image by OpenCV?
- ModuleNotFoundError installing yellowbrick in Python
- K-means clustering and nsamples for KernelExplainer
- Dunn index and inertia in kmeans algorithm
- Kmean get good group separation
- 3d Plot Python Tuple Index out of Range
- Clustering using Python
- How reliable is the Elbow curve in finding K in K-Means?
- Using SKLearn KMeans With Externally Generated Correlation Matrix
- silhouette calculation in R for a large data
- What is the use of predict() method in kmeans implementation of scikit learn?
- Algorithm to find k optimal representatives for subsets of a set with arbitrary cost function
- Error in do_one(nmeth) : NA/NaN/Inf in foreign function call (arg 1)
- Different color in the same price plot
- K-Means clustering in R error
- How to extract the specific colors from an image
- KMeans evaluation metric not converging. Is this normal behavior or no?
- How to use a color palette created with pyplot.Spectral for scatterplots in a loop?
- TypeError: len() of unsized object in pyclustering library
- KMeans Clustering rows in a DataFrame with many columns (integers)
- Is there a way to use found sequential patterns as input for a clustering algorithm
- k-means core function for temporal geo data
- How to save cluster assignments in output file using Weka clustering XMeans?
- How can I persist an Oracle Machine Learning (OML) k-means model?
- Sklearn kmeans with multiprocessing
- Unsupervised learning using TSNE and Kmeans