[SOLVED] Generating heatmap layer for milions of points

Generating heatmap layer for milions of points

I'm using heatmap layer from Google Maps to display a heatmap, however, I now have too many points and it stopped working, because browser cannot handle it anymore. I've found that they provide Fusion Tables, but they're also limited: to 100k rows, which is way too low. I need to render heatmap of milions or maybe even more points. I'd be perfect if my server can have some PHP script to render a heatmap, for example, once a day. And then a client from js will just download a preloaded heatmap (on the map like google maps, but may be different map too). Is this possible with some existing technology (can be commercial)?

Solution

All you need is to pre-clusterize your points into smaller number of points and pass these groups to Google Maps as if it were your original points. So, if you had 3 nearby points with values 3, 5 and 10, you create one point with value 18 at weighted average of their coordinates. Do it for all your dataset and you will reduce it 3 times. Change 3 to any appropriate value to further reduce your dataset.

The easy way to cluster your data is to use geohashing[1]. Here is nice geohashing library for PHP[2]. You may calculate a set of geohashes of different precision for each of your points only once when adding, and then use desired precision to reduce your dataset using simple GROUP BY. Example (meta):

use Lvht\GeoHash;

class DataPoint extends ActiveRecord {
    public geo_hash_precision4, geo_hash_precision5, geo_hash_precision6;
    public function save() {
        $this->geo_hash_precision4 = GeoHash::encode($this->lat,$this->lon, 0.0001);
        $this->geo_hash_precision5 = GeoHash::encode($this->lat,$this->lon, 0.00001);
        $this->geo_hash_precision6 = GeoHash::encode($this->lat,$this->lon, 0.000001);
        parent::save();
    }
}

class DataSet extends ActiveQuery {
    /**
     * @param int $p Desired precision
     */
    public function getValues($p = 6) {
        $data = $this->rawQuery("SELECT geo_hash_precision{$p}, SUM(value) FROM {$this->table} GROUP BY geo_hash_precision{$p}");
        // Add bounding box WHERE to reduce set for map size
        foreach ($data as $row) {
            list($minLon, $maxLon, $minLat, $maxLat) = GeoHash::decode($row["geo_hash_precision{$p}"]);
            $row['lat'] = ($maxLat - $minLat) / 2;
            $row['lon'] = ($maxLon - $minLon) / 2;
            unset($row["geo_hash_precision{$p}"]);
        }
    }
}