So, in short, does number of physical partitions always go up only or can it go down? (e.g. when a lot of data gets deleted and provisioned RUs lowered)
If it can go down, how&when that happens?
Cosmos DB scales capacity via additional physical partitions. As storage capacity needs grow, or RU/sec needs grow, a physical partition may be split into multiple physical partitions (with logical partitions then distributed across the physical partitions, keeping each logical partition within a single physical partition).
Once these new physical partitions are created, that is the new minimum baseline capacity for a particular container (or set of containers, if using shared resources). Logical partitions may come and go, but physical partitions only scale out: they may split, but they cannot be merged later.
The only way to shrink the number of physical partitions today: migrate data to a new collection. During migration though, just remember to keep the destination collection's RU/sec low enough to not cause a partition-split in that collection.
Note: Partition-merge is now available in Preview, with some limitations.