High performance computing (HPC) on the cloud is an emerging approach that can potentially provide a significantly cheaper alternative to supercomputers. However, clouds are largely oriented towards multiprogramming workloads with no significant intercommunications. The placement of tightly coupled HPC virtual machines is thus not guaranteed to be physically affine, resulting in unpredictable communication times. This paper proposes a new cloud analytical model that describes the physical placement of virtual machines in the communication hierarchy. The model is constructed through a set of automated experiments that measure virtual machines point-to-point communication speed parameters; the parameters are then clustered, and the topology of the cloud network seen by the virtual machines is identified. As a case study, the paper applies the model to the Amazon Cloud; the obtained hierarchical model is used to select a fast communicating subset of instances and discarding the other instances. For a message-passing all-to-all communication operation such selection resulted in 4.1 to 5.5 speedup enhancement in performance when randomly executing on a similarly sized subset.