Finite $p$-groups with minimum number of central automorphisms fixing the center element-wise

We characterize finite $p$-groups $G$ of order up to $p^7$ for which the group of central automorphisms fixing the center element-wise is of minimum possibe order.