Two geometric phases can dramatically differ from each other even if their evolution paths are sufficiently close in a pointwise manner.

One milestone in quantum physics is Berry's seminal work [Proc.~R.~Soc.~Lond.~A \textbf{392}, 45 (1984)], in which a quantal phase factor known as geometric phase was discovered to solely depend on the evolution path in state space. Here, we unveil that even an infinitesimal deviation of the initial state from the eigenstate of the initial Hamiltonian can yield a significant change of the geometric phase accompanying an adiabatic evolution. This leads to the surprising observation that two geometric phases can dramatically differ from each other even if their evolution paths are sufficiently close in a pointwise manner.