On the number of limit cycles in small perturbations of a class of hyper-elliptic Hamiltonian systems with one nilpotent saddle

Abstract In this paper, we make a complete study on small perturbations of Hamiltonian vector field with a hyper-elliptic Hamiltonian of degree five, which is a Lienard system of the form x ′ = y , y ′ = Q 1 ( x ) + e y Q 2 ( x ) with Q 1 and Q 2 polynomials of degree respectively 4 and 3. It is shown that this system can undergo degenerated Hopf bifurcation and Poincare bifurcation, which emerges at most three limit cycles in the plane for sufficiently small positive e . And the limit cycles can encompass only an equilibrium inside, i.e. the configuration ( 3 , 0 ) of limit cycles can appear for some values of parameters, where ( 3 , 0 ) stands for three limit cycles surrounding an equilibrium and no limit cycles surrounding two equilibria.