Edge-colorings avoiding rainbow and monochromatic subgraphs

For two graphs G and H, let the mixed anti-Ramsey numbers, maxR(n;G,H), (minR(n;G,H)) be the maximum (minimum) number of colors used in an edge-coloring of a complete graph with n vertices having no monochromatic subgraph isomorphic to G and no totally multicolored (rainbow) subgraph isomorphic to H. These two numbers generalize the classical anti-Ramsey and Ramsey numbers, respectively. We show that maxR(n;G,H), in most cases, can be expressed in terms of vertex arboricity of H and it does not depend on the graph G. In particular, we determine maxR(n;G,H) asymptotically for all graphs G and H, where G is not a star and H has vertex arboricity at least 3. In studying minR(n;G,H) we primarily concentrate on the case when G=H=K"3. We find minR(n;K"3,K"3) exactly, as well as all extremal colorings. Among others, by investigating minR(n;K"t,K"3), we show that if an edge-coloring of K"n in k colors has no monochromatic K"t and no rainbow triangle, then n=<2^k^t^^^2.