galaxy - How can ionized emission line flux decrease as a function of increasing metallicity or abundance?

Metallicity and abundance

Metallicity

Without specifying a given metal, the term "metallicity" — abbreviated $Z$ — usually refers to the total metallicity of all elements, i.e. the mass fraction of all metals to the total mass of some ensemble of elements, e.g. a star, a cloud of gas, a galaxy, etc. (as usual, the term "metal" refers to all elements that are not hydrogen or helium). For instance, the mass of all metals in the Sun, divided by the Sun's mass, is 0.02:
$$
Z_odot equiv frac{M_mathrm{C} + M_mathrm{N} + M_mathrm{O} + ldots}{M_odot} = 0.02.
$$

Sometimes we talks about the metallicity of a given element, e.g. oxygen. The mass fraction of oxygen in the Sun is 0.005 (i.e. oxygen comprises 1/4 of all metals by mass), so we say $Z_mathrm{O} = 0.005$.

Unfortunately it is not uncommon to implicitly talk about the metallicity of an object, divided by Solar metallicity, such that a galaxy which has one-tenth of Sun's metallicity is said to have $Z=0.1$, rather than $Z=0.002$.

Abundance

The term "abundance" is only used for a single element. It basically expresses the same thing as metallicity, and is often used interchangeably, but is expressed in terms of the number $N$ of element nuclei, and as the ratio not to all nuclei but to hydrogen nuclei. For wacky historical reasons, we also take the logarithm and add a factor of 12. Taking again oxygen as an example, the mass fraction of 0.005 corresponds to a nuclei fraction of roughly $5times10^{-4}$, so we say that the abundance of oxygen is (e.g. Grevesse (2009))
$$
A(mathrm{O}) equiv log left( frac{N_mathrm{O}}{N_mathrm{H}} right) + 12 = 8.7.
$$

Metallicity of a given species vs. total metallicity

In general, the ratio of a given element to all metals is roughly constant. That is, various elements are produced by stars approximately by the same amount. But various processes may cause elements to exist in various forms. For instance, metals deplete to dust, but some elements tend not to form dust, e.g. Zn. For this reason, Zn is a better proxy of the total metallicity than, e.g. Mg, since half of the Mg may be locked up in dust.

Metals increases cooling

Elements also appear in various excitation states, which depend on various processes. The lines you mention, [O II] and [O III], arise from collisionally ionized oxygen, which subsequently recombines (in my first answer I wrote, wrongly, that it was excited), and thus depend on the temperature of the gas. As the metallicity of the gas in a galaxy increases, the ratio of the intensity of these lines to that of hydrogen lines (e.g. H$beta$) first increases, as expected. However, the increased metallicity also allows the gas to cool more efficiently. The reason is that metals have many levels through which electron can
"cascade" down.
If the electron recombines to the level where it was before, a photon of the same energy will be emitted, which itself may radiatively ionize another atom. But the many levels in metals makes de-excitation to intermediate level more probable, such that the electron cascades down, emitting several low-energy (infrared) photons, which are incapable of ionizing atoms and thus escape. The result is that energy leaves the system, i.e. the system is cooled.

This in turn means that, above a certain metallicity threshold — which is specific to a given species — the abundance of the collisionally excited lines begin to decrease. The following figure is taken from Stasińska (2002), and shows the turnover for the two oxygen lines:

This means that measuring the metallicity of a single species in general gives two solutions for the total metallicity. Luckily, as the turnover is different for different elements, measuring the metallicity for several species can constrain the total metallicity.