The quote comes from a letter to James Lloyd dated January 1815. This letter can be found in “The Life and Works of John Adams” Volume 10, page 108- edited by Charles Francis Adams, Little Brown and Company-1856.
It does seem clear enough from the entire letter that, as you suggested, he was referring to the American public split regarding the French Revolution. This was in the context of a discussion regarding his missions to France during his administration which did get a small but temporary boost during the time that the French Navy was preying upon our shipping.
I agree with you that Adams' memory of the political balance is kinder to himself than the reality seems to have been.