In my mind, "f(x)" is shorthand for "f is a function whose inputs can be considered as a single object (probably a real number), which we will represent with x for now". That way, when I write "f(x) = O(x^2)", then I can read that as "the function f, when you look at large values of its input x, behaves similarly to the function that maps x to its square".
It may be true that mathematics is formalised with sets, but it is written in shortcuts and abbreviations and notations that are designed to communicate an idea, and much of the point of formalism is to ensure that the ideas behind those shortcuts are valid without needing to always explicitly state everything. It's the same shorthand that means that x and y are usually real numbers, z is typically complex if not combined with other pronumerals, i,j,k,m,n are integers, and most lower case Greek letters are probably representing angles.
With that understanding, I can write something like f(x^2) = O(ln(g(x)) and you can understand the general idea of what I'm trying to communicate, but without it I would need to do something like:
Let F: R -> R be a function such that F(x) = f(x^2), and let G: R -> R be a function such that G(x) = ln(g(x)). Then F = O(G).
And sure that's more accurate, but it doesn't actually convey any information that isn't at least implied in the original statement.