Ta có \(ab\le\frac{\left(a+b\right)^2}{4}\le\frac{1}{4}\) \(\Rightarrow\frac{1}{ab}\ge4\)
\(A=\frac{1}{a^2b^2+\frac{1}{a^2}+\frac{1}{b^2}+1}\le\frac{1}{a^2b^2+\frac{2}{ab}+1}=\frac{1}{a^2b^2+\frac{1}{64ab}+\frac{1}{64ab}+\frac{63}{32ab}+1}\)
\(A\le\frac{1}{3\sqrt[3]{\frac{a^2b^2}{64^2a^2b^2}}+\frac{63}{32}.4+1}=\frac{16}{145}\)
\(A_{max}=\frac{16}{145}\) khi \(a=b=\frac{1}{2}\)