\(P=3x^2-y^2+4xy=3x^2-y^2+4xy+x^2+y^2=4x^2+4xy\)
\(\Rightarrow\frac{P}{4}=\frac{4x^2+4xy}{x^2+y^2}\)
- Với \(y=0\Rightarrow P=16\)
- Với \(y\ne0\Rightarrow\frac{P}{4}=\frac{4\left(\frac{x}{y}\right)^2+\frac{4x}{y}}{\left(\frac{x}{y}\right)^2+1}\)
Đặt \(t=\frac{x}{y}\Rightarrow\frac{P}{4}=\frac{4t^2+4t}{t^2+1}\Leftrightarrow P.t^2+P=16t^2+16t\)
\(\Leftrightarrow\left(P-16\right)t^2-16t+P=0\)
\(\Delta'=64-P\left(P-16\right)\ge0\)
\(\Leftrightarrow-P^2+16P+64\ge0\)
\(\Leftrightarrow8-8\sqrt{2}\le P\le8+8\sqrt{2}\)
\(\Rightarrow P_{max}=8+8\sqrt{2}\) khi \(t=\sqrt{2}+1\) hay \(x=\left(\sqrt{2}+1\right)y\)