\(ĐK:x\ge\dfrac{1}{5};y\ge\dfrac{3}{8}\)
\(PT\left(1\right)\Leftrightarrow\dfrac{3x^2-3y^2}{\sqrt{5x^2+2xy+2y^2}-\sqrt{2x^2+2xy+5y^2}}=3\left(x+y\right)\\ \Leftrightarrow3\left(x+y\right)\left(\dfrac{x-y}{\sqrt{5x^2+2xy+2y^2}-\sqrt{2x^2+2xy+5y^2}}-1\right)=0\\ \Leftrightarrow\left[{}\begin{matrix}x+y=0\\\dfrac{x-y}{\sqrt{5x^2+2xy+2y^2}-\sqrt{2x^2+2xy+5y^2}}=1\left(1\right)\end{matrix}\right.\)
\(\left(1\right)\Leftrightarrow x-y=\sqrt{5x^2+2xy+2y^2}-\sqrt{2x^2+2xy+5y^2}\\ \Leftrightarrow\left(x-y\right)=\dfrac{3\left(x^2-y^2\right)}{\sqrt{5x^2+2xy+2y^2}+\sqrt{2x^2+2xy+5y^2}}\\ \Leftrightarrow\left(x-y\right)\left[\dfrac{3\left(x+y\right)}{\sqrt{5x^2+2xy+2y^2}+\sqrt{2x^2+2xy+5y^2}}-1\right]=0\)
\(\Leftrightarrow x=y\)
Với \(x+y=0\Leftrightarrow x=-y\), thay vào PT 2
\(\Leftrightarrow3\left(-y\right)\left(y-7\right)+10=\sqrt{10\left(-y\right)-2}+2\sqrt{8y-3}\\ \Leftrightarrow3y\left(7-y\right)+10=\sqrt{-10y-2}+2\sqrt{8y-3}\)
ĐK: \(\left\{{}\begin{matrix}-10y-2\ge0\\8y-3\ge0\end{matrix}\right.\Leftrightarrow y\in\varnothing\)
Với \(x-y=0\Leftrightarrow x=y\), thay vào PT 2
\(\Leftrightarrow3x^2-21x+10=\sqrt{10x-2}+2\sqrt{8x-3}\left(x\ge\dfrac{3}{8}\right)\\ \Leftrightarrow3x^2-24x+9=\sqrt{10x-2}-\left(x+1\right)+2\sqrt{8x-3}-2x\)
\(\Leftrightarrow3\left(x^2-8x+3\right)=\dfrac{-x^2+8x-3}{\sqrt{10x-2}+\left(x+1\right)}+\dfrac{2\left(-x^2+8x-3\right)}{\sqrt{8x-3}+x}\\ \Leftrightarrow\left(x^2-8x+3\right)\left(3+\dfrac{1}{\sqrt{10x-2}+x+1}+\dfrac{2}{\sqrt{8x-3}+x}\right)=0\)
Dễ thấy ngoặc lớn vô nghiệm với \(x\ge\dfrac{3}{8}>0\)
\(\Leftrightarrow x^2-8x+3=0\\ \Leftrightarrow\left[{}\begin{matrix}x=4+\sqrt{13}\left(n\right)\\x=4-\sqrt{13}\left(n\right)\end{matrix}\right.\Leftrightarrow\left[{}\begin{matrix}y=4+\sqrt{13}\\y=4-\sqrt{13}\end{matrix}\right.\)
Vậy HPT có nghiệm \(\left(x;y\right)\in\left\{\left(4+\sqrt{13};4+\sqrt{13}\right);\left(4-\sqrt{13};4-\sqrt{13}\right)\right\}\)