Hệ tương đương: \(\left\{{}\begin{matrix}x+y-\sqrt{xy}=3\\x+1+y+1+2\sqrt{\left(x+1\right)\left(y+1\right)}=16\end{matrix}\right.\)
\(\Leftrightarrow\left\{{}\begin{matrix}x+y-3=\sqrt{xy}\\x+y+2\sqrt{x+y+xy+1}=14\end{matrix}\right.\)
Đặt \(\left\{{}\begin{matrix}x+y=a\\xy=b\ge0\end{matrix}\right.\)
\(\Rightarrow\left\{{}\begin{matrix}a-3=\sqrt{b}\\2\sqrt{a+b+1}=14-a\end{matrix}\right.\) (với \(3\le a\le14\))
\(\Leftrightarrow\left\{{}\begin{matrix}a^2-6a+9=b\\4\left(a+b+1\right)=a^2-28a+196\end{matrix}\right.\)
\(\Rightarrow4\left(a+a^2-6a+9+1\right)=a^2-28a+196\)
\(\Leftrightarrow3a^2+8a-156=0\Rightarrow\left[{}\begin{matrix}a=6\Rightarrow b=9\\a=-\dfrac{26}{3}< 0\left(loại\right)\end{matrix}\right.\)
\(\Rightarrow\left\{{}\begin{matrix}x+y=6\\xy=9\end{matrix}\right.\) \(\Rightarrow x=y=3\)