\(5x^2-4xy+y^2-6x+8=0\)
\(\Leftrightarrow25x^2-20xy+5y^2-30x+40=0\)
\(\Leftrightarrow\left(5x-2y\right)^2+\left(y-15\right)^2=185=64+121=8^2+11^2\)
\(\Leftrightarrow\left[{}\begin{matrix}\left[{}\begin{matrix}5x-2y=8\\y-15=11\end{matrix}\right.\\\left[{}\begin{matrix}5x-2y=11\\y-15=8\end{matrix}\right.\end{matrix}\right.\)\(\Leftrightarrow\left[{}\begin{matrix}\left[{}\begin{matrix}5x=2y+8\\y=26\end{matrix}\right.\\\left[{}\begin{matrix}5x=11+2y\\y=23\end{matrix}\right.\end{matrix}\right.\Leftrightarrow}\left[{}\begin{matrix}\left\{{}\begin{matrix}x=12\\y=26\end{matrix}\right.\left(thoaman\right)}\\\left\{{}\begin{matrix}x=11,4\\y=23\end{matrix}\right.\left(kothoaman\right)\end{matrix}\right.\)
Vậy S={26,12}