Ta có: \(\left(x+2\right)\left(x^2+2x+2\right)-x\left(x-8\right)^2=\left(4x-3\right)\left(4x+3\right)\)
\(\Leftrightarrow x^3+2x^2+2x+2x^2+4x+4-x\left(x^2-16x+64\right)=16x^2-9\)
\(\Leftrightarrow x^3+4x^2+6x+4-x^3+16x^2-64x-16x^2+9=0\)
\(\Leftrightarrow4x^2-58x+13=0\)
\(\Leftrightarrow\left(2x\right)^2-2\cdot2x\cdot\frac{58}{4}+\frac{3364}{16}-\frac{789}{4}=0\)
\(\Leftrightarrow\left(2x-\frac{58}{4}\right)^2=\frac{789}{4}\)
\(\Leftrightarrow\left[{}\begin{matrix}2x-\frac{58}{4}=\frac{\sqrt{789}}{2}\\2x-\frac{58}{4}=-\frac{\sqrt{789}}{2}\end{matrix}\right.\Leftrightarrow\left[{}\begin{matrix}2x=\frac{\sqrt{789}}{2}+\frac{58}{4}=\frac{29+\sqrt{789}}{2}\\2x=-\frac{\sqrt{789}}{2}+\frac{58}{4}=\frac{29-\sqrt{789}}{2}\end{matrix}\right.\)
\(\Leftrightarrow\left[{}\begin{matrix}x=\frac{29+\sqrt{789}}{4}\\x=\frac{29-\sqrt{789}}{4}\end{matrix}\right.\)
Vậy: \(x\in\left\{\frac{29+\sqrt{789}}{4};\frac{29-\sqrt{789}}{4}\right\}\)