Phương trình hoành độ giao điểm:
\(x^2-2\left(m+1\right)x+m^2+2=0\)
\(\Delta'=\left(m+1\right)^2-\left(m^2+2\right)=2m-1\ge0\Rightarrow m\ge\dfrac{1}{2}\)
Theo hệ thức Viet: \(\left\{{}\begin{matrix}x_1+x_2=2\left(m+1\right)\\x_1x_2=m^2+2\end{matrix}\right.\)
\(A=\sqrt{2\left(x_1+x_2\right)^2-4x_1x_2+16}-3x_1x_2\)
\(A=\sqrt{8\left(m+1\right)^2-4\left(m^2+2\right)+16}-3\left(m^2+2\right)\)
\(A=\sqrt{4m^2+16m+16}-3\left(m^2+2\right)\)
\(A=2m+4-3\left(m^2+2\right)\)
\(A=-3m^2+2m-2=-3m^2+2m-\dfrac{1}{4}-\dfrac{7}{4}\)
\(A=\left(\dfrac{1}{2}-m\right)\left(3m+\dfrac{1}{2}\right)-\dfrac{7}{4}\le-\dfrac{7}{4}\)
\(A_{max}=-\dfrac{7}{4}\) khi \(m=\dfrac{1}{2}\)