Đặt: \(f\left(x\right)=\left(m^2-3m-4\right)x^2-2\left(m-4\right)x+3\).
Khi \(\left[{}\begin{matrix}m=-1\\m=4\end{matrix}\right.\) thì \(\left[{}\begin{matrix}f\left(x\right)=10x+3\\f\left(x\right)=-12x+3\end{matrix}\right.\). Dễ thấy \(f\left(x\right)< 0\) luôn có nghiệm.
Khi \(m\notin\left\{-1;4\right\}\)
Để \(f\left(x\right)< 0\) vô nghiệm thì \(f\left(x\right)\ge0\forall x\in R\)
Khi đó: \(\left\{{}\begin{matrix}m^2-3m-4\ge0\\\Delta'=\left[-\left(m-4\right)\right]^2-\left(m^2-3m-4\right)\cdot3< 0\end{matrix}\right.\)
\(\Leftrightarrow\left\{{}\begin{matrix}\left[{}\begin{matrix}m\le-1\\m\ge4\end{matrix}\right.\\-2m^2+m+28< 0\end{matrix}\right.\)
\(\Leftrightarrow\left\{{}\begin{matrix}\left[{}\begin{matrix}m\le-1\\m\ge4\end{matrix}\right.\\\left[{}\begin{matrix}m< -\dfrac{7}{2}\\m>4\end{matrix}\right.\end{matrix}\right.\Rightarrow\left[{}\begin{matrix}m< -\dfrac{7}{2}\\m>4\end{matrix}\right.\)
Vậy: \(f\left(x\right)< 0\) vô nghiệm khi \(m\in\left(-\infty;-\dfrac{7}{2}\right)\cup\left(4;+\infty\right)\)