\(c^2+d^2+25=6c+8d\)
\(\Leftrightarrow\left(c^2-6c+9\right)+\left(d^2-8d+16\right)=0\)
\(\Leftrightarrow\left(c-3\right)^2+\left(d-4\right)^2=0\)
\(\Leftrightarrow\left\{{}\begin{matrix}c-3=0\\d-4=0\end{matrix}\right.\) \(\Rightarrow\left\{{}\begin{matrix}c=3\\d=4\end{matrix}\right.\)
\(\Rightarrow P=25-3a-4b=25-\left(3a+4b\right)=25-Q\)
Xét \(Q=3a+4b\Rightarrow Q^2=\left(3a+4b\right)^2\le\left(3^2+4^2\right)\left(a^2+b^2\right)=25.2=50\)
\(\Rightarrow Q^2\le50\Rightarrow-5\sqrt{2}\le Q\le5\sqrt{2}\Rightarrow-Q\le5\sqrt{2}\)
\(\Rightarrow P\le25+5\sqrt{2}\)
\(P_{max}=25+5\sqrt{2}\) khi \(\left\{{}\begin{matrix}a^2+b^2=2\\\frac{a}{3}=\frac{b}{4}\\3a+4b=-5\sqrt{2}\end{matrix}\right.\) \(\Rightarrow\left\{{}\begin{matrix}a=-\frac{3\sqrt{2}}{5}\\b=-\frac{4\sqrt{2}}{5}\end{matrix}\right.\)