ĐK:\(a\ne0,b\ne0\)
Ta có \(\dfrac{5}{a+b\sqrt{2}}-\dfrac{4}{a-b\sqrt{2}}+18\sqrt{2}=3\Leftrightarrow\dfrac{5\left(a-b\sqrt{2}\right)}{\left(a+b\sqrt{2}\right)\left(a-b\sqrt{2}\right)}-\dfrac{4\left(a+b\sqrt{2}\right)}{\left(a+b\sqrt{2}\right)\left(a-b\sqrt{2}\right)}+18\sqrt{2}=3\Leftrightarrow\dfrac{5a-5b\sqrt{2}-4a-4b\sqrt{2}}{a^2-2b^2}+18\sqrt{2}=3\Leftrightarrow a-9b\sqrt{2}=\left(3-18\sqrt{2}\right)\left(a^2-2b^2\right)\Leftrightarrow a-9b\sqrt{2}=3a^2-6b^2-18a^2\sqrt{2}+36b^2\sqrt{2}\Leftrightarrow a-3a^2+6b^2=9b\sqrt{2}+36b^2\sqrt{2}-18a^2\sqrt{2}\Leftrightarrow a-3a^2+6b^2=9\sqrt{2}\left(b+4b^2-2a^2\right)\)Ta có a,b là số nguyên
Suy ra\(\left\{{}\begin{matrix}a-3a^2+6b^2=0\left(1\right)\\b+4b^2-2a^2=0\end{matrix}\right.\)\(\Leftrightarrow\)\(\left\{{}\begin{matrix}4a-12a^2+24b^2=0\left(2\right)\\6b+24b^2-12a^2=0\left(3\right)\end{matrix}\right.\)
Trừ (2) cho (3) ta được \(4a-6b=0\Leftrightarrow b=\dfrac{2}{3}a\left(4\right)\)
Thay (4) vào (1) ta có \(a-3a^2+6b^2=0\Leftrightarrow a-3a^2+\dfrac{6.4}{9}a^2=0\Leftrightarrow a-\dfrac{1}{3}a^2=0\Leftrightarrow a^2-3a=0\Leftrightarrow a\left(a-3\right)=0\Leftrightarrow\)\(\left\{{}\begin{matrix}a=0\\a=3\end{matrix}\right.\)\(\Leftrightarrow\)\(\left\{{}\begin{matrix}b=0\left(ktm\right)\\b=1\left(tm\right)\end{matrix}\right.\)
Vậy (a;b)=(3;1)