\(x^2-xy\left(a+b\right)+aby^2=x^2-xya-xyb+aby^2=x\left(x-ya\right)-yb\left(x-ya\right)=\left(x-ya\right)\left(x-yb\right)\)
\(x^2-xy\left(a+b\right)+aby^2\)
\(=x^2-axy-bxy+aby^2\)
\(=x\left(x-ay\right)-by\left(x-ay\right)\)
\(=\left(x-ay\right)\left(x-by\right)\)