a) \(x^2y-4y+y\left(x-2\right)^2=y\left[x^2-4+\left(x-2\right)^2\right]\)
\(=y\left[\left(x-2\right)\left(x+2\right)+\left(x-2\right)^2\right]=y\left(x-2\right)\left(x+2+x-2\right)=2xy\left(x-2\right)\)
b) \(x^5-2x^4+x^3-x^3y^2=x^3\left(x^2-2x+1-y^2\right)\)
\(=x^3\left[\left(x-1\right)^2-y^2\right]=x^3\left(x-1-y\right)\left(x-1+y\right)\)