A fast (heavy and positively charged) alpha particle would scatter the electrons like chaff, but would itself be somewhat deflected by the heavy positive sphere. The deflection could be calculated, since the speed and mass of the alphas, and the size of the atom were known. The calculations suggested a small fraction of a degree. (The animation above greatly exaggerates this deflection -- the point here is to illustrate how much more dramatic the deflections are if the charge and mass are concentrated in a nucleus).
In the experiment, the alphas went through a thin sheet of gold, so might encounter at most 400 or so atoms, and might be deflected as much as two degrees (at most) if these were Thomson atoms.
All the details can be found in my lecture on Rutherford Scattering.
The experimenters found some deflections of 180 degrees! The model was completely wrong...the atom was a nuclear atom!