More than likely, your starter bushing is worn out. It's a little brass bushing that is pressed into the bell housing of the transaxle that the end of the starter rides in. When it gets worn just a little, the starter will drag.
The bushing cost anywhere from 99 cents to two or three dollars depending on where you buy it. If it's your first one, they aren't easy to change but after you've changed a few, you can do it blindfolded.
There is a special tool to remove the bushing and I've seen several different home-made tools that work. But, for me, the easiest way is to thread a M12 tap into it. When you thread it all the way, the tap will bottom on the engine case . . . keep turning and it will walk the bushing right out.
Try to find a deep well socked that tapers down at the hex end. You want the diameter of the small end of the socket to be just a little smaller than the inside diameter of the bushing, and the big end to be a little bigger. Stick the bushing on the socket with wheel bearing grease (you don't want to drop down in the bell housing). Use a short extension on the socket if you have to . . . line the bushing up with the hole and tap it in. Stop when it's flush with the flywheel side of the bell housing. Put a little grease in the bushing and re-install the starter. If the starter has not damaged its self from dragging, it should speed up to normal.