Couple years ago, I put my '04 180 Horizon in and it would not crank. I called the dealer on the cell phone and he suggested shaking the throttle... which worked. He later repaired a bad crimp on a safety wire wich locks out the starter when in gear.
Last year I forgot to put the plug in after launching.

A month and a half later, I was running into the same problem your describing. Zeph is right on, it turned out to be the currosion build up in the starter solenoid. I bought the solenoid only instead of the entier starter. New starter is in the $400 range, Solenoid is $40. Installed and works like a charm. Check out
http://www.marinepowerservice.com for parts like this.
Another thing, the starter is extremely hard to get to on my 4.3L. There is no way to quickly test by jumping the solenoid, especially on the water. Speedy, try hitting the starter/solenoid with a hammer or cresent wrench, or what ever you have on board. That got me going a couple times when mine wouldn't start.