Yes, you can absolutely reduce crew to zero points.
It’s not silly all, and in fact was the direct source of one of the biggest rules disputes back in the very early days of the game. The “infinite oarsman trick”, primarily used with Captain Blackheart, resulted in the controversial (at the time) ruling that point reducers didn’t apply to your build total. That along with repeated reminders that the no-stacking rule applied to the oarsman ability just like any other ability shut down that massive game-breaker before it could become a widespread problem. (As much as I’d like to take credit for the ruling, it was before my time as the RA. Might have even been before Tony/Piranha.)