If you did as you described current would pass to both batteries. Probably be fine with trickle charger, but any larger amperage 'smart charger' battery charger could those features defeated - it's see both batteries as one large battery. So if you have one fully topped off battery and another that is depleted, the fully topped off battery could get over-charged. Better to connect the single bank charger directly to the depleted battery and make sure it's isolated from the topped off battery. Even better would be to go with a dual bank charger (of sufficient amperage) that could monitor and manage the batteries individually - even then you would not want the battery switch set to all.