I want to make a melee weapon that, if you score a critical hit, casts banishment on the target. I don't mind it being of a high rarity, but I don't want it to be broken. I wasn't planning on making this effect limited to once per day or even limited by charges (although if it turns out that this weapon is broken without it, I will consider charges). The spell save DC of 15 is also arbitrary at the moment, based on a very rare weapon.

Banisher

Weapon (any melee weapon), ??? (requires attunement)

When you roll a 20 on an attack roll made with this weapon, you cast banishment on the target you hit if the target is a creature (save DC 15). The weapon maintains concentration on the spell, which can only be interrupted if you roll another 20 on an attack roll made with this weapon before the spell's duration has elapsed.

(In other words, with respect to that last sentence in my magic item's description, if you crit twice within a minute, then the first target reappears again, i.e. isn't banished)

It's only a 4th level spell, which you cannot cast by choice but rather is cast on random chance, so I'm not sure what the rarity of this effect will be. If it's super-legendary or something, I can introduce charges to limit it and bring the rarity back down to something balanced, but if it's actually not as bad as I'm thinking, then I might make it a +1 or +2 weapon as well to bump it up to very rare (which is roughly what I'm aiming for).

So, what is the rarity of a melee weapon that casts banishment every time you crit?

It is rarity "nobody will want to use it"

Let's be honest, it doesn't really matter if a weapon is Common, Uncommon or Very Rare, as those are simply terms the DM can use to determine when a character should get an item, and it sounds like this is an item you'd really like your players to have. What really matters is how desireable an item is, and this thing is not desireable at all.

At first glance it might seem like an awesome weapon for somebody to use, but that novelty will quickly wear off the first time the fighter accidentally banishes the big bad into another dimension for a minute.

Because you have no control over when it happens, this weapon is going to lead to a lot of frustration when it happens at the wrong time and nobody can attack the thing they were fighting for a full minute, causing concentration buffs to run out and making concentration spells that were already affecting the creature end because it's suddenly on another plane of existence.

It will throw carefully planned attacks into complete disarray and the effects are entirely too random to be of any help. You can't even use it to make a quick escape if things go wrong, because you would have to be able to roll a 20 on demand somehow, and if you can crit on demand, you might as well just kill whatever it is you were fighting by critting a few times.

Want to make it attractive?

If you want players to actually use it, giving them the option of turning this effect on or off would go a long way at making it more attractive. It's still going to be extremely random, but at the very least you won't accidentally ruin your party's plans.

You'll likely also want to make it a +1 weapon, at which point it could be a decent enough Rare weapon for a fighter, and the effect might prove to be useful once every dozen or so sessions.

The effect is random and unpredictable and could easily disrupt your combat. The advantage of Banishment is that it goes off when you want it to, and it only lasts as long as you want it to.

Your weapon lacks both of those features. You'll end up with a lot of fights where you banish a dead foe (especially if you do crit damage first). The next most common effect is that you'll banish some medium-high-HP foe, have to do the fight, wait a minute, then restart the fight against the last foe.

Against single hard foes, Banishment (if it lands) will feel really weird, where a fight pauses for a minute, then restarts. It could easily suck as much as it benefits, as minute-duration buffs "time out", but you get a chance to rebuff/heal up using magic.

I'll suggest something slightly different.

Blood Soaked Banisher

This weapon can have up to 2 charges. It regain charges at a rate of 1 per minute, and regains 1 when it reduces a foe to 0 HP or 2 if you crit a foe.

Whenever this weapon deals reduces a foe to below half maximum HP, or deals damage to a foe with less than half maximum HP, you can expend 2 charges to cast the Banishment spell on the foe (save DC 15). You can decide if you want to try to Banish the foe after you know if the attack reduced the target to 0 HP.

The concentration of the Banishment spell is maintained by the weapon itself. It ends its concentration if you cease wielding the weapon (including drop, sheath, or otherwise), if the weapon is destroyed, if you clean the blood (or other viscera) of the target off the blade as a bonus action, or if you attack another foe with the weapon.

If you are banished while wielding this weapon, you can expend 1 charge to cut a hole back into reality, ending the banishment effect, at the start of your turn. You can do this despite being incapacitated while being banished.

I used charges to prevent it from being spammed; the weapon is usually charged, however. You get to use it once per combat, unless there are some weak foes to recharge it or you get lucky with crits.

The "cut a hole back" is flavour.