In my opinion, The catch chance should at a minimum be 75-25 at a very common creature like a chicken and when you get to Evoker and Guardians and such, it should be about 5-95. All mobs (Excluding bosses, of course) should be catchable.
For cost, I think the currency for using this shouldn't be redstone, glowstone, or anything valuable (In a way)/hard to get, but should be mined materials like iron. If not, I believe it should use Sugar, like 5 for common and 30< for Evokers.
As for EXP bottles...
I believe it should hold a maximum (Keyword: Maximum) That at level 0, if you were to use it you would get to halfway through 7. Filling bottles with EXP for storage should cost 2-5 Redstone/Glowstone, and if not that then $12-$30 to fill.