Not surprising at all - your experience is in line with a host of drivers who will disconnect their rear bar when looking for more grip out back (ex: driving in the rain).
Some tuners will even try to set up the rest of the car to not need a rear bar at all (e.g. Cortex prefers it this way). Of course, here lies the goal - to get the car's balance through all phases of the corner where you want it. Since this is hard to do, you typically see folks make a compromise in one phase for gains in another. A driver's preference and style do come into play when deciding which compromises are acceptable for which gains, but lap times tend to trump "style" if a given set up proves to deliver the goods.
This is why I think Whiskey's mantra should be tweaked a bit from "If you like it, rock it" to "If it works for you, rock it." Determining what genuinely works (and distinguishing what only feels good from what really works) takes time and testing, which is why they call it tuning.
BTW, I'm pretty sure Whiskey knows all that, I wasn't trying to correct him - just piggyback off of how he put it.
Also, the OP specifically asked for downsides, so also consider that (a) some of the ones Whiskey suggested will depend on the rest of the car's set up and (b) all things being equal, reducing the effective spring rate in the rear will lead to more body roll (which in itself isn't necessarily bad, but can induce geometry changes that weren't a concern before and be less confidence-inspiring to the driver). Since you're the driver, if you can make it work, great.
Best,
-j