I don't mean this rudely at all but I think you've missed what my problem really was. I don't doubt a lot of your rehearsal jams are hit and miss and often forgettable but I'd be surprised if you guys didn't make much effort to feed into each other and adjust to what you were hearing from each other while these jams were happening. I couldn't imagine staying in a band with people who seemed like they weren't listening (I've been in bands with one or two people like this and thankfully it didn't last long). These guys really felt like that to me. Like they just weren't listening to what was happening. Even if it was my first time sitting in with another musician I'd be doing my best to listen and make my best effort to complement the feel. The end result might be that you'd hear me changing up a lot and stopping a lot if I couldn't make it work but at least you'd know I was trying and that was another thing, they both played constantly, leaving no room for space so I tried to make up for it by stopping in a couple of places.My band starts every rehearsal with a jam. It’s always hit or miss, oftentimes forgettable, occasionally awesome. And we’re all on the same wavelength musically.
Expecting a random collection of musicians to gel around one person’s vision is expecting a lot, especially on the first attempt. I’d say don’t give up on the open mic if people were receptive, it could end up sounding great.
I've seen something similar even with some people who were really good musicians otherwise where they'd join a band and then just do their own thing over the top of the music without making much of an effort to actually get the music in question if you know what I mean. I dunno, I just feel like these are basic listening skills that are also really essential and I hope that I'm not guilty of violating them myself without having the self-awareness to realise it.