Unusual Gems, if stolen...

[storminmay]

I think I'm finally going to tackle the No Stone Unturned quest, but I have a question: if I steal one of the Unusual Gems, then get caught by a hold guard and agree to pay off my bounty, what happens to the gem I stole? Does it stay in my inventory because it's a quest item, or does it go into the evidence chest? I've seen conflicting info online and thought I'd ask.

[sinister]

It should stay.

[storminmay]

It should stay.

Thank you so much!