More than one way to do this but this approach is pretty common:
1. Gently press the string down between the 2nd and 3rd fret (closer to the 2nd fret) until it just makes contact with the top of the 2nd fret.
2. Check for clearance over the 1st fret. It should only be a few thousandths of an inch (maybe a little more for the low strings). If it's higher than that, the slot is not deep enough. If it's touching the 1st fret then the slot is too deep and you need a new nut (and it's causing open strings to buzz.)
You want the depth of the nut slots to be just as if there was another fret down there (like a ZERO fret) with a capo behind it. And then with perhaps a tiny bit of extra height. If you had a REAL zero fret down there then a proper level/crown job would put it in the same plane as the 1st and 2nd frets. That's why this test works.