Finally launched the goddamn user study that I've been struggling with for about two weeks. Here's hoping that it doesn't go terribly wrong somehow. I'm paying about 400$ in compensation to do this.
It did go terribly wrong after all, but in a way that was hard to spot.
Well, terribly wrong as in there was a programming bug that skrewed with the results. It probably only screwed with them a little, but it means the numbers are wrong and I can't use them.
This was a massive failure on my part. I noticed something was weird right before I kicked off the study, but I was under time pressure and wanted to get it done soon so I rationalized it away. Investigating that weird thing would have led me to the bug. I know because I investigated it right and it led me to the bug.
"noticing confusion" as it is generally called is a rationalist virtue, and I thought I had gotten a lot better at that. Actually I still think I did -- I remember that I did, in fact, notice the weird thing and that my brain told me t hat it could be a bug. I think I would have investigated if I had had more time. Certainly, while working on the main program, there have been many instances like this, and I think I did quite well.
So the failure isn't really in the noticing part, it's in the 'ignore it anyway because the would be very annoying to search for a bug right now' part. A different kind of failure, but no less bad.
Anyway, I hereby award myself minus 80 points.