Here's something I do that may help you. Or at least, I do for other things. I still have never reffed a game but I still think it may go a long way. Simply say, out loud (but quietly so you don't look like an idiot), the color and number when your arm goes up. It may not seem like much, but its AMAZING how well you can remember something when a million things are going on if you've actually said it out loud. Small detail, but its a simple enough trick that I've found effective in general.