I don't think you're being unreasonable.
However, I will play a little Devil's advocate here. You know they didn't tell you to come in. However, they may think they did. They obviously don't have their shit together. So while it's perfectly reasonable to decline work on short notice, they may believe they gave you full notice and you are flaking out. We know that's not the truth. But when you need the job, you sometimes have to be subject to things like this and go in, even when you're right and they're wrong.