Was this unreasonable of me?

No, chicken pox are horrible and very contagious to people who haven't had them before. They hurt like hell. You were right not to go round there.