Easy Answer.

There has to be a simple answer to the question: did you grow up with brothers. There has to be... and yet there isn't.
If I say no.. then its a lie.
But if I say yes... they might ask me about him.. in which case what do I say. I'd have to tell the truth and that is not easy for me.
Yes. I had an older brother. But he died two years ago. That's the simple, honest truth and it is more information than people want. They don't want to know because most people don't know what to say.