So I spoke to my rov on tuesday, he said I should go to Lakewood and that I should sit down and talk to my family (tet-a-tet, not to all at the same time) and to clear up the confusion.
Spoke to my mom and she said that everything is ok, hope it is. Haven't seen my brother though, so he found news from mom. That's not the fortunate option, but I wouldn't discuss these things on the phone - I am not a phone person and everyone knows it. So I am still not sure what he thinks has made me to go to Lakewood. Based on the short phone conversation I had with him yesterday/today he's worried something is wrong but I am not saying that. That's my personality - I rarely discuss personal stuff with family and he knows it. He noticed that many times and he mentioned that too.
So now I feel myself guilty that I've left them without an option at the last moment. As of today I could've had my mom going to four seudos (thanks to Moshe), now she goes only to two because she won't go alone. My brother didn't sound so happy either. Plus I told him I'll cover expenses on stuff that was supposed to be brought to hosts, now he doesn't want me to do so because I don't go. I think like I'm getting too deep into details I shouldn't be..
So all in all everyone has to be happy and no one really is. Otherwise everything is cool. I looking forward to visit Lakewood after 3 years of hiatus and hopefully meet some people I know there.