Going to a Bet Hakevarot

Yes your friend is correct that visiting a Beit Hakevarot without reason. However, when you go vis your grandfather’s Kever, even when it isn’t the day of his Yahrzeit you give his Neshama pleasure, and you fulfill the mitzvah of honoring your grandfather. That mitzvah will protect you from any damage to your Neshama.

The Magen Avraham 559:15 quotes the Ariza”l: Do not go to cemeteries except for a funeral. Especially those who have not fixed up their sin of Keri since the bad forces will attach themselves to him.
Similarly, the Pitchei Teshuva in Yoreh Deah 19519 quotes the Sefer Chamudei Daniel that during Niddot woman should not go to the Beit Hakevarot because it it’s a time of Tumaah. Therefore, it would seem from his understanding men and women who are pure-Tahor should have no problem gong to a Bet Hakevarot. HaRav Ovadia Yosef Z”L in Yabia Omer (4:35:5) understood Chamudei Daniel to include all women even not during Zman Niddah.
So ruled R’ Moshe Feinstein that one should not move a Kever of a loved one to a more local cemetery . one of the considerations he mentions is this Arizal not to go to cemetery without a strong reason, i.e. a funeral. Since moving a Kever is not essential for the Met rather for his descendant it would not be a strong enough reason to permit going into the Bet Hakevarot and the Arizal was very stringent int his matter.
The Gemara in Taanit however seem to contradict the Arizal who says only to go in for a funeral. The Gemara in Taanit daf 16a that during day of taanit it was customary to go to the Kvarot and to pray. R’ Levi bar Chama and R’ Chanina argued, one said to send a message to Hashem that we are like the dead in front of Him. The other said so the ones wo have passed on may ask for mercy for us. According to the first opinion one could go to Non-Jewish Kevarim as well. The other reason is that it is to awaken feelings of Mercy for us , in which case it would need to be Jewish Kevarim. The Shulchan Aruch rules 579:3 During these seven taaniyot after tefillas the nation should go to the Bet HaKevarot and cry and plead there meaning you will be like these dead people if we do not do Teshuva. The Rema adds therefore if a Jewish cemetery is not available go to Non-Jewish one. So, it would seem the Shulchan Aruch follows the opinion that we can go to a Bet Hakevarot not just for a Levaya but for any great need.
The Minchat Elazar Teshuvot 1:68 quotes the Siddur of the Ari: “Do not go to pray by Kevarim often since the sprits of people are here and they might cause harm. “ he explains that It seems the Ari was only particular if someone would go ‘often’ but occasionally for a Mitzvah would be fine. It says in Sefer Chasidim and in Reishit Chacham that the Meitim receive much pleasure when their relatives visit their Kevarim. This is definitely part of the mitzvah of Kibud Av V’eim, so a person is definitely permitted to go the Kevarim of his fathers. This is because the mitzvah will protect him from any possible harm which can come from the spirits there. He also continues that it would seem to him the only ones who can get damaged in the Bet Hakvarot are those who are saying certain types of Kabbalistic prayers there but regular tefillot should cause no problem.
In conclusion a person should not go to the Bet Hakvarot without any reason. R’ Moshe Feinstein held that even if one has a good reason, if the reason has no benefit for the Meit then it I also forbidden. If one goes to the Kever to daven or even just visit it gives pleasure to the Niftar so that is fine. Going for any mitzvah is permitted to go like for a Levaya or for Hakamat Matzeivah or for a Yahrzeit . If you are concerned go to the Mikvah before you go to add extra Taharah.