Every gear in your transmission is always spinning and will grind if you attempt to shift when that next gear up or down is spinning at a different speed that the gear you are leaving. There is a friction surface on your syncro rings similar to your clutch. That friction surface is designed to stop that next gear so you can shift into it cleanly. If that surface has worn it won't shift cleanly anymore. That surface can wear down quicker than normal through aggressive driving and abuse, other systems not working correctly like the clutch system, or even if the transmission fluid has detergents in it that can break down that friction surface on your synchro rings. Royal Purple was known for this. Check the fluid you are using for detergents.
Without knowing anything other than what you've described above, it sounds like your synchro rings are shot and if you're going to open up your transmission and do surgery, you might as well replace all rings and bearings and seals. And when the trans comes out for surgery, probably replace the clutch too since you're already there.