Figured it out, apparently the bolt that came with my underdrive pulleys was just to install the crank pulley. The instructions don’t make it as clear as I feel they should, that you’re supposed to reuse the old one. Which I had to go get a ridiculously expensive bolt because I tossed the old one thinking the kit provided a replacement one. Thankfully it didn’t cause any issues. Essentially the washer was just grinding on the crank pulley a bit sometimes. Still a little confused as to why I had it a bit before doing the pulley’s, best guess is that was from my idler pulleys.