Just open up Valheim's launch options by right clicking on the game in your Steam library and selecting "Properties." Then paste this string of text into the "launch options" field, close the window, and launch the game.
-window-mode exclusive -screen-fullscreen
This will force Valheim to launch in true fullscreen mode, instead of fullscreen windowed, and I can personally vouch for the difference it makes. I have a GeForce RTX 3070, and my frames went from holding steady at 100-110 FPS in my base camp to 130-140 with the new launch options and true fullscreen. The original poster saw an even more dramatic improvement, going from 50-60 FPS in their base to 90-110. Make sure your monitor's native refresh rate is selected in Valheim's in-game graphics options to see the full effect.