The kernel maintains the filesystem buffers/cache. It uses unused memory only, and will make it available immediately if a process tries to allocate it. If you restrict the size of the filesystem buffers/cache (I'm not sure you can set it, but let's suppose), you'll only limit performance. If you set it at a fixed size (again, suppose), you might run into a situation where processes get swapped to disk while you have unused buffer/cache space.
So my advice is, assuming you're talking about filesystem buffers/cache, i.e. the second line of numbers in the output of free -m, don't fiddle with it. The more memory the kernel can assign to it, the better.