How to release memory for sql?

From @amin_sushanth via Twitter:

On #windows2k12 R2 the memory does not get released @AzureSupport.I am running microsoft sql on windows 2012r2.Due to certain load they the memory usage goes to 23gb but after the task is completed the memory usage should come down? But it does not come down and it's stuck at 23gb so I have to restart the server (vm) after which the memory gets released. I have applied all the security patches through windows update tool.I am using D5_v2 size.

Thanks

@azuresupport