This is where flexible dieting can really save your diet entirely, get most of your calories from good whole foods then finish the last 10-20% with things you're craving. Keeping it too 'clean' for to long can lead to what you're describing and ultimately end in binging and putting on more weight.