This may be obvious but, if you are super familiar with the song you will hear it in your head and naturally feel it. I think even drummers can drift if they are just keeping a beat but, if they are mentally playing a song they know like the back of their hand timing is easier. So my advice is practice the song more. I guess that goes for any kind of technique. You should certainly work on it separately but, ultimately, it ought to be in context of the music you are attempting. If youre speeding up or slowing down, you arent familiar enough with the tempo of the song. Time is something Ive struggled with. I was the kid at Jr High dance that could not hear the beat and looked pretty funny dancing because of it. Its taken many years for me to get a clue.