While playing your records and even during storage of records, air pollution from outside (car exhaust, dust) and even from inside the house like smoking cigarettes, cooking fumes causes a tough film to develop on your records. This film is hard to clean and cannot be cleaned with a record brush only. The stylus, while playing back, will scrape part of the residue which will gather around the stylus tip, causing distortion in playback or even skipping out of the groove.
In order to have faithful playback of the records, check your stylus before playing or when the music starts sounding bad. You may think your stylus is worn and needs changing, but in fact the stylus only needs cleaning. The best brush for cleaning the stylus is the Tonar Clean tip carbon fiber brush, preferably in combination with a good stylus cleaning fluid.