Caffeine And Acne

I have suffered from acne since my teenage years..I can hardly remember a few times when skin was clear..
I have noticed that my acne becomes worse when I drink coffee or tea..
Even if I drink one cup of coffee or tea per day, my skin becomes excessively oily and breaks out terribly. I get a little relief if I go off caffeine..as per my observations.
I can a mange without coffee.. But I like to drink tea now and then..I thought tea has lesser caffeine and will not affect skin..but recently I noticed an increase in acne after drinking one cup of light tea per day and hence stopped that also..
My question is ..is it just me who is sensitive to caffeine..or anyone else also suffered from increased acne after having caffeinated drink regularly?
Any remedies for this so I can take an occasional cup of coffee or tea with my skin not getting affected?
Is it ok to drink green tea or will that also aggravate acne?
I see so many people who are compulsive coffee or tea drinkers, they drink 2-3 cups coffee tea per day and have clear skin..wonder why I'm so sensitive.