Yeah, it went downhill during Season 3. Up and down, sure, but you can't say Harmon leaving caused a decline when he was there for shite like the whole Emperor Chang thing. The gimmicks did get a bit to much too, I agree. The D&D episode is one of my favourite episodes of anything ever so I don't mind them to much when they could hit high points like that, and I don't think any of them were near as bad as the puppet one. Oh that was dreadful. It was a good thing you turned it off before it reached the end where they shared their worst secrets.
That said I haven't liked season 4 much at all even when it's being normal. It's gone overboard with lazy exposition, explaining their own jokes and throwing in references for the sake of it. Where Abed once was a character who used tv tropes as a way to cope with reality, he's now just used for lazy exposition under the guise of being meta. I don't mind his freak outs though. Autistic people sometimes react like that. They've done nothing with Troy all season except a few lines about him being dumb which is a real waste of a talent like Donald Glover. Annie seems waaay off too, with that secret of hers and the way she was pretending to be Jeff's wife. They've undone all her development and made her more childlike and naive than she was at the start of the show. There's still a few decent jokes, and the cast is good enough to elevate subpar writing to something watchable but it's a long way from the first 2 seasons me thinks.