I understand that you can't code by E&M time when using a psychotherapy add on.  I get it. What I'm talking about is ONLY using an E&M code (without psychotherapy) but with counseling when time spent counseling > 50%. This is clearly in the APA coding directions:

2013 Coding Change FAQs: American Psychiatric Association CPT Coding Resources for APA Members
"Q: What are the times for the various E/M codes for established patients, and is there any reason you couldn't use the 50% counseling and coordination of care for every follow up visit if it applies?
A: Correct coding requires that you choose the code that most closely represents the work performed. If more than 50% of your E/M service involves counseling and coordination of care, you can choose the code on the basis of time. [AND YES IT'S TRUE THAT...]You cannot choose the E/M code on the basis of counseling and coordination of care if you also bill a psychotherapy service in the same visit."

So my question is, is there any downside to coding (only) an E&M by time?