The answer to that question is up to you, in 2005 Tom Brady went down for the first time against the Broncos in 2005 and then went down the next year in the AFC title game 38-34 and again in 2007, but this time losing in the Super Bowl. Still, Brady had a record of 14-3, but his record took a downturn after that going just 4-5, how come? Defense.
In 2009, Tom Brady was fresh off of an ACL injury and wasn't 100% and rightfully so as many players do not play well after their ACL tears (except Adrian Peterson). But then the next season, the Patriots went one and done to the Jets after a 14-2 season. While Brady threw for 299 yards and two touchdowns, the defense against the Jets was just horrible and allowed Mark Sanchez throw for three touchdowns. In 2011, the Patriots brought another bad defense and still ran the table in large part thanks to Tom Brady and went to the Super Bowl where once again their chances were squashed by the Giants.
In 2012, after beating Houston, the Patriots took on the Ravens whom wanted revenge from last season and Brady's play wasn't as strong as he barely threw for 50% and threw two interceptions to one touchdown. And then of course this season, where the Patriots won against Indy and then went down against Denver.
Nevertheless, Tom Brady's playoff struggles shouldn't be a concern, but the Patriots have struggled as a team. The defenses have taken a facelift and gone are the days of Corey Dillon. The Patriots just aren't the same team anymore, but they are getting better.