To add something to hanimizzle great post, the empire got too big... There was no way to hold it.
Some Imperators saw this, like Adriano, who builded the famous wall and put limits to the expansion; against the tradition of an "empire with no limits".
(Ironically, Hadrian was from Spain.)
It's just as well that he built his famous wall. The fens and hills of the British Isles, and the fierce Celts who lived in them, turned out to be a nightmare for the Romans, and they couldn't rout the natives as long as they didn't face them on the open field, fighting for personal honor in a very disorganized way.
All told, Hadrian had a good legacy, but cracks in the facade showed by then. Remember that the Romans were also nearly pwned in the Second Jewish-Roman War under his rule, but for the fatal arrogance of the leader of the uprising, a strange Samson/Ho Chi Minh combination named Shimon Bar Kochba. He and his followers uniquely waged war through a system of passages carved in the sturdy limestone of the Holy Land, to their great benefit.
If he were able to control his hair-trigger temper, some say the Romans almost certainly would have lost.