Hatton - no jab, wasn't throwing enough punches (but you can also argue Floyd was neutralizing his style)
Baldomir - punches too slow and wide, no jab, no head movement
Chavez - too much blocked ineffective punching to the body, no jab, no head movement
Augustus - too wild, no technique (I heard that Floyd said he was his toughest opponent)
De La Hoya - needed to be more accurate, stopped throwing the jab, no head movement
I'm not trying to be a Flomo, I don't think Castillo was so tough because of Floyd being inexperienced, he looked very experienced in his fights before this... but Floyd honestly didn't look the same in that match. He complained a couple of times to his corner about his arm hurting, he wasn't throwing enough punches, wasn't moving his head much... He just looked uncomfortable and looked like he was trying to survive, and then was told to stand in there and fight in the last rounds. Floyd easily dominated in the second fight IMO, I wouldn't say he just improved that much in the 8 month time frame from their rematch.
People think Pacquiao will beat Floyd because of the theory of pressure, but do people pay attention to how Pacquiao fights? He's not even a inside pressure fighter, he's more of a mid-range brawler. Plus, most of those these pressure fighters being named also liked to ignore punishment, and were walking through Floyd's punches. Pacquiao doesn't like to get hit and take punishment, he usually backs up and sometimes often goes to a defensive mode.
My observation and somewhat theory on how to successfully beat Floyd,
Floyd likes to box - Floyd is most comfortable in the center of the ring. Most of his opponents were either not too good at boxing, or were not fast enough. I think someone would need a consistent, accurate, and fast jab. I would say a decent reach, but if you look at when Floyd fought someone like Tony Pep, Pep had a long reach, but it wasn't fast enough, and Floyd eventually neutralized it.
Floyd is fast - If you're not faster or just as fast, than be powerful.
Floyd is accurate - Head movement; most people who fought Floyd don't consistently move their head, or move it at all.
Floyd has good reflexes and head movement - That's why you need to be accurate as possible, and deceptive.
Floyd is deceptive and a counter puncher - Floyd usually likes to lead with a jab to the body, a jab to the head, a left hook, or a straight right. He often likes to use a stiff stance, which one he'd throw is sometimes tricky, but you should be expecting either... the counter-right should be easy to see coming, his stance when he's about to throw it so obvious. I guess someone would need a hawk's eye to consistently see and counter what he'd throw (something Judah was able to do in the earlier rounds). Always keep your guard up, and not just at the head or you'd get jabbed to the body.
Floyd can fight backwards - Most of his opponents weren't very good at fighting backwards. Floyd can, and when you're trying to pressure the fight, be weary of being countered.
Floyd has good footwork - Floyd's positioning and movement is arguably the best I've seen... Not really an answer to this other than being fast on your feet and also trying your best to cut him off.
Floyd is defensive - Most people don't seem to have an answer to his defense, the common theory is pressure, as everyone has seen so far, everyone has failed :/ I think the reason why that is, is because of a lack of other attributes, such as a good jab, accuracy, head-movement, speed, countering, deceptiveness, stamina... They have a good chin, but that's not enough to beat Floyd, as he's still going to counter and hit you and gain points. I also noticed that when people get him in his defensive stance, they almost never throw an over hand right to the head, something Castillo did do and caught him solidly a few times.
Floyd is versatile - He can box, he can brawl, he can fight on the inside, fight forward, fight backwards, can counter-punch...etc... He's just multi-dimensional. You need to be more than one or two-dimensional, as for when your plan A or plan B is neutralized, what else are you going to do?
Floyd is smart - Floyd is too good at predicting his opponent, adjusting, and neutralizing. You can be as smart as Floyd, but it won't matter if you can't perform how you think. Or can't think how you perform

Floyd has stamina - He can do this for 12 rounds. If you can't, train harder, build that stamina, or try to get something in the earlier rounds; Many have tried but have failed.
To put it simple in my opinion, not saying Floyd is unbeatable, but you almost need the full package. Some people have done well but they lack many or specifically other qualities which is why they fall apart and/or get dominated.