Sir Anthony might help out, but I think he's got a smaller estate and fortune than the Crawleys would have anyway.
Matthew also didn't grow up the same way Mary did. He is from a pretty solidly middle class family (both of his parents worked) and went through school and became a lawyer. Even though being the Earl of Grantham and living in a sweet house like Downton would be fantastic, it wasn't the life he imagined having and it's not the only life he's equipped to live. The money isn't nearly as important to him as it is to Mary, who faces the possibility of seeing everything she knew slipping away. Coupled with the honor that romantico mentioned, of course Matthew isn't jumping at the chance to take the Swire fortune.
When I see the phrase "the ____ estate", I imagine a vast mansion in the country full of monocled men and high-collared women receiving letters about productions across the country and doing spit-takes at whatever they contain.
-Kad