And I just don't think Matt Smith is the right guy to cast for him at all.
How so? It's not like you get some overwhelmingly obvious description as to his looks/physique, etc.
Spoiler!
I didn't have the expectation he looks like Hafþór Júlíus Björnsson. Also, you consider the time the show portrays and the ability to look the way some "powerful" actors do nowadays and I just don't think it's a match. A guy such as Rob Archer is just too big/cut for that time period. Matt Taylor might be your max, and not a bad choice. I just really think they got a good actor and making him fit the "description," will be something they'll work to maintain.