From Strunzy's Hunter Guide on Ensidia.com
Before getting into what and how you should gem, I'm going to break the ice and explain why some hunters choose to gem for Arm Pen, and others for Agi. The thing with Arm Pen, is that it increases exponentially.
Exponential growth is just like the classic problem, where every time you double the number, its relative distance from the last number increases vastly. (2,4,8,16,32,64,128,256,512,1024,2058). As you can see, especially on a larger scale, at first the numbers are small and relatively close to one another, but once you get higher, the yield beings to increase a lot more.
The formula for Armor is the exact opposite. The more armor you have, the less of an increase in damage reduction you gain per armor. For instance, a tank with 5k armor has about 25% reduction, but a tank with 10k armor has about 40% reduction. The armor doubled yet the increase in damage reduction only went up by about 50%, not 100%. This is why the armor pen has the exponential growth. The less and less armor the mob has, the more significant the loss in damage reduction.
The thing is though, people are expecting that the second they hit this "magical" amount of Arm Pen, their DPS will sky rocket from the 5k they were doing before to the 8-9k they see others doing. This is far from the truth. The increase is minimal yet it does exsist. A hunter though that decides to gem Agi instead is still highly competitive, they'll just excel in certain area's that the other hunter won't. So what are those area's?
The Arm Pen hunter will top the meters on your class tank and spank fights. The less movement involved, and the less AoE involved the better. You'll become a one trick pony, in the sense that you can't move, and you can't AoE, you can only single target. The Agi hunter will excel in the these same things.
So when will you know if gemming Arm Pen is the right thing for you? When you have around 800 Arm Pen rating, without gemming for it, and without having any trinket procs, then you know you are ready.