As much as I am embarrassed to admit, I have been using vi/vim for the last 10 years or so (admittedly only on remote servers for editing configs for 8 of those years) and I am just now getting around to getting better at it.
Folks may argue that the transaction cost of getting to the arrow keys is insignificant, but it ignores that when you use the arrow keys, you essentially are moving your hand away from every important vim command. That minor increase in transaction cost by moving over to the arrow keys is multiplicative with the transaction cost of using the standard vim commands. In other words, you are far less likely to learn other vim commands when you have the friction of moving your hand back in addition to the friction of learning the command itself.
Once you have relinquished your hands to being ready to learn, you can start adding on top of that with new commands as you move around.
Keep in mind whilst using vim: it is more-or-less feature complete. If there is something that you feel you are not doing as fast as possible–you are doing it wrong. So whenever you feel the friction you felt when you used the arrow keys, look around and find that command that lubricates that action.
Ultimately, the reason that vim zealots are so frothy about their editor is that once you have the muscle memory to be completely fluid in the editing of text, you can focus on the actual task at hand (and not the tool being used). Every time you have to do that minor transaction between your hand and move it to the mouse, or feel encumbered by your editor, you are an artist concerned with the sanding of your brush handle or the easel holding your canvas. These are things incidental to the end result: the product. The focus. The very point of using a tool: to produce.