As a foreigner who has left Japan, I am able to claim back the pension tax I paid. As nothing administrative in Japan can be easy, I spent 5 hours today filling out the form. I noticed they did include a number to call with questions, however, written underneath was:

Now remember, this form is only for foreigners, not for Japanese. Yet, in order to get help they require you speak Japanese.

Here is the illustration from the Pension reclaim form. Little cute cartoons are found on almost everything in Japan and westerners are occasionally depicted with crosses for eyes, which I always thought was a symbol of death or drunkenness.