Well the Japanese government, as part of their efforts to attract more tourists, will be rolling out a WiFi system of their own that is being aimed specifically at tourists. Basically how it works is that visitors to Japan can get an access ID either by downloading a smartphone app (click here for iOS or Android), or showing their passport at tourist spots or airports.
This will allow them to access free WiFi in train stations and other locations, such as Tokyo, Hakone, Mt. Fuji, Yokohama, Nagano, Nikko, Kusatsu, Tohoku, Hokkaido, and Fukushima. The government is expected to set up a committee in which they will attempt to come up with a common ID that can be shared across a variety of locations, such as hotels, airports, railways, and so on. However the government expects that their WiFi system will only be made available in fiscal 2016, so there’s still some waiting for those planning to make a trip anytime soon.
In the meantime there are alternatives, such as going to certain convenient stores as well as heading to Starbucks. Local telco NTT East also has a similar service, although it might not be as universal or as widespread as it could be.