What this means is that there are three weeks in between the first semester (from April through July) and the second semester (from the end of August to the end of the year). But this does not mean the teachers or even the staff gets three weeks off. Every day of that three weeks is technically a work day, so if you want even one day off, you use one day of annual leave to take it. In my school, everyone gets 20 days of paid leave per year. This is a lot, for Japan-- many companies give you only five days. Also, in general you have to take vacation days when you're sick, because if there's no doctor's note, it didn't happen. And remember that the 20 days is also to be used over winter break and spring break. Also, sometimes you're told to take one of those days: if there's a conference or something that you can't participate in, but it's a work day, no one is at the school, so there's no point in going, but it is a work day.
For the teachers, the problem is not just with the shortness of the vacation, but with all the other stuff they have to do too. All of the teachers have many administrative duties besides teaching, and the whole Japanese school structure compounds that. For instance, there is no kitchen in the school; school lunch is made in a school lunch center a few miles away, and then distributed by the students. One of the English teachers' jobs is to liase with the school lunch center and oversee the student distribution process. I have no idea what most of the teachers do-- there's budget stuff, PTA stuff, sports teams stuff-- but there's a lot of it.
Anyway, add it all up and most teachers are taking one week off for their summer vacations. And this is a pretty long time by Japanese standards. My school, in northern Nagano, in central Japan, is close enough to Tokyo that you can go there any weekend, and we're close to Nagano City and Karuizawa too. But for anything else, you really need a longer break. A few teachers are going to go overseas, because it's actually cheaper to go to India or Hawaii than it is to go to Hokkaido. Travelling in Japan is expensive. And travelling outside of Japan is cheap. A friend of mine visited Seoul, and the travel company arranged for her round-trip flight and lodging for 3 days, and it cost her I think about $300.
That said, I do not plan to travel. It's not the money or even the time--it's just too much work. Yes, there are many beautiful and interesting places, but I think they are almost as beautiful in books or on video. My perfect vacation does not involve getting up early (which I would have to do if I were to travel) or spending long amounts of time in trains, or trying to find out where I'm supposed to go. I have to do enough of that any other day. No, my perfect vacation involves sleeping for as long as I want to, piles of books, and never spending any more effort than it takes to cause brownies to appear. Even in America, I had plenty of books, but never enough time to read them. And planning a trip and then taking it just feels like more work. I want to be lazy. I want to do nothing. A plate of brownies, a stack of paperbacks, and a couch-- that's all I need. Why leave home?