Hitoyoshi Castle makes most sense as a short historic site while you are already near Hitoyoshi. Use it as a pause between larger stops.