Jōroku-ji makes most sense as a short place of interest while you are already near Nabari. Use it as a pause between larger stops.