If smaller local sights appeal to you, note Asuka-dera Temple while visiting Asuka. Allow extra time only if the architecture interests you.