I feel very much the way you do about most of these scenes, only you express it better. ;) I too always assumed that Giles at Port Wade was either on duty or on a tightly planned schedule of time off, with no room for younger sisters suddenly appearing. It's also interesting to think that this is really the first time Giles is interacting with either of the twins as independent people, as opposed to small-children-in-bosom-of-family, and he doesn't have a good sense yet of the right pitch for the interaction between twenty-whatever and thirteen (twelve?), which is not quite the same as older brother kindly playing with baby sisters.
(And what I seem to end up doing is getting absorbed in the world of the book as I read, and then chewing it over later for fun with analysis...I can't do both at once, at least not with books I'm this fond of.)