>>>I meant "slumming" to mean going to a poorer part of town, not the library!

Well, you'd expect that, if you mean a publicly funded library.
I remember when it was more socially acceptable to subscribe to a circulating library than to join the public library:
"Oh I never borrow the books - I just use the reference Library."
Daughter of Time (Tey)has a character who is surprised that the book she could only find in the public library "isn't at all soupy".