To indulge oneself in something that only serves themself in general can seem selfish. But is it really? When it comes to journaling I don't think so. Sure you are doing something just for yourself, BUT given how journaling can bring peace of mind and inspire, isn't that good for everyone around you? I say it is.
So journal away and enjoy that time. Nay, demand that time for yourself to journal. You and the people around will be glad you did.