You're leading whether you intend to or not
What you do and how you do it makes a difference in what everybody else does.
If you are a leader, you can't 'turn it off'. It's a full-time job. Even if you shirk your responsibility you are still leading. Notice how others act when you lose conviction or pretend mistakes didn’t happen.
In my experience I have allowed myself to become distracted from my responsibility. Not in the sense of resting and taking care of my needs, but in the sense of focusing on what I wanted and neglecting the attention, decisions and communication of my role as a leader. My due attention was always brought back by a problem, never a little victory.
I've watched this happen, too. Discussions among the staff about 'what should we do?' or 'how are we supposed to resolve this?' while the leader was just absent. Not there even if he was there in the room doing most of the talking, or present in the meeting or on the phone.
We were missing the leader's decisions that would shape all the actions and judgement calls we faced. We were trying to get in alignment on our own.
Your jokes send messages. What is valued? What deserves ridicule? You can't turn off influence.
Your attention sends the message: what matters? Where should efforts be directed? And it communicates that the places your attention doesn't reach doesn't matter.
Leaders aren't in control of whether they are leading or not. Their followers do. A leader can shrug off followers when he damages their trust long enough or makes big enough mistakes.
A leader's job is to make decisions than align everyone, and to consistently communicate the message so people can stay aligned and work together well. Since your followers are paying attention to what you do and say, make sure it's geared toward the results you really want.