Engineering Lead series, Chapter 2 - Deep Dives

It seems like my previous post was liked by a number of people, maybe this is a topic I will continue writing about. Here is Chapter 2...

Today I had a team member tell me during stand up that he couldn't figure out the ticket he was working on, so he decided to just work on something else.  Then, our team's coop piped in and said he too looked at the same ticket, and couldn't figure out how to replicate the bug (which clearly still exist in our production and staging environment), so he put it back into the Ready column of our Jira board.

This, got me rather upset.

It would have been perfectly acceptable if they had said, "Hey, I need hand with this", but to just give up? And worse, give up and put the ticket back in to Ready state? What sort of solution is that? How is this going to help any of the other team members? How is this going to help our customers? No, giving up is simply not acceptable.

I imagine one of the reasons we get into the field of technology is we like problem solving.  While we may not admit it, the most odd duck weird one off bugs are the stuff that we enjoy digging at the most (sometime at the frustration of our PMs since these tickets take far longer than expected to resolve). It should be that professional curiosity that drives us to think, "hey, I don't understand why this is happening, but the code is all there, and the problem clearly exists, so I am going to figure this one out!"

I wonder if perhaps, stubbornness is a trait that is actually is helpful in our field. Unless you aspire to be a mindless code monkey and never want to tackle any real tough problems in our field, there will be plenty of times where something just doesn't work the way you expect it to work, and it's going to take a lot of time and energy to figure it out. I honestly think these are character building moments and opportunities for real learning. When you do finally get the breakthrough "aha" moment, it will be well worth the effort. Sure, there will some occassions where that "aha" moment also came with the sudden realization that it was some terrible typo that caused the whole thing, but there are also many other moments when you really learn about how something works under the hood. When you get that sort of a moment, it feels immensely satisfying to have done the deep dive.

I think it was this stubbornness that drove me to spend thousands of hours honing my skills in the shooting sports. Standing on top of the podium and having "O'Canada" played for me isn't something that comes easily. It is the same stubbornness that when I encounter technical situations like what my team member ran into today, that I will dive in with both feet with the aim of getting to the bottom of the issue!

I am an Engineering Team Lead, but what do I actually do?

My title is Engineering Team Lead, but what does that actually mean, what do I actually do?

I am Project Manager

Nearly every business request impacts multiple components, front end, back end, and various other subsystems.  I receive business requirements from the Product Manager and break down the request into workable chunks. I translate business requirements to engineer speak, and create "ready to be worked" on Jira tickets. I make sure each ticket has as much background information as possible so the individual working on this ticket can have some perspective of the larger picture. When multiple Jira tickets are required to implement a business change, I have organize them so tickets are put into the sprints without unmet external dependencies.  In order to do this, often I have to actually dig into the code bases, roughly plan out the changes that would be required, sometimes testing some rough ideas before I can confidently write a Jira ticket. It is either that, or every ticket is going to be an “Investigate”, which often results in no actual value to customers.

Most of the time there are also multiple projects running simultaneously. This means I have to mentally organize and prioritize them into our sprints, manage their progress against their due dates, and communicate both up and down to make sure everyone is on the same page. I also run all of our backlog grooming, sprint planning, and retrospective sessions. I collect metrics on team as well as our product performance, track and plot them over time so the team can continue to be diligent on improving quality.

I am Gate Keeper

Since I’ve been at the company the longest, and know how all the small tiny gears fit together to make this giant colossus work, I review almost every bit of code that is in our team’s various repositories. While other reviewers pick out style or syntax improvement opportunities, I am often the person who discovers larger issues in logic, data flow, deployment considerations, or sometimes simply the wrong implementation in the wrong place. In order to do this, I often have to dig deep into the code base, thinking about what a good technical solution would be, and then evaluate that against the diff I am actually reading.  Of course I don’t always come up with the best solution. I am ecstatic when I see a better solution implemented, but even then, there are often rough edges that must be smoothed out before it can go to production. Larger code reviews have to wait for the weekend because it is the only quiet time I have, it is either that, or staying late at the office long after everyone has left.

I work on guiding our team towards higher test coverage, more robust fault tolerance, better documentation and easier to read code. Rarely am I able to just accept a patch as is, since often my younger team members due to lack of knowledge or experience have written code that would have broken one part of the system or another. When something fails in production, upper management come to me to demand answers. As such, I suppose these efforts are as much for self preservation as it is for team improvement. We do also require two reviewers approval before any change is accepted, on a small team, it pretty much means I have to look at every diff.

I am Quality Assurance

After code changes are finalized and landed on the master branch, individual contributors are happy to move on to the next bit of work. I on the other hand, have to do another two rounds of checks, once when the changes are released to our staging environment, and a final round when it makes it to production.  Thankfully most of the time these checks do not reveal problems, but there certainly has been cases when bugs have been discovered due to subtle environment differences.

I am Customer Support

When something breaks in production, other teams and upper management come to me for answers.  So even when I am not on triage duty for the week, I am still on. This can be as innocuous as a “can we do this?” question. This will usually lead me to dig into our code bases, looking at logs, thinking about potential conflicts with other systems, and then offering an answer or alternative solutions. It may seem innocent, but a few of these questions in a day, and there will be little time to do anything else. Have I mentioned that we have offices around the globe and operate in numerous time zones? Yes, so that means in order to provide a high level of service to all of my team’s customers (not only the end clients, but the other offices that depend on our services), I find myself responding to email/slack/Jira messages at all sorts of odd times of the day.

I am Mentor, Psychiatrist, and Jury

I have a lot of young developers on my team. Some find the day to day work dull, others try to come up with complicated solutions. I try to mentor and help them achieve their best, offering support and guidance along the way. When a team member is not engaged, I have to figure out what the issue is, and how I can get them back on track. I do this by day to day working alongside my team members, with lunch time conversations, and with my bi-weekly one on ones with them.  I am always listening, I am always watching. When it comes time to do performance reviews, I have to assemble everything I know about a person’s contributions, and try to come up with constructive advice to help improve their future performance.

I am Visionary and Architect

On top of all the day to day work, I also have to keep up with the industry’s latest trends, in order to offer my team a glimpse of the future to look forward to. This means I spend my holidays and weekends reading and learning about new tech that’s always coming out faster than I can consume.  I honestly wish I could be in the Matrix and just download the latest flight program to a Huey directly to my brain. I also have to think about how to improve our processes, technology, and architecture in order to better our customer experience, both internal and external.

I am Talent Acquisition

Since our team’s small size often causes problems in availability of resources, we’ve ramped up hiring. This means I need to go through resumes of potential candidates and go through rounds of interviews. Thankfully a team member has stepped up to significantly reduce my effort in this task. Ultimately however, if I hire someone who doesn’t meet expectations, then I will have another problem on my hands.

These are the things that I do nearly on a daily basis. Most days I feel like I am being pulled in a hundred different directions. It is exhausting.