Launch stages

Agreeing to incremental launch stages and the criteria that govern promotion from one stage to the next makes launch commitments, aka deadlines, less risky. For example, Google SRE’s launch plan documentation describes four: Early Access Preview, Alpha, Beta and General Availability (GA). Committing to a GA launch is easier after a successful Beta, which is easier after a successful Alpha, end so on.

Comparable to a deployment pipeline, promotion from one stage to the next ideally concerns stability rather than features. In other words, if the difference between Alpha and Beta is a length of time without issues, we can promote with confidence after that time, but if the difference is a set of new features, we don’t know how those features will perform.

Rotating cadence lead

Rotating leadership for recurring team “cadence” meetings is a beneficial pattern I’ve seen on several teams.

“Cadence” meetings help a team or project march to a rhythm. All teams I’ve been on have had them, but not all have called them “cadence”. In agile terms, a daily standup is one form. A form I like is Monday kickoff, Wednesday discussion, Friday retro.

The pattern is simple:

  1. As a team, define the responsibilities of the lead, eg keeping the meeting focused on the agenda, cancelling the meeting if there’s nothing on the agenda, etc
  2. Identify the consistent attendees of the meeting and rotate the lead role among them

Such a rotation has a few benefits:

  1. There’s no single point of failure for keeping the team organized
  2. All members of the team get leadership experience, and no single person is stuck with this form of glue work
  3. Sharing roles engenders empathy between roles. For example, experience motivating participation as a lead can encourage participation as a non-lead. As opposed to “taxation without representation” 🙂

A couple anti-patterns I’ve seen:

  1. Non-overlapping leads and attendees. For example, having the eng oncall rotation also lead a cadence including EMs, PMs, designers, etc who aren’t on the oncall rotation
  2. Having the rotation include optional attendees, which can result in last-minute adjustments

I’m also curious about rotating team leads, eg as part of the Engineer Manager Pendulum, but I don’t have experience with that yet.

Bugs vs tasks vs goals

I am aware of three common approaches for tracking work:

  1. Bugs, eg Github issues, JIRA tickets, etc
  2. Tasks, eg items in a list of things to do
  3. Goals, eg some end state

There are probably many more, but I commonly see teams struggle to reconcile these three.

Part of the challenge is they’re all related and required in some context, but no one is sufficient. Bugs are required because people external to a team need a way to request work. Further, some of this work is essential, so bugs can’t be ignored. Bugs generally represent unplanned work.

Tasks are required because we need a way to deconstruct large projects into more manageable pieces. Tasks generally represent planned work.

Goals are required to separate implementation details from an objective. One of my colleagues phrased it well: setting goals shouldn’t be controversial.

Sometimes tasks can be represented as bugs, but bugs by nature are relatively formal, which breaks down when tasks change at a high rate. Some teams strive for tasks that take no more than one iteration, which is tedious to represent as bugs. I like the pattern of stating goals for the week, and reviewing progress against those goals at the end of the week, but this is tedious to represent as bugs or tasks.

My fantasy is something like:

  1. Monday kickoff and Friday review focused on a simple, written (so we can remember on Fri) list of goals for the week
  2. A support rotation monitors bugs. Active work on bugs is represented as goals for the week
  3. Large projects have independent task tracking. Active work on tasks is represented as goals for the week

The fact that my goal for the week concerns planned or unplanned work matters less than communicating to the team what I’m working on and how it contributes to the team’s priorities.

Praise for Markdown eng docs

Google has a technical documentation system called “g3doc”. The “The Knowledge: Towards a Culture of Engineering Documentation” presentation at SRECon16 described it well, so this post just highlights a few details:

  1. Documentation is collocated with code
  2. Documentation is rendered from code-like Markdown

The first point enables me to include documentation changes and code changes in the same commit.

The second point is appealing because it reduces the cost of context switching between code and documentation. For example, I can edit both in the same editor.

I think part of the appeal is Google’s monorepo. Everything is path-indexed, but things under a “g3doc” dir are rendered into web pages. Searching the repo returns results for code and docs.

Outside of Google, I think Github’s rendering of Mardown content is comparable.

Project governance

I was recently looking for an organizational pattern to 1) help design documents gain visibility, and 2) build a community of senior engineers. We have OWNERS files, but they specify lists of people for ease of maintenance, which complicates the task of finding an appropriate person to review design proposal for affected code. Engineers often have informal conversations about design options, but there’s no body of expertise to query before an impersonal inter-/intra-net search. I needed something in the middle.

This search made me aware of the Fuchsia project, and in particular, its use of the phrase “governance” for the the type of patterns I was looking for. In short: an “eng council” provides “a small group of senior technical leaders responsible for providing a coherent technical vision”; a Request For Comments (RFC) process provides “a consistent and transparent path for making project-wide, technical decisions”; an API council provides “a group of people who are accountable for the quality and long-term health of the Fuchsia API Surface. The council will collaborate constructively with the people who create and modify Fuchsia’s APIs to help guide the evolution of those APIs.” The Fuchsia project recently revised its governance model as part of opening the project for external contributions.

Google has an AIP process, which is like RFCs for APIs.

My team had an API Council, but that focused on the external API surface rather than internal technical decisions. The external focus and the fact it operated at the highest level, required more structure than could be justified for internal discussions. It was helpful to see this council in the context of governance, but I still needed a new structure.

With this in mind, I proposed an eng council structure for the team. We identified ~10 people with several years of diverse experience on the team. We have a weekly meeting, which we cancel if there’s nothing on the agenda.

Interestingly, it now appears team members are better able to find reviewers outside the process, perhaps because the range of reviewers is now more well known. I’ve also heard from eng management that the process has helped teammates have more confidence in their projects. Also of interest, this governing body seems to have a life of its own and needs to be cared for. After several weeks of empty agendas I proposed cancelling the process in favor of exploring options, but several teammates expressed appreciation for a weekly checkpoint, even cancellation is the common outcome.

Technical-organizational balance ⚖️

I recently started focusing on technical work after several months of organizational work and it’s been a lot of fun. I explicitly missed a couple things: being directly involved in a project, especially focusing deeply on a problem, and working closely with a small team.

A manager friend half-jokingly described the switch as career-limiting. He also joked we’re not paid to work only on things we enjoy. I can see his point, but there must be a balance. We are paid to do things well, and I think that’s difficult when we don’t enjoy what we’re doing, at least in part.

This recent switch has me thinking of “The Engineer/Manager Pendulum” and the follow-up “Engineering Management: The Pendulum Or The Ladder”. All the quotes below are from these essays.

If management isn’t a promotion, then returning to hands-on work isn’t a demotion, either.

I prefer the term “organizational” to “management” for the non-technical work I do because most people think of the latter as people management. I wasn’t a people manager, but I was focused on project management, spent most of my time in meetings and learned to avoid any technical work in the critical path because I had no uninterrupted time to focus on it.

A tech lead is a manager … but their first priority is achieving the task at hand, not grooming and minding the humans who work on it.

The author provides appropriate advice:

Stop writing code and engineering in the critical path

The author mentions skill erosion after two years, but I experienced something similar after just a few months. Perhaps because I needed to make room in my head for a diversity of projects, I lost the context to go deep on any one of them. My activities were described as “leadership”, but I felt like those more directly involved were actually leading in a technical sense. I can see a need for leadership that stays out of the weeds, eg to avoid the sunk cost fallacy, but my role felt like an awkward middle-ground.

I think of this dichotomy as “technical” vs “organizational”. Both are important, but difficult to do well at the same time.

Management is highly interruptive, and great engineering — where you’re learning things — requires blocking out interruptions. You can’t do these two opposite things at once.

“Maker’s Schedule, Manager’s Schedule” is an essay I think of often on that topic.

Anyway, I think this feeling of fun is positive feedback that it was time for the pendulum to swing from organizational work back to technical work.

… you can feel your skills getting rusty and your effectiveness dwindling. You owe it to yourself to figure out what makes you happy and build a portfolio of experiences that liberate you to do what you love.

I find the phrase “career growth” often refers to increased prestige, rather than fulfillment.

Try to resist the default narratives about promotions and titles and roles, they have nothing to do with what satisfies your soul.

Impact

A professional koan: of the projects I can work on, which has the most impact for the business?

Thinking about this highlights the importance of a prioritized backlog. If impact is a component of the prioritization, then the project with the highest impact is clear.

A senior manager recently commented on some faulty advice they’d heard about service development being more important than client development. They clarified that impact determines importance. We should be working on projects with impact.

“The Secret to Growing Your Engineering Career If You Don’t Want to Manage” makes several good points.

Many engineers become managers because management provides an obvious and well-defined leverage point to scale your impact.

It’s relatively easy to produce more if I can delegate work out to a team.

The less conventional paths outside of management require more creativity, and there are fewer available narratives of successful engineers outside of management for us to model ourselves after.

I can attest to this, and explore it further in “Technical-organizational balance”, but I find it surprising given the idea of parallel tech and management tracks is ostensibly common practice.

Your ability to decide where to spend your efforts to maximize your impact — what code to write, what software to build, and which business problems to tackle … You identify and solve problems that are core to the business, or you enable those around you to more effectively solve those core business problems.

Hopefully our ability to identify projects with high impact improves over time.

“How to Grow as an Engineer (Working Remotely)” also touches on impact.

It’s also not enough to just solve any problems. You need to be solving the right ones. You should constantly make sure there’s alignment between what you want and what the business needs…

Working agreement

I’ve had a good experience with something called a “working agreement” in the last couple teams I’ve worked on.

The main value I’ve seen is in the discussion, when everyone can have a voice regarding how a high-performing team operates. The process can help a new team gel. The resulting artifact provides a third value of resolving operational details for future reference. For example, rather than debate an issue anew, we can just reference the agreement, which all parties had a role in.

I’ve found a scaffolding helpful for structuring the conversation:

  • Core values
  • Expectations
  • Norms
  • Agreements

Core values are simple ideals and can be aspirational. For example, “we provide each other psychological safety”, or “we do our best work together.” An artificial constraint of, say, five values can help motivate discussion, and improve accessibility of the resulting list.

Expectations provide an opportunity to state explicitly what we might be assuming. For example, “I assume best intentions” or “I assume folks have the big picture in mind, even for small changes.”

Norms provide an opportunity to express preferences. For example, “(given designing, writing and reviewing code requires focused attention) I benefit from no-meeting blocks”, or “I read emails, but aggressively purge (so I don’t mind a follow-up a day later).” Note that norms are becoming more operational.

Agreements build on and finalize the preceding sections. For example, “(To work efficiently and respectfully) we’ll respond to code review requests within a day, or communicate otherwise” or “We’ll use a single email adress for the team, differentiated by suffix (so it’s easy to find all emails, but it’s also possible to filter).”

It can take some time to build rapport, especially if the team is new, so budget an hour for discussion, and likely an additional hour on another day to finalize. It’s helpful to share the scaffolding in advance, so folks can start adding ideas. Having a different person lead development of each section improves participation.

These teams have a stated goal to review the agreement periodically, but in practice I’ve found on-demand is sufficient.

“The Rise and Fall of Getting Things Done” by Cal Newport

Cal Newport, who wrote a book I like called Deep Work, recently wrote an article “The Rise and Fall of Getting Things Done” on the decline in popularity of productivity tools, largely due to their inefficacy dealing with a steadily increasing onslaught of said things.

I’ve tried GTD and various todo apps and can relate. Inbox zero still has value for me, though I’ve had more success with regular, aggressive purging than meticulous categorization. The former has also been an effective strategy in general, eg binary prioritization.

I’ve found team autonomy to be an effective goal, and was interested to learn about a history of the term in the workplace:

[Peter] Ducker argued that autonomy would be the central feature of the new corporate world

Newport makes a distinction that autonomy doesn’t mean isolation:

Productivity, we must recognize, can never be entirely personal

I agree. Prolonged isolation is an anti-pattern, but I still think there’s value in reserving time for focus work, focusing on the things we can change, and ensuring folks have what they need to make the changes they’re tasked with.

Newport has a great insight regarding overload being caused by a lack of awareness into other’s time:

Because so much of our effort in the office now unfolds in rapid exchanges of digital messages, it’s convenient to allow our in-boxes to become an informal repository for everything we need to get done. This strategy, however, obscures many of the worst aspects of overload culture. When I don’t know how much is currently on your plate, it’s easy for me to add one more thing […] Consider instead a system that externalizes work [emphasis added]. Following the lead of software developers, we might use virtual task boards, where every task is represented by a card that specifies who is doing the work, and is pinned under a column indicating its status. With a quick glance, you can now ascertain everything going on within your team and ask meaningful questions about how much work any one person should tackle at a time

The article repeatedly reminded me of agile, which is eventually alluded to: “Following the lead of software developers … What if you began each morning with a status meeting in which your team confronts its task board? …”

I like the idea of constraining input to status meetings. It’s a challenge in practice, but worth exploring in principle.

Bias to the server-side

Problem statement

I was recently working on a support issue, which had client- and server-side aspects. Complicating the issue, we only had partial visibility into server-side health and no visilibity into client-side health. It was hard to even tell where to start investigating. We were also working closely with a partner, who could give us some visilbility, but with high coordination cost.

One approach was to create client visiblity for us and the partner, but this would take time to roll out, didn’t immediately reduce the coordination cost and risked fatiguing the partner.

An alternative approach was to increase the server-side visiblity. We took this approach because we could start investigating immediately (no coordination cost or roll out latency). We might even be able to resolve the issue without requiring the partner to do any work. Also, having more visibility and confidence in the server-side would help if/when we do need to make client-side changes.

Solution

So, my takeaway is simple: when faced with a choice between server- and client-side options, bias toward server-side.

From a product perspective, any work required of customers is friction to adoption. From a technical perspective, it’s much easier to change servers than clients.