Problem statement

I was recently working on a support issue, which had client- and server-side aspects. Complicating the issue, we only had partial visibility into server-side health and no visilibity into client-side health. It was hard to even tell where to start investigating. We were also working closely with a partner, who could give us some visilbility, but with high coordination cost.

One approach was to create client visiblity for us and the partner, but this would take time to roll out, didn’t immediately reduce the coordination cost and risked fatiguing the partner.

An alternative approach was to increase the server-side visiblity. We took this approach because we could start investigating immediately (no coordination cost or roll out latency). We might even be able to resolve the issue without requiring the partner to do any work. Also, having more visibility and confidence in the server-side would help if/when we do need to make client-side changes.

Solution

So, my takeaway is simple: when faced with a choice between server- and client-side options, bias toward server-side.

From a product perspective, any work required of customers is friction to adoption. From a technical perspective, it’s much easier to change servers than clients.

Problem statement

A project I’m familiar with recently had a series of issues. Each issue was investigated somewhat independently. It was hard to share common code, share data across roles and track progress over time.

Solution

1. Capture canonical queries in version control
2. Periodically run queries, persist and visualize output (aka ETL)
3. At a higher level, invest in tooling to facilitate such dashboard creation

The end result is much more awareness of the underlying data. Folks in different roles can see the data and ask questions, which often improves the quality of analysis. For example, we now review the dashboard weekly and look for changes as we roll out fixes. Because we now have a pipeline, we can also run different data sources through it to check the analysis.

I’ve recently been reviewing student resumes as part of a university recruiting program and a few best-practices stand out.

Google has a good video on resume formatting that describes most of these practices. In these cases, I mention a time in the video.

Ok. Here are the best-practices:

• Recommended resume format at ~2:30
• Resumes are read in the context of a job description (example). Companies hiring for a job read a resume to see if you have the training and/or experience to do that job, so
  • You can briefly describe what the project was, but invest most space describing what you did and how you did it. Google recommends a phrasing at ~6:00: “Accomplished [x] as measured by [y] doing [z]”
  • Bold key words to highlight your toolkit (~5:25), eg “backend engineer”, “python”, etc, so the reader can match the resume to the role at a glance.
• It takes a lot of effort to tune a resume for all applications, but may be worth it for the 1-2 jobs you really want
• Ideally, your resume tells a story, eg some Python in year 1, more Python in year 2, created a service using Python in year 3 –> this person has experience in Python and is using it to start exploring service eng
• Submitting a resume may feel like an impersonal process, but it’s just people on the other side looking for new teammates. That’ll be you after you’re hired 🙂 The points above try to make it easy for that person to see you’re a great fit for the job.

This is an organizational pattern I like:

• 2-5 ppl
• Cross-functional
• Focused on a specific goal
• Weekly demo to squad

I’ve heard this refered to as a “squad”, “swarm”, “e team” and “feature team”.

One of the nice things is the sense of comraderie from working closely with a small group on a specific goal. Another nice thing is the group dissolves after the goal is accomplished, giving the members closure and chances to try new things without switching teams. Another benefit is broad awareness of how a system works.

This pattern works well in a larger context:

• Shared code ownership
• 10-50 ppl
• Focused on a specific, but larger goal
• Fortnightly demos from all squads
• Shared calendar for all squad coordination meetings

When a squad accomplishes its goal, the members dissolve into the larger group. Individuals can learn about work in the larger group by attending the fortnightly demos and/or sitting in on other squads’ coordination meetings.

For example, a product may be supported by several teams. To avoid exposing the org chart in the product, make a large team who’s goal is to make that product excellent. Define a squad for each significant work item. All members of the large group are free to contribute code to the product.

An underlying principle is alignment around customer value over specific products or features. Rather than a group of people owning a code base in perpetuity, regardless of the amount of work required, squads form in response to need.

A counter-example would be several teams supporting a product and that product having disjoint features. Another counter-example would be a large team trying to maintain the interest of all its members in weekly coordination meetings. Yet another counter-example would be lots of individuals working in isolation, esp if they’re doing so to avoid coordination co

Prioritizing unplanned work is hard, especially when we’re overwhelmed.

In this context, I’ve found it helpful to split things into two groups: important or not. My team calls this “binary prioritization”. We work on the important stuff and forget everything else; anything misidentified as unimportant will bubble back up.

Brandon Chu’s essay “Ruthless Prioritization” provides a helpful distinction: work within a project vs work between projects. The former benefits from binary prioritization; the latter from a more rigorous process, like OKRs.

This also reminds me of Amazon’s “Bias for action” principle, and the Agile principles. For example, Agile embraces change (“Agile processes harness change for the customer’s competitive advantage”). Binary prioritization enables us to embrace a high volume of change.

Here’s the pattern:

1. Identify the owner
2. Have the owner describe the problem and solution in a doc
3. Invite stakeholders to comment on the doc
4. If discussions on the doc linger, schedule a meeting to finalize
5. Document conclusions in the doc, to facilitate communication and closure, and for future reference

This may seem obvious, but I often forget it, especially in cases where a task starts small and grows in complexity. A problem may seem too small to formally “own”. A solution may seem too trivial to document. Stakeholders may attend a meeting without context. A meeting may conclude with stakeholders feeling like they’ve expressed themselves, but there’s no actionable plan to resolve the problem.

Identifying one person from a group of stakeholders to own the project and be responsible for leading work to completion reduces organizational ambiguity.

Documenting the problem and proposed solution in writing reduces ambiguity by capturing ideas from a variety of mediums in a single, relatively objective place that stakeholders can comment on.

Documentation alone may achieve closure, but it may also spawn extensive commentary. Meetings are relatively expensive, but scheduling a meeting to drive closure reduces ambiguity by distilling commentary on the document into conclusions.

Documenting conclusions reduces ambiguity by rendering them in an objective form all stakeholders can agree on.

A few symptoms that indicate when this pattern might be useful:

1. Endless back-and-forth in chat, bug comments, etc, which can give the impression of progress, but never resolve the issue
2. Multiple and/or cross-functional stakeholders, which can obscure priorities
3. Multiple people opining on a solution and/or answering questions, which can obscure ownership
4. A problem that drags on, which can indicate it’s important, but inappropriately owned

This ties into larger discussions around project planning (e.g., managing planned vs unplanned work), and meeting efficiency (e.g., inviting stakeholders, assigning pre-work and clarifying outcomes), but the point here is just to succinctly identify an organizational pattern and when it can be helpful.

Subbu Allamaraju wrote a great essay: The Value is in Dealing with the Messy Stuff.

I’ve been thinking about it since I read it, and have recommended it to a few people. A few lines stand out to me.

I struggled, in the beginning, to get such problems solved. I used to get frustrated and sometimes considered moving on instead of trying to find a way …

I can relate. In my case, I wondered: am I really the first person to think this situation is sub-optimal? Or, I must not belong.

My attitude changed once I changed my mental model.

I was happy with most things and didn’t want to move on. I was also wary of “grass is greener” thinking; as The Galapagos Affair put it, “Paradise is not a place.” If I wasn’t going to move physically, the only way to flex was mentally.

Now I see such problems as being ambiguous in nature

This word “ambiguous” carries extra weight for me because it’s used in performance reviews, eg how comfortable am I with ambiguity?

I’ve been interpreting the word to refer to lack of guidance, i.e., how comfortable am I picking from several solutions independent of guidance, but that didn’t alleviate the frustration mentioned above.

Allamaraju’s essay makes me wonder if another interpretation of ambiguity is a tolerance or even appreciation of mess; how comfortable am I with imperfection?

It requires empathy with existing silos, humility to let go of your ideas, and patience and tenacity to influence others

A colleague recently distilled what he’s learned over the past few years: it’s about people.

As Preet Bharara mentioned in his discussion with Dave Chang, we want calm people who encourage innovation in leadership positions; fear and intimidation don’t work.

Dr. Robert Waldinger summarized a long-term study of adult development: “The clearest message that we get from this 75-year study is this: Good relationships keep us happier and healthier. Period.”

There’s a lot of institutional knowledge to learn in an organization with history. The rate of change is high in a large organization. We want people to think independently, but this complicates the process of affecting change. There are many reasons to have “strong opinions, weakly held.”

… instead of acting like a victim facing a villain. You should be comfortable with the mess that comes with owning up an ambiguous problem

Sometime’s there’s a person responsible for some unfortunate circumstance. I recently heard a story about an executive who didn’t perform any engineering due diligence before signing a contract, resulting in much organizational stress. More often, I think the problem is entropy. There’s no one person at fault, and the unfortunate situation is due to a combination of circumstances.