Software engineering notes

Binary prioritization

leave a comment »

Prioritizing unplanned work is hard, especially when we’re overwhelmed.

In this context, I’ve found it helpful to split things into two groups: important or not. My team calls this “binary prioritization”. We work on the important stuff and forget everything else; anything misidentified as unimportant will bubble back up.

Brandon Chu’s essay “Ruthless Prioritization” provides a helpful distinction: work within a project vs work between projects. The former benefits from binary prioritization; the latter from a more rigorous process, like OKRs.

This also reminds me of Amazon’s “Bias for action” principle, and the Agile principles. For example, Agile embraces change (“Agile processes harness change for the customer’s competitive advantage”). Binary prioritization enables us to embrace a high volume of change.

Written by Erik

August 7, 2020 at 7:24 pm

Posted in org

Bias for action

leave a comment »

This is straight from Amazon’s leadership principles. It also seems related to binary/ruthless prioritization, the Agile principle of “Simplicity — the art of maximizing the amount of work not done”, and customer focus.

An ambiguous problem may have many possible solutions. We risk slowing to a halt investigating all of them. It’s helpful to regularly question if a given task is important. “Analysis paralysis” might describe this, but I prefer Amazon’s phrasing: “Bias for action”.

As a concrete example, in an interview, there may be many possible ways to solve a problem, but time is limited and solving the problem by any means is preferable to identifying all possible solutions without implementing any.

Written by Erik

August 7, 2020 at 7:03 pm

Posted in org

Tagged with , ,

A pattern for reducing ambiguity

leave a comment »

Here’s the pattern:

  1. Identify the owner
  2. Have the owner describe the problem and solution in a doc
  3. Invite stakeholders to comment on the doc
  4. If discussions on the doc linger, schedule a meeting to finalize
  5. Document conclusions in the doc, to facilitate communication and closure, and for future reference

This may seem obvious, but I often forget it, especially in cases where a task starts small and grows in complexity. A problem may seem too small to formally “own”. A solution may seem too trivial to document. Stakeholders may attend a meeting without context. A meeting may conclude with stakeholders feeling like they’ve expressed themselves, but there’s no actionable plan to resolve the problem.

Identifying one person from a group of stakeholders to own the project and be responsible for leading work to completion reduces organizational ambiguity.

Documenting the problem and proposed solution in writing reduces ambiguity by capturing ideas from a variety of mediums in a single, relatively objective place that stakeholders can comment on.

Documentation alone may achieve closure, but it may also spawn extensive commentary. Meetings are relatively expensive, but scheduling a meeting to drive closure reduces ambiguity by distilling commentary on the document into conclusions.

Documenting conclusions reduces ambiguity by rendering them in an objective form all stakeholders can agree on.

A few symptoms that indicate when this pattern might be useful:

  1. Endless back-and-forth in chat, bug comments, etc, which can give the impression of progress, but never resolve the issue
  2. Multiple and/or cross-functional stakeholders, which can obscure priorities
  3. Multiple people opining on a solution and/or answering questions, which can obscure ownership
  4. A problem that drags on, which can indicate it’s important, but inappropriately owned

This ties into larger discussions around project planning (e.g., managing planned vs unplanned work), and meeting efficiency (e.g., inviting stakeholders, assigning pre-work and clarifying outcomes), but the point here is just to succinctly identify an organizational pattern and when it can be helpful.

Written by Erik

August 2, 2020 at 2:52 pm

Posted in org, pattern

JShell🐚

leave a comment »

TLDR: like other REPLs, JShell provides an easy way to test Java one-liners, and, like the Rails console, a handy ad hoc CLI.

I appreciate a REPL for quickly checking the validity of small snippets. For example, I can improve the quality of my code reviews by verifying an idea works in a REPL before recommending it in a review.

My first exposure to a Java-esque REPL was the Scala REPL, which could also interpret Java. This was handy, but only easily available when Scala is installed.

When Scala wasn’t installed, I used Repl.it for Java, but this is a public site , so I need to be mindful not to use it for anything confidential, and it can take some time to load.

Recently, I learned about JShell, which is included in the JDK as of version 9.

Aside, in case you’re on a Chromebook, Google’s Cloud shell is great ad hoc terminal.

Per the JShell docs, I can start/stop the shell:

$ jshell
|  Welcome to JShell -- Version 11.0.6
|  For an introduction type: /help intro

jshell> /exit
|  Goodbye

Hello world:

jshell> System.out.println("hi")
hi

Note implicit semicolons for simple code. Return statements appear to need explicit semicolons, though, eg:

jshell> String get(){
   ...> return "s"
   ...> }
|  Error:
|  ';' expected
|  return "s"
|            ^

From the docs, I see there are “scratch variables”, which reminds me of the Scala REPL’s res variables:

jshell> 2+2
$3 ==> 4

jshell> $3
$3 ==> 4

jshell> $3 + 2
$5 ==> 6

The feedback is comparable to javac, eg:

$ cat MyMap.java
import java.util.HashMap;
import java.util.Map;

class MyMap {
        static Map<String, String> m = new HashMap<>();
        public void put(String k, String v){
                m.put(k,v);
        }
        public void get(String k){
                return m.get(k);
        }
}

$ javac MyMap.java
MyMap.java:10: error: incompatible types: unexpected return value
                return m.get(k);
                            ^
1 error
$ jshell
|  Welcome to JShell -- Version 11.0.6
|  For an introduction type: /help intro

jshell> /open MyMap.java
|  Error:
|  incompatible types: unexpected return value
|                  return m.get(k);
|                         ^------^

We can use tab completion:

jshell> List
List                 ListIterator         ListResourceBundle

Signatures:
java.util.List<E>

<press tab again to see documentation>

jshell> List.
class     copyOf(   of(

jshell> List.of("1", "2")
$3 ==> [1, 2]

jshell> $3.stream().forEach(System.out::println)
1
2

The up arrow scrolls back through history. We can also print it:

jshell> /list

   1 : List.of("1", "2")
   2 : $3.stream().forEach(n->System.out.println(n))

We can also search history, eg Ctrl + R for reverse search:

(reverse-i-search)`hi': System.out.println("hi")

Editing multi-line code is cumbersome, eg:

jshell> class Foo {
   ...> String get(){}
   ...> }
|  created class Foo

// Up arrow to edit method definition

jshell> String get(){
   ...> return "s";
   ...> }
|  created method get()

// Creates new function instead of editing Foo.get

jshell> get()
$10 ==> "s"

jshell> Foo f = new Foo();
f ==> Foo@1e88b3c

jshell> f.get();

jshell>

I’d recommend using an external editor for anything non-trivial.

JShell has an /edit command to launch an external editor, but it doesn’t appear to save the output.

jshell> /set editor vim
|  Editor set to: vim

jshell> class Foo {}
|  created class Foo

jshell> /edit Foo // add bar method to Foo
|  replaced class Foo

jshell> Foo f = new Foo()
f ==> Foo@56ac3a89

jshell> f.bar()
|  Error:
|  cannot find symbol
|    symbol:   method bar()
|  f.bar()
|  ^---^

jshell> /edit Foo // Observe bar method is undefined

I’d recommend just having an editor open in a separate terminal, and using JShell’s /open command to load the file after changes.

For folks using Google Cloud Shell, it appears to have an implicit tmux session, which makes it easy to edit in one pane and use JShell in another.

In practice, I’m guessing there’s little use for JShell when editing complex code, but it does provide a handy CLI for exploring complex code. We could have a build target, like pants repl, or a CLI for our app, like rails console.

For example, given a naive script build_to_repl.sh:

javac -d bin src/main/com/example/* \
        && jar cf bin/MyMap.jar -C bin com \
        && jshell --class-path bin/MyMap.jar

We could:

$ ./build_to_repl.sh                                                                                                                                                          
|  Welcome to JShell -- Version 11.0.6
|  For an introduction type: /help intro

jshell> import com.example.MyMap;

jshell> MyMap m = new MyMap();
m ==> com.example.MyMap@6a41eaa2

jshell> m.put("k", "v")

jshell> m.get("k")
$4 ==> "v"

Written by Erik

May 16, 2020 at 11:27 pm

Posted in tool

Tagged with ,

Architecture for Light

leave a comment »

The Illuminating Engineering Society has a couple interviews with Kim and Paul Mercier, authors of a book called Architecture for Light. The content is fascinating, touching on anthropology, physics, engineering, design and their professional experience, but a few things stood out to me in particular as overlapping with software engineering:

  1. The role of people
  2. Efficiency
  3. Multi-disciplinary planning

People

Perhaps the connection to people is more immediate in the field of lighting design than software engineering, but the two have a couple things in common that can negatively affect usability: creative purity and cost efficiency.

In software and architecture, we can design something that’s aesthetically pure, but difficult to incorporate or maintain.

One of the best tools I’ve found in project management is a focus on the customer. This can help us prioritize and avoid ethical pitfalls.

Related, healthy organizations recognize software is built by people, and people thrive in a nurturing environment. For example, Google’s research indicated psychological safety correlated strongly with high-performing teams.

Efficiency

The Merciers describe an approach to lighting that minimizes electrical usage and maximizes sales, but focusing light on products. (A common alternative is to uniformly light a space.)

They mentioned the most cost-effective solution is to turn the lights off. This reminds me of a balance in software engineering: we need to maintain everything we build, so not building minimizes maintenance cost. A similar balance comes up in SRE: we can maximize stability by minimizing changes. Obviously, doing nothing has other costs, but having all options on the table opens up opportunities, such as only invest light where a customer’s attention produces value.

The Merciers propose a method that maximizes efficiency and usability. This was an amazing observation: not too long ago, daylight was the most efficient light source; it’s still very efficient, but we need to rediscover the skills to use it.

Multi-disciplinary planning

The Merciers recommend including lighting designers from the “imagination” phase of a project rather than pulling them in later. Lighting designers can then advise on structural changes to maximize lighting efficiency. They describe the opposite as an anti-pattern: bringing specialists in late to fix problems.

This reminds me of a tension when organizing software development teams: a “waterfall” model where each specialty contributes in sequence, versus an “agile” model where teams are composed of cross-functional members.

They describe several examples in their interview, but one reminded me of complex workarounds that can arise in software: a building design that let in too much daylight. They observed a small overhang would be ideal, but because the design was already finalized the building had to be outfitted with a mechanical shade.

Written by Erik

May 10, 2020 at 3:31 pm

Posted in perspectives

Tagged with

Customer feedback

leave a comment »

Customer feedback can come through a variety of channels. Here’s a list of the ones I’ve found valuable, and practices for aggregating feedback through these channels.

Channels

Inline survey

This is an in-context prompt for feedback, eg
“Like the service? 😀 | 😐 | 😭”

Pros:

  • Relatively easy for feedback provider, resulting in broad participation
  • Confidential

Cons:

  • Limited signal
  • Hard to filter for quality participation

User research

This is when a company makes an open call for feedback.

Pros:

  • Long-form discussion
  • More control over participation than inline surveys
  • Confidential

Cons:

  • Quality of participants can vary

Partner interviews

This is when a company solicits feedback from valuable customers. I’ve found it useful to do this periodically, eg yearly.

Pros:

  • Long-form discussion
  • High-quality participants
  • Confidential

Cons:

  • Risk of focusing too closely on feedback from a small group

GitHub issues

GitHub provides a way for users of a repository to report issues.

Pros:

  • Public, so customers can pile on and follow progress
  • Intuitive for GitHub users
  • Reactions help capture sentiment
  • Easy to discover, eg via Web search
  • Enables customers to crowd-source support

Cons:

  • Specific to a repository

Slack

This is when a company maintains a public Slack channel.

Pros:

  • Good for quick Q/A
  • Enables customers to crowd-source support

Cons:

  • Can be noisy/interruptive. A support rotation can help distribute cost across team.

Stack Overflow

Stack Overflow enables people to ask questions about anything, but companies can use it’s features to support customers and collect feedback.

Pros:

  • Easy to discover, eg via Web search
  • Enables customers to crowd-source support

Aggregation

One challenge of having numerous feedback channels is distilling themes, identifying task and communicating progress. I’ve seen a few patterns to address this.

Product management

Collecting feedback from customers and using that feedback to guide planning is basic product management. Engineers may end up doing aspects of this, but they should get appropriate credit.

Dedicated support

This is when a company staffs a team to monitor all feedback channels. This team can also aggregate reports and/or work with product management to identify themes.

Support rotation

Distribute the cost of monitoring feedback sources across the team using a weekly rotation. This person will be completely distracted and may identify issues, so it pairs well with an on-call rotation.

Centralized tracking

This can be a spreadsheet or a full-featured bug tracking tool like Jira, but it’s helpful to have a single place to prioritize all issues and feature requests.

Delegated communication

If there’s a team that, say, owns a given GitHub repository, they can own the task of keeping GitHub issues for that repository synchronized with centralized tracking state.

Crowd-sourcing themes

One pattern I’ve seen work well is to split a team into groups, assign each group a subset of feedback, and ask each group to identify themes in that feedback. For example, as part of quarterly planning. This works well with large bodies of inline survey results.

Written by Erik

April 26, 2020 at 5:33 pm

Posted in product, support

Tagged with

Batch to SSTable

leave a comment »

A pattern I’ve seen a couple times for immutable data:

  1. Generate the data using a batch process
  2. Store the data in an an indexed structure (like SSTable)
  3. Expose the structure through an API

The result is a key-value store with extremely high read performance.

The first time I heard about this was Twitter’s Manhattan database. Recently, I saw the pattern again at a different company. Ilya Grigorik wrote about it several years ago in the context of log-structured data, BigTable and LevelDB.

My takeaway is: this pattern is worth considering if:

  • my current store is having issues (no need to fix what’s not broken)
  • I have heavy read traffic
  • I can tolerate latency on updates

The context of log-structured makes me think that might open a door to write access too. Twitter’s post mentions a “heavy read, light write” use-case, although it also describes use of a B-tree structure rather then a simple sorted file for that case. Grigorik’s post mentions BigTable uses a “memtable” to facilitate writes.

Note Web’s IndexedDB has a similar access pattern to SSTable. If I think about remote updates as an infrequent write, then the pattern described here might be a common use-case for Web, which might bring this around full circle: Google crawls the Web in a batch process and updates an index which is read-heavy.

Written by Erik

April 12, 2020 at 5:18 pm

Easy & advanced

leave a comment »

An API design pattern I’ve found helpful is to think about usage in two modes: easy and advanced.

This is especially helpful in debates. We may be able to accommodate a valid, but advanced, feature without cluttering the API, by housing it in an advanced subset of the API and docs.

I recently heard another phrasing of the same idea from David Poll: “Common case easy. Uncommon case possible.”

Written by Erik

February 25, 2020 at 8:13 pm

Posted in pattern

Resilience testing

leave a comment »

Google periodically devotes a week to Disaster Recovery Testing (DiRT).

Netflix automates resilience testing per-job using Chaos Monkey, and per-cluster using Simian Army.

Written by Erik

February 11, 2020 at 9:47 pm

Posted in SRE

The Manager’s Path

leave a comment »

I’m an individual contributor, but I want to better understand management’s concerns, so I’m reading Camille Fournier’s excellent The Manager’s Path. These are my notes.

Many think a neutral relationship with management is good because at least it’s not negative, but there is such a thing as a positive relationship w mgmt.

1-1 mtngs:

  • two purposes:
    • Human connection
    • Private conversation, eg feedback
  • I agree w the above two, and would add a third:
    • To ensure time w otherwise busy ppl; the junior person has the priority
  • Not for status
  • Prepare an agenda. I like a living doc, linked from the mtng invite. Items can be added and referenced any time
  • “Regular 1-1s are like oil changes; if you skip them, plan to get stranded …”
  • “Try to keep notes in a shared document” 👍 I like to link an agenda doc from the mtng invite. (Same for most recurring mtngs.)

As you become more senior, feedback decreases.

Appreciate the fact that current peers turn into future jobs.

Uncertainty
– common every 5-10 yrs
– lots of uncertainty in the world
– ultimately, we have to rely on ourselves

People aren’t good at saying what they mean in a way others can understand, so we have to listen carefully to words, and non-verbal cues indicating the person feels understood.

“Be prepared to say anything complex a few times and in diferent ways.” I’ve found such repetition frustrating in the past. It’s validating to see this advice.

Effective teams have good onboarding documents. Have new hires update the docs as their initial contribution.

“What you measure, you improve.”

Beware alpha-geek tendencies. In particular, the tendency to lecture and debate.

Mentorship skills:
– keep an open mind, since the mentee brings fresh eyes
– listen and speak their language. If you can’t hear the question being asked, you can’t provide good answers
– use the mentorship to build your network

“Tech lead is not a job for the person who wants the freedom to focus deeply on the details of her own code.”

“… the tech lead role may be held by many different stages of engineer, and may be passed from one engineer to another without either person necessarily changing his functional job level.”

“… we know from the title that it is expected to be both a technical position and a leadership role.” In other words, it’s not necessarily superlative, ie TL != best.

“The tech lead is learning to be a strong technical project manager… and [is] learning how to handle difficult management and leadership situations”

“Realistically, it is very hard to grow past senior engineer 2 without ever having acted as a tech lead, even on the individual contributor track… people skills are what we’re asking the new tech lead to stretch, more than pure technical expertise.” This stands out to me because of the tension between manager and maker modes, to use Paul Graham’s terminology.

“Being a tech lead is an exercise in influencing without authority …” Including building a psychological skill set for managing associated stresses.

“From now on … balancing is likely to be on of your core challenges.”

Currently, it feels like I’m working two jobs, manager mode during the day, and maker mode in morning and evenings. Regular, project-specific “cadence” meetings have helped reduce ad hoc discussions, fwiw.

Ah, a few lines later: “Some days you’re on maker’s schedule, and some days your on manager’s schedule…It’s very difficult to get into the groove of writing code if you’re interrupted every hour by a meeting.”

“Part of your leadership is helping the other stakeholders … respect the team’s focus and set up meeting calendars that are not overwhelming for individual contributors.” I’m very happy to see this in a book about managing thought workers.

Main roles of tech lead

  • Architect and business analyst. Design the system enough to provide estimates and ensure requirements are met
  • Project planner. The goal is to maximize parallelization
  • Developer and lead. Write code, but not too much. The goal is the project (and team development), not individual tasks

“Sometimes tech leads are tempted to go to heroics and push through obstacles themselves… [but] you should communicate the obstacle first.” I can relate with the former and appreciate the actionable latter.

“Teams often fail because they overworked themselves on a feature their product manager would have been willing to compromise on.” So, communicate.

“… most managers will expect their tech leads to continue writing as much code as before … It’s generally a pure increase in responsibility …”

The goal of a project plan is a “degree of forethought, in places where you can reasonable make predictions and plan … The plan itself … is less important than the act of planning.”

Take time to explain. No one who’s not actively working on a project should be expected to immediately know and understand project details.

Do a premortem as part of project planning. How could the system fail, and what could we do to recover?

“Having the focus to build something big yourself is a distant memory.”

The agile principles can be a healthy alternative to rigid process 👍 I think they’re great.

“… no two great teams ever look exactly alike in process, tools or work style” The best thing I’ve seen is an appreciation of experimentation and iteratively building a style that works for the current team. A basic project plan, ie list of tasks, also seems like a universal business requirement. Put another way, revisiting that plan periodically seems like a reasonable, universal starting point.

Qualities of a great tech lead:

  • Understand the architecture
  • Help build, but involve others
  • Lead decisions, but do so collaboratively
  • Communicate

“You want to encourage others on your team to learn the entire system … but you don’t always need to be self-sacrificing” There’s the need for a sense of balance again.

“Your productivity is now less important than the productivity of the whole team.” But how to improve the productivity of the team without putting on a management hat? Fournier gives an example: “Represent the team in meetings.”

Possession of communication skills differentiates successful leaders.

“Practice repeating things back to people to ensure you understand them.” I like this! I think it pairs well w advice earlier in the book to listen and observe non-verbal cues.

Communicate and listen.

I’d add that the tech lead label can also make one a focal point for questions, eg support, which can disrupt focus work. I like the pattern of having a support rotation, but depending on the company, the convention may be to simply ping the TL.

“Respect the ‘maker schedule’ for reports” 👍 As a general rule, I appreciate biasing toward contiguous meeting blocks.

Autonomy … is an important element of motivation.” I see this w external contributions too. Maximizing an integrating team’s autonomy frees them to meet their goals w minimal bottlenecks.

Written by Erik

January 31, 2020 at 9:54 am

Posted in book, notes, org, Uncategorized