Technical-organizational balance ⚖️

I recently started focusing on technical work after several months of organizational work and it’s been a lot of fun. I explicitly missed a couple things: being directly involved in a project, especially focusing deeply on a problem, and working closely with a small team.

A manager friend half-jokingly described the switch as career-limiting. He also joked we’re not paid to work only on things we enjoy. I can see his point, but there must be a balance. We are paid to do things well, and I think that’s difficult when we don’t enjoy what we’re doing, at least in part.

This recent switch has me thinking of “The Engineer/Manager Pendulum” and the follow-up “Engineering Management: The Pendulum Or The Ladder”. All the quotes below are from these essays.

If management isn’t a promotion, then returning to hands-on work isn’t a demotion, either.

I prefer the term “organizational” to “management” for the non-technical work I do because most people think of the latter as people management. I wasn’t a people manager, but I was focused on project management, spent most of my time in meetings and learned to avoid any technical work in the critical path because I had no uninterrupted time to focus on it.

A tech lead is a manager … but their first priority is achieving the task at hand, not grooming and minding the humans who work on it.

The author provides appropriate advice:

Stop writing code and engineering in the critical path

The author mentions skill erosion after two years, but I experienced something similar after just a few months. Perhaps because I needed to make room in my head for a diversity of projects, I lost the context to go deep on any one of them. My activities were described as “leadership”, but I felt like those more directly involved were actually leading in a technical sense. I can see a need for leadership that stays out of the weeds, eg to avoid the sunk cost fallacy, but my role felt like an awkward middle-ground.

I think of this dichotomy as “technical” vs “organizational”. Both are important, but difficult to do well at the same time.

Management is highly interruptive, and great engineering — where you’re learning things — requires blocking out interruptions. You can’t do these two opposite things at once.

“Maker’s Schedule, Manager’s Schedule” is an essay I think of often on that topic.

Anyway, I think this feeling of fun is positive feedback that it was time for the pendulum to swing from organizational work back to technical work.

… you can feel your skills getting rusty and your effectiveness dwindling. You owe it to yourself to figure out what makes you happy and build a portfolio of experiences that liberate you to do what you love.

I find the phrase “career growth” often refers to increased prestige, rather than fulfillment.

Try to resist the default narratives about promotions and titles and roles, they have nothing to do with what satisfies your soul.

Impact

A professional koan: of the projects I can work on, which has the most impact for the business?

Thinking about this highlights the importance of a prioritized backlog. If impact is a component of the prioritization, then the project with the highest impact is clear.

A senior manager recently commented on some faulty advice they’d heard about service development being more important than client development. They clarified that impact determines importance. We should be working on projects with impact.

“The Secret to Growing Your Engineering Career If You Don’t Want to Manage” makes several good points.

Many engineers become managers because management provides an obvious and well-defined leverage point to scale your impact.

It’s relatively easy to produce more if I can delegate work out to a team.

The less conventional paths outside of management require more creativity, and there are fewer available narratives of successful engineers outside of management for us to model ourselves after.

I can attest to this, and explore it further in “Technical-organizational balance”, but I find it surprising given the idea of parallel tech and management tracks is ostensibly common practice.

Your ability to decide where to spend your efforts to maximize your impact — what code to write, what software to build, and which business problems to tackle … You identify and solve problems that are core to the business, or you enable those around you to more effectively solve those core business problems.

Hopefully our ability to identify projects with high impact improves over time.

“How to Grow as an Engineer (Working Remotely)” also touches on impact.

It’s also not enough to just solve any problems. You need to be solving the right ones. You should constantly make sure there’s alignment between what you want and what the business needs…

MLCC: Neural Networks

I am working through Google’s Machine Learning Crash Course. The notes in this post cover the “Neural Networks” module.

Does “deep learning” imply neural networks?

The introductory video refers to “deep neural networks”, so I’m wondering what the relationship is between deep learning and neural networks.

Yes, according to Quora’s “Does deep learning always mean neural network or can include other ML techniques?”.

“To give you some context, modern Convolutional Networks contain on orders of 100 million parameters and are usually made up of approximately 10-20 layers (hence deep learning)” – https://cs231n.github.io/neural-networks-1/

“Deep Learning is simply a subset of the architectures (or templates) that employs ‘neural networks’” – https://towardsdatascience.com/intuitive-deep-learning-part-1a-introduction-to-neural-networks-aaeb3a1500df (TDS)

“Deep learning” in Google’s glossary links to “deep model”: “A type of neural network containing multiple hidden layers.”

“However, until 2006 we didn’t know how to train neural networks to surpass more traditional approaches, except for a few specialized problems. What changed in 2006 was the discovery of techniques for learning in so-called deep neural networks.” – http://neuralnetworksanddeeplearning.com/about.html

Toward’s Data Science’s “Intuitive Deep Learning Part 1a: Introduction to Neural Networks” clarifies “deep learning” is a subset of machine learning. I guess they’re both “learning”. I like the comparison of an algorithm to a recipe, and in this context, ML optimizes a recipe. Deep learning is a subset of optimization techniques.

When to use neural networks?

Small data with linear relationships → LSR

Large data with linear relationships → gradient descent

Large data with simple, nonlinear relationships → feature crosses

Large data with complex, nonlinear relationships → NN

“Neural nets will give us a way to learn nonlinear models without the use of explicit feature crosses” – https://developers.google.com/machine-learning/crash-course/introduction-to-neural-networks/playground-exercises

“Neural networks, a beautiful biologically-inspired programming paradigm which enables a computer to learn from observational data” – http://neuralnetworksanddeeplearning.com/index.html

NN “have the flexibility to model many complicated relationships between input and output”- https://towardsdatascience.com/intuitive-deep-learning-part-1a-introduction-to-neural-networks-aaeb3a1500df

“That’s not to say that neural networks aren’t good at solving simpler problems. They are. But so are many other algorithms. The complexity, resource-intensiveness and lack of interpretability in neural networks is sometimes a necessary evil, but it’s only warranted when simpler methods are inapplicable” – https://www.quora.com/What-kinds-of-machine-learning-problems-are-neural-networks-particularly-good-at-solving

Why are there multiple layers?

“each layer is effectively learning a more complex, higher-level function over the raw inputs” – https://developers.google.com/machine-learning/crash-course/introduction-to-neural-networks/anatomy

“A single-layer neural network can only be used to represent linearly separable functions … Most problems that we are interested in solving are not linearly separable.” – https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/

The universal approximation theory states that one hidden layer is sufficient for any problem – https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/

“How many hidden layers? Well if your data is linearly separable (which you often know by the time you begin coding a NN) then you don’t need any hidden layers at all. Of course, you don’t need an NN to resolve your data either, but it will still do the job.” – https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw

“One hidden layer is sufficient for the large majority of problems.” – https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw

“Even for those functions that can be learned via a sufficiently large one-hidden-layer MLP, it can be more efficient to learn it with two (or more) hidden layers” – https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/

“Multi-layer” implies at least one hidden layer: “It has an input layer that connects to the input variables, one or more hidden layers” – https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/

Chris Olah’s “Neural Networks, Manifolds and Topology”, linked from the crash course, visualizes how data sets intersecting in n dimensions may be disjoint in n + 1 dimensions, which enables a linear solution. Other than that, though, Olah’s article was over my head. Articles like TDS are more my speed.

Why are some layers called “hidden”?

“The interior layers are sometimes called “hidden layers” because they are not directly observable from the systems inputs and outputs.” – https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/

How many layers do I need?

Task 4 in the exercise recommends playing around with the hyperparameters to get a certain loss, but the combinatorial complexity makes me wonder if there’s an intuitive way to think about the role of layers and neurons. 🤔

“Regardless of the heuristics you might encounter, all answers will come back to the need for careful experimentation to see what works best for your specific dataset” – https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/

“In sum, for most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.” – https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw

“3 neurons are enough because the XOR function can be expressed as a combination of 3 half-planes (ReLU activation)” – https://developers.google.com/machine-learning/crash-course/introduction-to-neural-networks/playground-exercises Seems narrowing the problem space to ReLU enables some deterministic optimization.

“The sigmoid and hyperbolic tangent activation functions cannot be used in networks with many layers due to the vanishing gradient problem” – https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/

“use as big of a neural network as your computational budget allows, and use other regularization techniques to control overfitting” – https://cs231n.github.io/neural-networks-1/#arch

“a model with 1 neuron in the first hidden layer cannot learn a good model no matter how deep it is. This is because the output of the first layer only varies along one dimension (usually a diagonal line), which isn’t enough to model this data set well” – https://developers.google.com/machine-learning/crash-course/introduction-to-neural-networks/playground-exercises

“A single layer with more than 3 neurons has more redundancy, and thus is more likely to converge to a good model” – https://developers.google.com/machine-learning/crash-course/introduction-to-neural-networks/playground-exercises

Two hidden layers with eight neurons in the first and two in the second performed well (~0.15 loss) on repeated runs.

Heuristics from spiral solution video:

  1. Tune number of layers and nodes. Max neurons in the first layer, tapering down a couple layers to the output is a reasonable start. Each neuron takes time to train, though, so reduce total neurons if training is too slow. This is reinforced by the practice exercise, which started with two layers of 20 and 12 neurons, and then tried to reduce the number of neurons while keeping loss stable.
  2. Reduce the learning rate to smooth loss curve
  3. Add regularization to further smooth loss curve
  4. Feature engineering helps with noisy data
  5. Try different activation functions. Ultimately, tanh had the best fit
  6. Iterate from 1

Even after all this, tuning hyper parameters still seems combinatorially complex.

Activation functions

A neural net consists of layers. Nodes in the bottom layer are linear equations. Nodes in a “hidden” layer transform a linear node into a non-linear node using an “activation function”. The crash course states “any mathematical function can serve as an activation function”.

A sigmoid is an example of an activation function. I remember from the module on logistic regression (notes) that we used a sigmoid to transform a linear equation into a probability.

Why is it called a “neuron”?

The glossary definition for “neuron” is pretty good: 1) “taking in multiple input values and generating one output value”, and 2) ”The neuron calculates the output value by applying an activation function.” Aside: this reminds me of lambda architecture. I appreciate TDS clarifying neurons “often take some linear combination of the inputs”, like w1x1 + w2x2 + w3x3. I suppose this is what the glossary means by “a weighted sum of input values”.

TDS references a single image from the biological motivations section of Stanford’s CS231n, but I find both the images from that section useful for comparison.

I like TDS’ definition of a “layer” as “a “neural network” is simply made out of layers of neurons, connected in a way that the input of one layer of neuron is the output of the previous layer of neurons”. In that context, the hidden layer diagrams from the crash course makes sense.

Working agreement

I’ve had a good experience with something called a “working agreement” in the last couple teams I’ve worked on.

The main value I’ve seen is in the discussion, when everyone can have a voice regarding how a high-performing team operates. The process can help a new team gel. The resulting artifact provides a third value of resolving operational details for future reference. For example, rather than debate an issue anew, we can just reference the agreement, which all parties had a role in.

I’ve found a scaffolding helpful for structuring the conversation:

  • Core values
  • Expectations
  • Norms
  • Agreements

Core values are simple ideals and can be aspirational. For example, “we provide each other psychological safety”, or “we do our best work together.” An artificial constraint of, say, five values can help motivate discussion, and improve accessibility of the resulting list.

Expectations provide an opportunity to state explicitly what we might be assuming. For example, “I assume best intentions” or “I assume folks have the big picture in mind, even for small changes.”

Norms provide an opportunity to express preferences. For example, “(given designing, writing and reviewing code requires focused attention) I benefit from no-meeting blocks”, or “I read emails, but aggressively purge (so I don’t mind a follow-up a day later).” Note that norms are becoming more operational.

Agreements build on and finalize the preceding sections. For example, “(To work efficiently and respectfully) we’ll respond to code review requests within a day, or communicate otherwise” or “We’ll use a single email adress for the team, differentiated by suffix (so it’s easy to find all emails, but it’s also possible to filter).”

It can take some time to build rapport, especially if the team is new, so budget an hour for discussion, and likely an additional hour on another day to finalize. It’s helpful to share the scaffolding in advance, so folks can start adding ideas. Having a different person lead development of each section improves participation.

These teams have a stated goal to review the agreement periodically, but in practice I’ve found on-demand is sufficient.

Norvig’s summary of ML for software engineers

Peter Norvig summarized the value of ML from a software engineering perspective in his “Introduction to Machine Learning” for Google’s Machine Learning Crash Course:

First, it gives you a tool to reduce the time you spend programming … Second, it will allow you to customize your products, making them better for specific groups of people … And third, machine learning lets you solve problems that you, as a programmer, have no idea how to do by hand.

From my perspective, the first two can be rephrased as:

  1. Models add a new dimension to code reuse
  2. For a class of problems, training models scales better than hand-writing code

There’s also a fourth point linked from the bottom of the intro:

Rule #1: Don’t be afraid to launch a product without machine learning

That fourth point reminds me of the “build” vs “grow” domains – until we’ve built a product that lots of people find useful, statistics-based growth tools, like large-scale AB testing, can be relatively high-cost, low-value.We might even say such optimizations only make sense once we have more users than can be efficiently contacted directly. Put another way, if we only have one user, and she says she only wants to see articles about sports, we don’t need ML to predict her interests.

I think about these four points a lot, almost like a koan. They provide a helpful anchor as I try to distill a large amount of theory into tools I can apply to the problems I’m familiar with.

“Beautiful Future: How Deschutes Uses Artificial Intelligence & Machine Learning to Brew Better Beer”

Craft Beer and Brewing’s article “Beautiful Future: How Deschutes Uses Artificial Intelligence & Machine Learning to Brew Better Beer” describes an intuitive application of ML. Deschutes brewery wanted to more accurately predict when a given fermentation process would complete. The problem statement is simple:

Produce the same amount of beer in less time, while maintaining or improving the quality of the beer along the way, and you’ll have more resources for the intentional play that leads to new beers that drinkers love.

I like the explicit recognition that reducing toil frees time for more valuable activities. This is reiterated later:

Most beer consumers aren’t concerned with how efficiently or cost-effectively a brewery makes their beer—they want high-quality beer, and they want new and exciting beers.

Fermentation sounds like a relatively simple curve to plot. It’s easy to imagine manually monitoring something like sugar content vs time, and then using that data to train a model.

Brewers now trust automation to act on the predictions:

Today, cellar operators at Deschutes have such a high level of confidence in the algorithm that they typically allow the software to trigger next steps in the brewing process.

The automation is also easy to imagine. Deschutes’ Brewery Pi project targets Raspberry Pi, which I can see being used to drive hardware to adjust temperature, add nutrients, drain a fermentation vessel, etc. I really like how Deschutes made the code open-source 🍻

MLCC: Regularization for sparsity

I am working through Google’s Machine Learning Crash Course. The notes in this post cover the “Regularization for Sparsity” module.

Best-practice: if you’re overfitting, you want to regularize.

“Convex Optimization” by Boyd and Vandenberghe, linked from multiple glossary entries, touches on many of the points made by the crash course:

  • “A problem is sparse if each constraint function depends on only a small number of the variables”
  • “Like least-squares or linear programming, there are very effective algorithms that can reliably and efficiently solve even large convex problems”, which would explain why gradient descent is a tool we use
  • Regularization is when “extra terms are added to the cost function”
  • “If the problem is sparse, or has some other exploitable structure, we can often solve problems with tens or hundreds of thousands of variables and constraint”, so it would seem performance is another motivation for regularization

Ideally, we could perform L0 normalization, but that’s non-convex, and so, NP-hard (slide 7). (I like Math is Fun’s NP-complete page🙂 As noted wrt gradient descent, we need a convex loss curve to optimize. L1 approximates L0 and is easy to compute.

Quora provides a couple intuitive explanations for L1 and L2 norms: “L2 norm there yields Euclidean distance … The L1 norm gives rise to what can be referred to as the “taxi-cab” distance”

Rorasa’s blog states “Norm may come in many forms and many names, including these popular name: Euclidean distance, Mean-squared Error, etc … Because the lack of l0-norm’s mathematical representation, l0-minimisation is regarded by computer scientist as an NP-hard problem, simply says that it’s too complex and almost impossible to solve. In many case, l0-minimisation problem is relaxed to be higher-order norm problem such as l1-minimisation and l2-minimisation.”

The glossary summarizes:

  • L1 regularization “penalizes weights in proportion to the sum of the absolute values of the weights. In models relying on sparse features, L1 regularization helps drive the weights of irrelevant or barely relevant features to exactly 0”
  • L2 regularization “penalizes weights in proportion to the sum of the squares of the weights. L2 regularization helps drive outlier weights (those with high positive or low negative values) closer to 0 but not quite to 0”

MLCC: Logistic regression

I am working through Google’s Machine Learning Crash Course. The notes in this post cover the “Logistic Regression” module.

“Logistic regression” generates a probability (a value between 0 and 1). It’s also very efficient.

Note the glossary defines logistic regression as a classification model, which is weird since it has “regression” in the name. I suspect this is explained by “You can interpret the value between 0 and 1 in either of the following two ways: … a binary classification problem … As a value to be compared against a classification threshold …”

The “sigmoid” function, aka “logistic” function/transform, produces a bounded value between 0 and 1.

Note the sigmoid function is just y = 1 / 1 + e ^ - 𝞼 where 𝞼 is our usual linear equation. I suppose we’re transforming the linear output into a logistic form.

Regularization (notes) is important in logistic regression. “Without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions”, esp L2 regularization and stopping early.

The “logit”, aka “log-odds”, function is the inverse of the logistic function.

The loss function for logistic regression is “log loss”.

MLCC: Classification

I am working through Google’s Machine Learning Crash Course. The notes in this post cover the “Classification” module.

New metrics for evaluating classification performance:

  • Accuracy
  • Precision
  • Recall
  • ROC
  • AUC

Accuracy

“Accuracy” simply measures percentage of correct predictions.

It fails on class-imbalance, aka “skewed class”, problems, though. Neptune AI states is bluntly: “You shouldn’t use accuracy on imbalanced problems.” Heuristic: is the percent accuracy > the imbalance? For example, if a population is 99% disease-free, an accuracy of 99% requires no intelligence. This is called the “accuracy paradox”. Precision and recall are better suited to class-imbalance problems.

Tip: calculate odds independently if possible to compare with accuracy.

Confusion matrix

A “confusion matrix”, aka “classification matrix”, quantifies predicted vs actual outcomes, which is useful for evaluating model performance.

A false positive is a “type one” error. A false negative is a “type two” error. When the cost of error is high, type two must be minimized. In other words, when the cost of error is high, maximize recall.

Precision and recall

Andrew Ng’s “Lecture 11.4 — Machine Learning System Design | Trading Off Precision And Recall” provides a helpful phrasing:

  • Precision = true positive / predicted positive
  • Recall = true positive / actual positive

Regarding the accuracy paradox, if a model simply predicts negative all the time (eg because 99% of email isn’t spam), it will fail recall and precision because it never has a true positive.

Wikipedia makes a point: “It is trivial to achieve recall of 100% by returning all documents in response to any query”

Precision and recall are important, and in tension. Classification depends on a “threshold”. Increasing the threshold increases precision, but decreases recall. Wikipedia uses surgery for a brain tumor to illustrate: a conservative approach increases the risk of false negative; an aggressive approach increases risk of false positive. Plotting the “precision-recall curve” can also help demonstrate the relationship, as demonstrated by Andrew Ng.

Wikipedia has a nice visualization differentiating precision and recall:

ROC and AUC

The “ROC curve” helps identify the best threshold.

“AUC” compares ROCs, helping identify the best model.

StatQuest’s “ROC and AUC, Clearly Explained!” states precision is a better metric than the false positive rate for class imbalance problems because it doesn’t take true negatives into account.

Keras gives us AUC for a model, but what’s the corresponding threshold? The crash course clarifies: “AUC is classification-threshold-invariant. It measures the quality of the model’s predictions irrespective of what classification threshold is chosen.” Ok, then why use anything but AUC? Neptune AI summarizes: “… use it when you care equally about positive and negative classes.”

Prediction bias

Seems like this is another way of quantifying model performance. If we know a probability of occurrence and the model produces a significantly different probability, that indicates something’s amiss.

The formal definition is: average predicted occurrence – average actual occurrence. There’s a helpful note that a model simply returning the average occurrence would have zero prediction bias, but would still be a bad model.

The crash course gives a few causes for bias. StatQuest’s “Machine Learning Fundamentals: Bias and Variance” adds another: the inability of a ML algorithm to capture the true relationship between features and labels, eg linear regression trying to capture a curved relationship.

Fix prediction bias in the model, rather than adjusting the model output.

Interesting clarification that predicted values are a probability range, but actual values are discrete, so we need to segment values and average them to make a comparison.