Software engineering notes

Archive for the ‘tool’ Category

JShell๐Ÿš

leave a comment »

TLDR: like other REPLs, JShell provides an easy way to test Java one-liners, and, like the Rails console, a handy ad hoc CLI.

I appreciate a REPL for quickly checking the validity of small snippets. For example, I can improve the quality of my code reviews by verifying an idea works in a REPL before recommending it in a review.

My first exposure to a Java-esque REPL was the Scala REPL, which could also interpret Java. This was handy, but only easily available when Scala is installed.

When Scala wasn’t installed, I used Repl.it for Java, but this is a public site , so I need to be mindful not to use it for anything confidential, and it can take some time to load.

Recently, I learned about JShell, which is included in the JDK as of version 9.

Aside, in case you’re on a Chromebook, Google’s Cloud shell is great ad hoc terminal.

Per the JShell docs, I can start/stop the shell:

$ jshell
|  Welcome to JShell -- Version 11.0.6
|  For an introduction type: /help intro

jshell> /exit
|  Goodbye

Hello world:

jshell> System.out.println("hi")
hi

Note implicit semicolons for simple code. Return statements appear to need explicit semicolons, though, eg:

jshell> String get(){
   ...> return "s"
   ...> }
|  Error:
|  ';' expected
|  return "s"
|            ^

From the docs, I see there are “scratch variables”, which reminds me of the Scala REPL’s res variables:

jshell> 2+2
$3 ==> 4

jshell> $3
$3 ==> 4

jshell> $3 + 2
$5 ==> 6

The feedback is comparable to javac, eg:

$ cat MyMap.java
import java.util.HashMap;
import java.util.Map;

class MyMap {
        static Map<String, String> m = new HashMap<>();
        public void put(String k, String v){
                m.put(k,v);
        }
        public void get(String k){
                return m.get(k);
        }
}

$ javac MyMap.java
MyMap.java:10: error: incompatible types: unexpected return value
                return m.get(k);
                            ^
1 error
$ jshell
|  Welcome to JShell -- Version 11.0.6
|  For an introduction type: /help intro

jshell> /open MyMap.java
|  Error:
|  incompatible types: unexpected return value
|                  return m.get(k);
|                         ^------^

We can use tab completion:

jshell> List
List                 ListIterator         ListResourceBundle

Signatures:
java.util.List<E>

<press tab again to see documentation>

jshell> List.
class     copyOf(   of(

jshell> List.of("1", "2")
$3 ==> [1, 2]

jshell> $3.stream().forEach(System.out::println)
1
2

The up arrow scrolls back through history. We can also print it:

jshell> /list

   1 : List.of("1", "2")
   2 : $3.stream().forEach(n->System.out.println(n))

We can also search history, eg Ctrl + R for reverse search:

(reverse-i-search)`hi': System.out.println("hi")

Editing multi-line code is cumbersome, eg:

jshell> class Foo {
   ...> String get(){}
   ...> }
|  created class Foo

// Up arrow to edit method definition

jshell> String get(){
   ...> return "s";
   ...> }
|  created method get()

// Creates new function instead of editing Foo.get

jshell> get()
$10 ==> "s"

jshell> Foo f = new Foo();
f ==> Foo@1e88b3c

jshell> f.get();

jshell>

I’d recommend using an external editor for anything non-trivial.

JShell has an /edit command to launch an external editor, but it doesn’t appear to save the output.

jshell> /set editor vim
|  Editor set to: vim

jshell> class Foo {}
|  created class Foo

jshell> /edit Foo // add bar method to Foo
|  replaced class Foo

jshell> Foo f = new Foo()
f ==> Foo@56ac3a89

jshell> f.bar()
|  Error:
|  cannot find symbol
|    symbol:   method bar()
|  f.bar()
|  ^---^

jshell> /edit Foo // Observe bar method is undefined

I’d recommend just having an editor open in a separate terminal, and using JShell’s /open command to load the file after changes.

For folks using Google Cloud Shell, it appears to have an implicit tmux session, which makes it easy to edit in one pane and use JShell in another.

In practice, I’m guessing there’s little use for JShell when editing complex code, but it does provide a handy CLI for exploring complex code. We could have a build target, like pants repl, or a CLI for our app, like rails console.

For example, given a naive script build_to_repl.sh:

javac -d bin src/main/com/example/* \
        && jar cf bin/MyMap.jar -C bin com \
        && jshell --class-path bin/MyMap.jar

We could:

$ ./build_to_repl.sh                                                                                                                                                          
|  Welcome to JShell -- Version 11.0.6
|  For an introduction type: /help intro

jshell> import com.example.MyMap;

jshell> MyMap m = new MyMap();
m ==> com.example.MyMap@6a41eaa2

jshell> m.put("k", "v")

jshell> m.get("k")
$4 ==> "v"

Written by Erik

May 16, 2020 at 11:27 pm

Posted in tool

Tagged with ,

A nice data mart ๐Ÿช

leave a comment »

The data mart is a subset of the data warehouse and is usually oriented to a specific business line or team

https://en.m.wikipedia.org/wiki/Data_mart

I don’t have a lot of experience with data marts, but I recently met one that seems nice and simple.

The store benefits from a few other abstractions:

  1. a service that just ingests and persists client events
  2. a query abstraction, like Hive
  3. trustworthy authentication and list membership infra

Given these, the store in question simplifies the process of utilizing data by abstracting a few common requirements:

  1. a simple config DSL specifies which query to run, the frequency to run it, the output table, deletion conditions, etc. Specifying config via files enables use of common source control tools.
  2. three predefined processing stages (raw-to-normalized, normalized-to-problem-specific, problem-specific-to-view-specific). New event sources, aggregations and views can be independently defined by adding new config files.
  3. common styling and libraries for data visualization
  4. access is generalized to a few tiers of increasing restriction, eg team, division, company. The lowest level might be freely granted to teams for their own business intelligence, and the highest level restricted to executives for making revenue-specific decisions.

In retrospect, this seems pretty straightforward. I’m remembering a tool from another team (basically Hadoop + Rails + D3) that had the same goals, but didn’t have the query, scheduling or ACL abstractions underneath. It was replaced by an external tool that was terrible to the point of being unusable, but more secure. Eventually, we dumped normalized data in a columnar store that was also secure and easier to use for our team’s business intelligence, but would’ve been insufficient for things like periodically updating charts. I guess it’s the combination of data store features and supporting infra that makes the magic happen.

Written by Erik

October 14, 2019 at 8:52 pm

Posted in pattern, tool

Praise for a task tracker ๐Ÿ“‰

leave a comment »

I just wrapped up a three month project in a new tech stack. From the beginning to the end, I was regularly asked: “Will this be done in time?” I was new to the tech, so it was difficult to estimate. The best I could do was document tasks as they revealed themselves, and then point at the percentage of tasks complete as a measure of progress.

For the first couple months, I added more tasks than I closed. The scope of the project seemed like it would grow unbounded, but then it leveled off. Today, I closed the last task. Amazing, and it wasn’t that hard, because we at least had an agreed-upon task tracker. I just needed to feed it.

Some features I like:

  • Using the same system used for tracking bugs. This frees folks to use whatever tracker they prefer. One colleague prefers a simple spreadsheet of tasks in descending priority, for example. It also enabled folks to use familiar bug tracking tools, like subscribing to updates, assigning ownership, commenting, etc, to communicate about tasks
  • At least trying to present a burn-down chart, though it doesn’t yet know how to work with nested milestones. A burn-down chart is the most accurate tool I’ve found for estimation; even if we can’t estimate the completion time, we can say the job will be done when the number of tasks drops to zero, and easily follow along on the chart.
  • Organizing tasks in a variety of ways, notably, differentiating organization by project from organization by week
  • Organizing tasks by blocking relationship. In other words, enabling me to specify this task depends on that task. Identifying branch nodes in this tree as milestones is helpful for maintaining a sense of momentum

Some usage patterns I like:

  • Presenting the tracker in a periodic (agile, 4DX) cadence meeting and using it to structure discussion. Avoiding tracker maintenance was helpful for maintaining meeting pace
  • Agreeing on one approach to tracking helped focus attention and minimize maintenance cost, though an alternative popped in and out mid-project, and the project launch used a different tracker

Written by Erik

September 25, 2019 at 10:01 pm

Posted in org, tool

View

leave a comment »

The joy of top-down rendering.

Problem

I want to present data, ideally as view = render(data).

Solution

I really like the view mechanics provided by choo/yo-yo/bel.

const html = require('bel')
const nanobus = require('nanobus')
const yo = require('yo-yo')

const bus = nanobus()
const render = yo.update.bind(yo, document.body)
const emit = bus.emit.bind(bus)

bus.on('change', (name) => {
  const state = {}
  state.name = name.toUpperCase()
  render(view(state, emit))
})

function view(state, emit){
  return html`
    <body>
      Hello, <input value="${state.name}" placeholder="name" onkeyup=${onKeyUp}>
    </body>
  `
  function onKeyUp(e){
    emit('change', e.target.value)
  }
}

Written by Erik

October 4, 2017 at 6:13 pm

Posted in pattern, tool

Tagged with , , , , ,

Object path

leave a comment »

Problem

I want to reduce conditional assignment when setting nested keys in an object, ideally:

{a:{b:{c:value}}} = set(a/b/c, value)

This is handy for data manipulation and abstracting path-based tools like LevelDB and Firebase Realtime Database.

Solution

Use object-path or lodash’s set/get.

Note: the tools mentioned above interpret numeric path segments as array indices, which may cause unexpected results when inserting arbitrary values, eg:

set(store, 'users.5.name', 'Kwan') // store.users.length --> 6

If this is an issue, consider:

function set(obj, path, val){
  path.split('/').reduce((parent, key, i, keys) => {
    if (typeof parent[key] != 'object') {
      if (i === keys.length - 1) {
        parent[key] = val
      } else {
        parent[key] = {}
      }
    }
    return parent[key]
  }, obj)
}
function get(obj, path){
  return path.split('/').reduce((parent, key) => {
    return typeof parent === 'object' ? parent[key] : undefined
  }, obj)
}

Examples

Inverting an object:

const posts = {1: {tags: {sports: true, news: true}}, 2: {tags: {news: true}}}
const byTag = {}
Object.entries(posts).forEach(([id, post]) => {
  Object.keys(post.tags).forEach(tag => {
    set(byTag, `${tag}/${id}`, true)
  })
})
// byTag --> { sports: { '1': true }, news: { '1': true, '2': true } }

Creating and querying a prefix tree:

const flatten = require('flat')

// populate tree
const emojis = {
  '๐Ÿ™‚': 'smile',
  '๐Ÿ˜€': 'grinning',
  '๐Ÿ˜': 'grin'
}
const tree = {}
Object.entries(emojis).forEach(([emoji, name]) => {
  let path = name.split('').join('/') + '/' + emoji
  set(tree, path, true)
})

// lookup prefix
const prefix = 'g'
const path = prefix.split('').join('/')
const subtree = get(tree, path) || {}
const matches = Object.entries(flatten(subtree)).map(([key, val]) => {
  return key.slice(-2)
})
console.log(matches) // --> ["๐Ÿ˜€", "๐Ÿ˜"]

Written by Erik

October 3, 2017 at 6:01 pm

Posted in pattern, tool

Tagged with , , , ,

Adventures in blogging ๐Ÿ“

leave a comment »

Motivation

I like the idea of shared learning and working together as a team, and in a way that’s independent of employment.

The desire for independence takes a couple forms:

  1. Minimizing loss of work due to employment boundaries. For example, learning documented on an internal blog may be lost when changing roles.
  2. Maximizing opportunities that would otherwise be constrained by company goals. For example, exploring professional growth areas that may never be a company priority.

Prakhar pointed me at Nathan Marz’s post on blogging, which provides excellent motivation specific to blogging.

Megha shared her experience at Write/Speak/Code, which recommends professional writing as an essential aspect of career development. I like Write/Speak/Code’s recommendation to seek feedback from trusted sources. Inline comments and explicit comment permission, like in Google Docs, would be ideal.

(Write/Speak/Code also brought Open Source Misfeasance to my awareness ๐Ÿ‘ esp the slide “open source is like being an adult – it seems magical until you realize nobody knows what the hell they’re doing.” ๐Ÿ™‚

I’m inspired by the meta-knowledge community on Github.

A New Yorker article on Jonathan Ledgard mentions he “carried two notebooksโ€”red for his reporting notes, blue for thoughts and observations to use in fictionโ€ฆ”

Current approach

A simple note app (currently Google Keep) to capture and access thoughts with minimal fuss.

WordPress for blog hosting. It feels relatively old-school, but it’s stable, feature-rich and has a large community. Signal v Noise, Joel Spolsky and Bill Gates use it, so I’m in good company. A few features I appreciate in particular:

  • Full-text search, which is especially useful for finding and updating posts
  • Navigation by tag and category
  • A convenient native app
  • Easy navigation from viewing to editing
  • Basic stats and “likes” aren’t the reason I’m writing, but are reassuring

Separate professional from personal content, to facilitate usage of content at work, hence the desire for multi-account support.

Alternatives explored

  • Medium is clean and modern, but the monetization strategy of charging readers seems relatively high-friction, and it doesn’t have multi-account support
  • Write.as is clean, and has a sustainable business model and an interesting tie-in to content federation, but it’s missing comments, search, etc
  • Github Pages is simple, but relatively inconvenient for frequent content management
  • Forestry CMS to manage Github Pages is an improvement, but I still want for search, comments, etc
  • Siteleaf is good, but doesn’t provide preview in free mode, and renames files according to date front-matter (after import)
  • I tried Prose.io in the past, but saving content was flaky
  • jekyll-admin seems interesting if/when it’s supported by Github Pages

Written by Erik

September 27, 2017 at 4:53 pm

Posted in tool

Tagged with , , , , , , ,

Source control

leave a comment »

Problem

I want to snapshot incremental progress and designate a working version.

I want a tool that’s widely available and easy to reason about.

Solution

Use Git. [1]

Use a single repo [2] until it’s unwieldy [3].

[1]: Some folks (Facebook, Google) argue Murcurial is preferable because it provides an extensible API, but in my experience, git has been sufficient for non-trivial work with tens of eng.

[2]: Looking forward to things like GVFS for scaling.

[3]: Managing multiple interdependent repos has a cost, which is why I trend to a monorepo, but I’ve also worked w a monorepo that took ages to sync. I’d be curious to see someone experiment w tools for managing several repos as an explicit alternative to a monorepo.

Written by Erik

September 19, 2017 at 4:13 pm

Posted in tool

Tagged with , , ,

Solarized code highlighting for Jekyll

leave a comment »

Problem

I’d like to improve readability of the code on this blog by highlighting syntax, solarized ideally.

Solution

Jekyll makes syntax highlighting easy, and the following gist makes solarization easy too:

Written by Erik

August 17, 2017 at 4:07 pm

Posted in tool

Tagged with ,

The weird world of observable keys ๐Ÿ”‘๐Ÿ‘€

leave a comment »

The get that keeps on getting.

Solution

Given a bus:

import Foundation
struct Value : Event {
  let key: String
  let val: Any?
}
protocol Store {
  func get(_ key: String)
  func set(_ key: String, _ val: Any?)
}
class LocalStore : Store {
  let db: UserDefaults
  let bus: Bus
  init(db: UserDefaults, bus: Bus){
    self.db = db
    self.bus = bus
  }
  func get(_ key: String) {
    let val = db.object(forKey: key)
    bus.pub(Value(key: key, val: val))
  }
  func set(_ key: String, _ val: Any?){
    db.set(val, forKey: key)
    bus.pub(Value(key: key, val: val))
  }
}
let bus = Bus()
bus.sub(StdoutSubscriber())
let local = LocalStore(db: UserDefaults.standard, bus: bus)
local.set("foo", "bar")
local.get("foo")

A couple features I like:

  • The bus provides a consistent interface for adding and removing subscriptions
  • There’s a straightforward get method to flush a value into the bus

Problem

If rationalizing events bubbling up from UI and disparate data sources is challenging, normalizing to a single bus interface may be helpful.

Related

Written by Erik

August 15, 2017 at 3:47 pm

Posted in tool

Tagged with , ,

A case for programmatic views in iOS

leave a comment »

Context

If you’re new to ios, building ui in xcode, and frustrated with inability to search for help effectively.

Suggestion

Layout your ui programmatically. I’ve found PureLayout helpful for this.

Benefits

  1. Search based on concrete syntax (as opposed to search for the button that looks like … in xcode version …)
  2. Reference other people’s ui, eg PureLayout’s demos, without having to load it in xcode
  3. Comment on configuration (credit to Yong for noticing this)
  4. Code-review changes
  5. Reduce cognitive load because you only have to understand autolayout, not autolayout + interface builder

Caveat

Interface builder does a lot (I recently learned from Riz that LaunchKit used compiled storyboards for dynamic ui!), and is Apple’s recommended approach. I suspect I’ll have a better experience with it once I have more ui experience.

Written by Erik

July 29, 2017 at 1:10 pm

Posted in tool

Tagged with , ,