Thomas Countz

Utilizing Docker for Bundling in a Specific Ruby Environment

2023-07-06T00:00:00+00:00

One of the tasks we often find ourselves completing on Zendesk’s Core Ruby Engineering team is helping to keep a variety of projects up to date. This can involve running bundle update across dozens of repositories, each potentially requiring a different Ruby environment. Although we’ve developed tooling to accomplish this rather efficiently, there are often edge cases that require us to work in each repository locally. However, installing so many different versions of Ruby locally can be cumbersome and time-consuming.

In such scenarios, Ruby Docker images provide a lighter weight solution compared to running a project’s Dockerfile (which can contain dozens of dependencies) or spinning up our Kubernetes-based development tooling (which gives us access to an entire structured environment of all of Zendesk). Individual Ruby images allow us to run commands like bundle update in a specific Ruby environment, such as JRuby, without needing to install that Ruby version locally.

Let’s take an example of running bundle install in a project that requires JRuby. We’ll use JRuby as an example as it’s not always as straightforward to install as CRuby.

First (assuming you hav Docker running locally), pull the Docker image from DockerHub for the Ruby version you need.

docker pull jruby:latest

Next, we’ll run the bundle install command within a Docker container using the JRuby image. If your project uses a private gem repository, you might need to pass in credentials. If you have these credentials stored in an environment variable locally, you can pass that to the Docker container. Docker will use the value from the local environment:

docker run --rm --volume "$(pwd)":/usr/src/app --workdir /usr/src/app --env PRIVATE_GEM_REPO_CREDS jruby:latest bundle update my_gem --conservative

Here’s a breakdown of the command:

--rm (-rm for short): This option removes the container after it exits.
--volume "$(pwd)":/usr/src/app (-v for short): This mounts the current directory (assumed to be your project directory) to /usr/src/app in the container.
--workdir /usr/src/app (-w for short): This sets the working directory inside the container.
--env PRIVATE_GEM_REPO_CREDS (-e for short): This passes the PRIVATE_GEM_REPO_CREDS environment variable from your local environment to the Docker container. Replace PRIVATE_GEM_REPO_CREDS with your actual environment variable that holds the credentials for your private gem repository.
jruby:latest: This is the Docker image we’re using.
bundle update my_gem --conservative: This is the command we’re running inside the container. The --conservative flag with instruct Bundler to update as few gems as necessary.

This command runs bundle update in the context of the Docker container, using your stored credentials for authentication and modifying your local directory. Your Gemfile.lock should be modified (if applicable) and any vendor-ized gems will be updated. You can now commit and push the changes.

Docker’s flexibility allows us to run bundle update in any Ruby environment, including but not limited to JRuby, without local installation, while also handling authentication via environment variables. This approach can be applied to similar scenarios where specific environments are required for package installation. It’s a streamlined, efficient alternative to more heavyweight solutions.

A Thought on Practice

2022-02-09T00:00:00+00:00

tl;dr Do a thing you already know how to do.

Our favorite psychologist, Mihaly Csikszentmihalyi, has taught us that Flow maneuvers between anxiety and boredom. I like to say that I err towards “anxiety” because I’m ambitious, I like to learn new things, I like a challenge. But, I like a bit of boredom too! The boredom range is where I find inspiration, curiosity, and discovery.

So, where is the line between the two? When I think of boring work, I think of work that does not require gaining new knowledge or skills, and therefore does not pose a challenge. For a Ruby on Rails engineer like me, this could be setting up a new CRUD (create, read, update, and delete) operation, TDD-ing (test-driven developing) a helper method, or changing a button from blue to red. These tasks, being tasks I already know how to do, ebb towards boring. Therefore, if given these tasks, I might find myself in boredom.

But, do I really know how to do these tasks?

It’s been so long since I’ve done them—and I’ve spent so much time scoffing at them—that figuring out how to edit CSS, writing unit tests for simple functions, and building a new simple Rails app are nearly foreign to me!

rails new leaves me with analysis paralysis, stubbing a method call fills me with doubt, and working with CSS might as well be working with live electrical wires!

This is because I’m out of practice.

I don’t spend time doing the things I think I know how to do because I’m so focused on learning new things—being pulled towards a challenge and away from “boredom.”

The things that I’ve written off as boring are ripe with new potential for growth and discovery.

That old rails new option that I’ve never really looked into is just as exciting as a new Rails feature! Building a simple command-line Ruby program is a brand new experience given how much I’ve learned since my first tic-tac-toe game! Front-end engineering is almost an entirely new discipline compared to when I was first starting out!

Instead of chasing the dopamine rush of buying new toys, I’m cracking open my old toy box with a new appreciation.

Københavns Luftkvalitet og Cykling

2021-08-02T00:00:00+00:00

Dataset retreived from: Utrecht University & Google, 2021, via Google Environmental Insights Explorer (August 2021)

The Jupyter notebook used to do this analysis is available here as a Binder and here as a Gist for you to explore!

Introduction

I used to be a bicycle taxi driver, and I remember coming home after a day of work covered in soot from riding along the roads all day. It was a gross, yet kind of fascinating, experience to realize that not only was this stuff in the air covering my skin, I was also spending the day breathing it in.

Due to global warming, our consciousness is ever-focused on air pollution; especially within our cities. Many cities are, therefore, attempting to reduce emissions by reducing the amount of cars on the road and increasing the amount of bikes.

This is old news for cities like Copenhagen, Denmark. Their strong bike culture is only surpassed by their comprehensive bike infrastructure. So when The City of Copenhagen collaborated with Utrecht University and Google to map the city’s air pollution, I was eager to get my hands on the data.

Between November 2018 and February 2020, Google strapped three different air quality sensors on their Street View vehicles and took thousands of measurements throughout the city. Today, we’re going to take a look at the data and start to answer the question: what’s the healthiest bike route between A and B, based on air quality?

Data Ingestion & Initial Aggregation

The data can be found on Open Data DK’s website, and we can use the download link there to pull the .geojson file directly into our project.

url = "https://kkkortdata.spatialsuite.dk/airview/CAV_25May2021.geojson"
remote_data = urllib.request.urlopen(url)
raw_data = gpd.read_file(remote_data)

Let’s take a preview of the data.

raw_data.head()

FID	Shape_Leng	ROAD_FID	Mixed_NO2	Mixed_UFP	Mixed_BC	geometry
0	58.821698	1	-999999	-999999	-999999.0	LINESTRING (12.57828 55.69392, 12.57752 55.69423)
1	31.898519	2	10	18500	0.8	LINESTRING (12.62463 55.63461, 12.62473 55.63433)
2	36.276259	3	9	12800	0.8	LINESTRING (12.58604 55.60184, 12.58595 55.602…
3	40.416066	4	9	10700	0.7	LINESTRING (12.58629 55.59317, 12.58569 55.59305)
4	32.734397	5	13	14500	1.1	LINESTRING (12.58135 55.59200, 12.58130 55.59171)

The measurements have the following units:

Black Carbon: μg/m³
Nitrogen Dioxide: μg/m³
Ultrafine Particles: particles/cm³

The Open Data DK site mentions that “some missing data points are listed in datasets as -999999”, so we’ll go ahead and remove any rows in our data that contain this value.

Note that we could change this value to, let’s say, 0, however, 0 will mean something very specific when we look at the distribution later. We could also be sure not to remove the entire row if only one data point is missing, but for my initial analysis, I’m opting to more simply get rid of everything to keep all Series in the DataFrame the same dimension.

cleaned_data = raw_data.drop(raw_data[(raw_data.Mixed_BC == -999999) | (raw_data.Mixed_NO2 == -999999) | (raw_data.Mixed_UFP == -999999)].index)

Now we can look at the correlation between each measurement. I assume their’s a strong correlation: “where there’s high concentration of air pollution of type ‘x’, there’s probably a high concentration of air pollution of type ‘y’”, but it’s always good to check assumptions.

cleaned_data[["Mixed_BC", "Mixed_NO2", "Mixed_UFP"]].corr()

—	Mixed_BC	Mixed_NO2	Mixed_UFP
Mixed_BC	1.000000	0.938192	0.790519
Mixed_NO2	0.938192	1.000000	0.702058
Mixed_UFP	0.790519	0.702058	1.000000

And we can take a look at another statistic that might be interesting, the overall mean:

cleaned_data[["Mixed_NO2", "Mixed_UFP", "Mixed_BC"]].mean()

Mixed_NO2       17.572950
Mixed_UFP    14280.222977
Mixed_BC         1.131806
dtype: float64

It’s worth noting that these averages are similar to the ones published on Google’s project site, which gives me a pretty good confidence that I’ve loaded the data correctly. The difference between my results and theres may be that fact that I likely dropped some data while doing cleanup.

Google also shows the distributions of the amounts of pollutant per road section, so we can go ahead and look at that too.

fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(33, 7))

ax[0].title.set_text("Black Carbon")
cleaned_data.Mixed_BC.plot.hist(bins=25, color="xkcd:moss green", rwidth=0.9, ax=ax[0], legend=True)

ax[1].title.set_text("Nitrogen Dioxide")
cleaned_data.Mixed_NO2.plot.hist(bins=25, color="xkcd:moss green", rwidth=0.9, ax=ax[1], legend=True)

ax[2].title.set_text("Ultrafine Particles")
cleaned_data.Mixed_UFP.plot.hist(bins=25, color="xkcd:moss green", rwidth=0.9, ax=ax[2], legend=True)

Now for the interesting bit: let’s take a look at a map of the data. GeoPandas quickly let’s you plot a Series or DataFrame containing geometries that it created based on the geojson data we imported.

fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(33, 21))

ax[0].title.set_text("Black Carbon")
ax[0].set_facecolor("xkcd:grey")
cleaned_data.plot(column="Mixed_BC", ax=ax[0], legend=True, legend_kwds={'label': "Black Carbon", "orientation": "horizontal"}, cmap="YlOrRd")

ax[1].title.set_text("Nitrogen Dioxide")
ax[1].set_facecolor("xkcd:grey")
cleaned_data.plot(column="Mixed_NO2", ax=ax[1], legend=True, legend_kwds={'label': "Nitrogen Dioxide", "orientation": "horizontal"}, cmap="YlOrRd")

ax[2].title.set_text("Ultrafine Particles")
ax[2].set_facecolor("xkcd:grey")
cleaned_data.plot(column="Mixed_UFP", ax=ax[2], legend=True, legend_kwds={'label': "Ultrafine Particles", "orientation": "horizontal"}, cmap="YlOrRd")

I love these maps! One, I think they’re beautiful (thank you matplotlib!) and two, Its fascinating that there’s not underlying map of Copenhagen here. The streets represent the data points and only the streets represented in the data are projected onto the plot/map.

Route Querying & Data Joining

Querying for a route is done via the Google Maps API, or more specifically, the Google Maps Python client library. In this example, we’re being sure to use bicycling as the mode so that Google Maps knows to optimize for bicycling routes. It would also be interesting to query for walking routes!

gmaps = googlemaps.Client(key=GOOGLE_DIRECTIONS_API_KEY)
starting_location =  "Øste Gasvæk Teater"
ending_location = "Copenhagen Boulders"

try:
  directions_result = gmaps.directions(starting_location, ending_location, mode="bicycling")
except googlemaps.exceptions.ApiError:
  print("It's likely that Google cannot find one or both locations that you've entered")

if not directions_result:
  print("It's likely that Google cannot find one or both locations that you've entered")

The JSON returned from the API has a ton of information that you might need to display the route to the user: HTML snippets, warnings/cautions, street names, and even landmarks! However, we don’t need any of that. The only thing we’re interested in is the “polyline.” Google Polylines are encoded latitude/longitude pairs. This line encodes the geometries of the entire route. As an example, here is the polyline of the route we’ll look at below:

{x_sIskxkAAuFDKvAG^E?_@@mB?_BB]Lc@T_@POz@]dG{BrJyDnBu@f@Wz@o@fAo@^QdAYxBa@zCoAdGyDjDwBz@g@rAe@dAWjD_@jBOh@B`@FFHNJn@p@nB~CzAdCrBjD~AnClAxBpA~BTp@hApBPTTNLQb@cAZs@v@oBVqANe@p@{A|@kBP_@b@i@|@aAVMd@OpAk@hDsAfCaAjEcB|Ao@nB~ArBdBfCtBlCvBlFbEbDnC|CbCpB~APNNBJFl@b@LFt@C~@U|@e@zAY`AWJI|@W|Ae@DBF?`@KZCJ@DQBILUpAm@PMj@QNEjC{@b@Sd@QjA[n@NVTr@zAzA|DXhARdA`@rDZtCZzAfAbEd@~Ax@~ArAlCbFrKrAnCZl@JZVfBHt@Nb@pEtJbCdFbBvDnBhEdKtYzAlEp@fBTj@B\\Hp@`@hA`DdJx@lCt@tBj@`Br@|Bt@`CnApDDHLB`@BVf@T^b@n@`@b@bAn@h@Rr@Rd@FrAJhDJr@Jp@Vr@f@j@n@r@lAP\\Xt@\\xANxAd@nFZ`Ef@rGb@tGf@`GFf@NvATtDTfDHfAZ~Bl@rC`@tAn@rAj@v@nD`DbAv@xAbA`BvAl@j@rDjDz@v@\\RHDf@D~DcBhD{ArD_BjBy@p@QhBu@vBw@b@M|@g@Lt@r@z@fFvF|@t@jAv@z@`@n@Rp@PZDlAL`A?hAIlB]

We capture this in a variable

pline = directions_result[0]['overview_polyline']['points']

And then use the polyline library to decode it as geojson, or more specifically, as long/lat pairs that can be fed into a GeoPandas DataFrame:

decoded_polyline = polyline.decode(pline, geojson=True)
route_data = pd.DataFrame(decoded_polyline, columns=["x","y"])

route = gpd.GeoDataFrame(route_data, geometry=gpd.points_from_xy(route_data['x'], route_data['y']), crs=cleaned_data.crs)

We can then use the route data to overlay onto our original map. We do this by using by using a spacial index build by GeoPandas by using RTree to quickly query based on a spacial predicate. In this case, we’re looking for all data points that are covered by a box that fits around our route.

route_boundary = shapely.geometry.box(*route.total_bounds).buffer(0.01)
overlap = cleaned_data.iloc[cleaned_data.sindex.query(route_boundary, predicate='covers')]

fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(33,21))

ax[0].set_aspect('equal')
ax[0].set_facecolor("xkcd:grey")
cleaned_data.plot(ax=ax[0], column="Mixed_UFP", cmap="YlOrRd")
route.plot(ax=ax[0], marker='o', color='yellow', markersize=50)

ax[1].set_aspect('equal')
ax[1].set_facecolor("xkcd:grey")
overlap.plot(ax=ax[1], column="Mixed_UFP", cmap="YlOrRd")
route.plot(ax=ax[1], marker='o', color='blue', markersize=100)

plt.show()

This is primarily for visual reasons: to see the route placed onto the map, and to also provide a “zoomed in” version. A zoomed in version would be easier with an interactive mapping library, such as Plotly, but there’s no reason we can use a little bit of geometric querying to help us!

Next, for what I’m really interested in: which of our air quality data points intersect with our route. Here, I’m buffering the route geometry by a tiny amount to turn it from a 1D point geometry to a 2D polygon. This is because the precision on the longitude and latitude coordinates (and thus the geometry coordinates) is so high, I often found that the points from the route and the lines from the air quality data wouldn’t intersect at all!

intersections = cleaned_data.iloc[cleaned_data.sindex.query_bulk(route.geometry.buffer(0.0001), predicate='intersects')[1]]

fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(33, 21))

ax[0].title.set_text("Black Carbon")
ax[0].set_facecolor("xkcd:grey")
intersections.plot(column="Mixed_BC", ax=ax[0], legend=True, legend_kwds={'label': "Black Carbon", "orientation": "horizontal"}, cmap="YlOrRd")

ax[1].title.set_text("Nitrogen Dioxide")
ax[1].set_facecolor("xkcd:grey")
intersections.plot(column="Mixed_NO2", ax=ax[1], legend=True, legend_kwds={'label': "Nitrogen Dioxide", "orientation": "horizontal"}, cmap="YlOrRd")

ax[2].title.set_text("Ultrafine Particles")
ax[2].set_facecolor("xkcd:grey")
intersections.plot(column="Mixed_UFP", ax=ax[2], legend=True, legend_kwds={'label': "Ultrafine Particles", "orientation": "horizontal"}, cmap="YlOrRd")

Now with this data, we can aggregate the air quality data filtered by the route to get some meaningful statistics like how many times more polluted is our route compared to the average:

(intersections[["Mixed_NO2", "Mixed_UFP", "Mixed_BC"]].mean() / cleaned_data[["Mixed_NO2", "Mixed_UFP", "Mixed_BC"]].mean())

Mixed_NO2    1.637403
Mixed_UFP    1.432781
Mixed_BC     1.518163
dtype: float64

Or what’s the maximum average exposure along the route:

intersections[["Mixed_NO2", "Mixed_UFP", "Mixed_BC"]].max()

Mixed_NO2       52.0
Mixed_UFP    38500.0
Mixed_BC         3.0
dtype: float64

Next Steps

The Jupyter notebook used to do this analysis is available here as a Binder and here as a Gist for you to explore!

Some interesting ideas:

Exposure amounts based on time. The Google Maps API returns a Polyline and duration for each part of the trip!
Using machine learning to fill in gaps in data, what about those -999999 data points or roads that aren’t represented at all?
Are there correlations between this data and any other data available at https://opendata.dk? Housing prices? Bike parking?

Sargable Queries & MUL Indexes; or Why My Query is Slow

2021-07-09T00:00:00+00:00

You’re on Ops. Debugging Error-s in Invoicing Rails app, as usual. They’re stored in MySQL, and accessed through ActiveRecord and some helper methods. They’ve already been updated with a status_message, so now it’s time to dig in and investigate.

irb(main):001:0> Error.column_names
[
    "id",
    "message",
    "backtrace",
    "status_message",
    "created_at",
    "updated_at",
    "runtime_environment",
    "error_type",
    "ticket_id"
]

The sun beams through your window on another record-high heat wave afternoon, but you’re cool as a cucumber: you brought your sunglasses and a tall glass of trusty H2O.

Let’s do this!

Error.where_message_like("NoMethodError: undfined method `is_approved_by_finance?\\' for nil:NilClass")

…

You check the time… it’s been over 45 seconds.

…

…maybe you shouldn’t have run this in the console??

…

Another 23.7 seconds pass and you decide to CTRL-C your way out of there before you cause a production incident. You decide to just pull the Error IDs from the JIRA ticket like a pleb. Suddenly the heat wave feels even hotter.

But this is nagging you. I mean, what gives?? Why doesn’t this method ever just do the thing it says on the tin? Sure, if we use the .no_status scope (WHERE status_message IS NOT NULL) it seems to work, but why can’t we get simple text search to work on this column? Sure there are a lot (a lot = 8612441) of rows, but I feel like I’ve seen text searching work before. Should we switch to Postgres?

Sargable & Non-sargable

Sargable is a gate keep-y portmanteau (<- also gate keep-y. A portmanteau is a word that smashes together the sounds and meaning of two words, like spoon + fork = spork) for “search argument-able.” We use the term to describe the ability of the database’s SQL optimizer to leverage an index. There are a few things that determine a query’s sargability (not sure if that’s a word), but a big one to look out for is using wildcards (%) when doing text searches.

Let’s look at the implementation of our where_message_like method to see if we might be running into something like this.

irb(main):001:0> ActiveRecord::Base.connection.to_sql(Error.where_message_like("NoMethodError: undefined method `is_approved_by_finance?\\' for nil:NilClass"))
=> "SELECT `errors`.* FROM `errors`  WHERE (message like '%NoMethodError: undefined method `is_approved_by_finance?\' for nil:NilClass%')"

Aha, so we have a wildcard (%) on either end of string in the WHERE clause:

'%NoMethodError: undefined method `is_approved_by_finance?\' for nil:NilClass%'

To understand why this is a problem, let’s pretend you’re the SQL query optimizer. Let’s say that I ask you to find all of the words in the dictionary that contain the substring get:

SELECT * FROM entries WHERE entries.word LIKE '%get%'

You’ll have to scan through every word (row) in the dictionary (entries table) to find all of the instances:

...
- GETaway
- fidGETy
- veGETal
- exeGETe
- nugGETy
...

If, instead, I asked you to get all of the words in the dictionary that begin with get, you could just flip to the correct page and give me all the words! In this way, the dictionary being in alphabetical order is a type of index:

SELECT * FROM entries WHERE entries.word LIKE 'get%'

...
- GETaway
- GETable
- GETting
- GETters
...

“But, Thomas,” I hear you asking, “what if we want to do a substring search rather than search from the beginning of the string?”

That’s a great question! For our Error scenario, that’s seldom the case, but there are more intentional indexes we could use to get us a more robust sub-string search, like MySQL’s FULLTEXT or Postgres’ ts_vector. It’s worth noting that full-text indexes aren’t magic, and your tables and queries have to change to support them.

For our example, we don’t have these things readily at our disposal, so we’ll continue with the assumption that we’re searching for a substring at the beginning of the column’s contents.

`TEXT` & `VARCHAR`

So now, let’s see this sargable business in action by looking at the query plans!

First the non-sargable, double wildcard (%), query:

irb(main):002:0> puts ActiveRecord::Base.connection.explain(Error.where_message_like("NoMethodError: undefined method `is_approved_by_finance?\\' for nil:NilClass").to_sql)
  EXPLAIN (0.5ms)  EXPLAIN SELECT `errors`.* FROM `errors` WHERE (message like '%NoMethodError: undefined method `is_approved_by_finance?\' for nil:NilClass%')
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
| id | select_type | table          | type | possible_keys | key  | key_len | ref  | rows    | Extra       |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
|  1 | SIMPLE      | errors         | ALL  | NULL          | NULL | NULL    | NULL | 6739497 | Using where |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)

Then the sargable, search from the beginning of the contents, query:

irb(main):003:0> puts ActiveRecord::Base.connection.explain(Error.where("message LIKE \'NoMethodError: undefined method `is_approved_by_finance?\\' for nil:NilClass%\'").to_sql)
  EXPLAIN (0.5ms)  EXPLAIN SELECT `errors`.* FROM `errors` WHERE (message LIKE 'NoMethodError: undefined method `is_approved_by_finance?\' for nil:NilClass%')
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
| id | select_type | table          | type | possible_keys | key  | key_len | ref  | rows    | Extra       |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
|  1 | SIMPLE      | errors         | ALL  | NULL          | NULL | NULL    | NULL | 6739497 | Using where |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)

Oh crap. What gives?! They look exactly the same! Why isn’t MySQL optimizing!? We sarged(?) it!

Unfortunately for us, our message doesn’t have an index. Further, the column is a TEXT column, and (given its extremely varied and undefined length) if we want to add an index, MySQL requires us to specify a prefix length, which tells MySQL how much of the beginning of the string should be indexed. Further further, only the InnoDB tables can be indexed this way. (Our errors table uses the InnoDB engine, so no problem there).

How it looks with an index

Even though our message TEXT column doesn’t have an index, our ticket_id column does:

irb(main):004:0> puts ActiveRecord::Base.connection.explain('errors')
  EXPLAIN (2.6ms)  EXPLAIN errors
+---------------------+--------------+------+-----+---------+----------------+
| Field               | Type         | Null | Key | Default | Extra          |
+---------------------+--------------+------+-----+---------+----------------+
| id                  | int(11)      | NO   | PRI | NULL    | auto_increment |
| message             | text         | YES  |     | NULL    |                |
| backtrace           | text         | YES  |     | NULL    |                |
| status_message      | varchar(255) | YES  | MUL | NULL    |                |
| created_at          | datetime     | YES  |     | NULL    |                |
| updated_at          | datetime     | YES  |     | NULL    |                |
| runtime_environment | text         | YES  |     | NULL    |                |
| error_type          | varchar(255) | YES  | MUL | NULL    |                |
| ticket_id           | varchar(255) | YES  | MUL | NULL    |                |
+---------------------+--------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)

The ticket_id column is of type varchar(255), which means we have no problem putting an index on there. And that’s exactly what we did! In our case, it uses a MUL (or “Multiple”) key, meaning that its value is used at the beginning of a non-unique key (multiple records can have the same index).

Let’s look at how our knowledge of sargable wildcard queries plays a role with this type of column.

First the non-sargable, double wildcard (%), query:

irb(main):005:0> puts ActiveRecord::Base.connection.explain(Error.where('ticket_id LIKE \'%A00057440\'').to_sql)
  EXPLAIN (0.5ms)  EXPLAIN SELECT `errors`.* FROM `errors` WHERE (ticket_id LIKE '%A00057440%')
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
| id | select_type | table          | type | possible_keys | key  | key_len | ref  | rows    | Extra       |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
|  1 | SIMPLE      | errors         | ALL  | NULL          | NULL | NULL    | NULL | 8612441 | Using where |
+----+-------------+----------------+------+---------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)

Then the sargable, search from the beginning of the contents, query:

irb(main):006:0> puts ActiveRecord::Base.connection.explain(Error.where('ticket_id LIKE \'%A00057440\'').to_sql)
  EXPLAIN (1.2ms)  EXPLAIN SELECT `errors`.* FROM `errors` WHERE (ticket_id LIKE 'A00057440%')
+----+-------------+----------------+-------+--------------------------------------+--------------------------------------+---------+------+------+-----------------------+
| id | select_type | table          | type  | possible_keys                        | key                                  | key_len | ref  | rows | Extra                 |
+----+-------------+----------------+-------+--------------------------------------+--------------------------------------+---------+------+------+-----------------------+
|  1 | SIMPLE      | errors         | range | index_errors_on_ticket_id            | index_errors_on_ticket_id            | 768     | NULL |  214 | Using index condition |
+----+-------------+----------------+-------+--------------------------------------+--------------------------------------+---------+------+------+-----------------------+
1 row in set (0.00 sec)

Aha! In the first query plan, MySQL tells us it will scan through ALL 8612441 rows Using where. The second, sargable query plan tells us that it will only need to scan through a range of 214 by Using index condition!

Wow… a single % can make a huge difference.

I’ll leave it as an exercise for the reader to SET profiling = 1; to see the real cost. See documentation here.

Back on Ops

Ahhh, the sun is finally starting to set. We haven’t finished (or really even begun) our investigation, but we’re a little wiser about MySQL’s indexing patterns.

How can we use this knowledge to make our jobs a little easier and beat the heat?

I guess the first thing to know is that where_message_like could take a very long time if there are no other WHERE conditions. This is because it’s a TEXT column with no index; but even if it had an index, using a wildcard at the front of our search string isn’t doing us any favors.

Also, it’s nice to know that the ticket_id column is indexed, and if we want to search, we can use a sargable query to get our results much faster! Having a consistent format for ticket_id might be something to consider; if we can deduce what the beginning of the string is, finding all of the relevant records could be super efficient!

Lastly, if we did want to index the message column, we have a few options:

We could migrate it from TEXT to VARCHAR. The advantage there is gaining access to an easy index, but we give up the benefits of choosing TEXT in the first place.

We could use a column prefix key by specifying a prefix limit. That might get us some good bang for our buck if we use sargable queries.

Or, we could invest in a full-text index and really leverage what InnoDB can do! However, this will require us to alter our table a bit more and adjust our query patterns. Whether or not this plays nice with Rails is another question we’d need to answer.

Nevertheless, it’s 5:00 p.m. (4:56 p.m., but who’s counting), so it’s time to call it a day and grab the last few rays of sun down by the water.

Before you ⌘-Q out of Slack, you remember something. “…didn’t Brianna say that they were going to look at this issue??”

You refresh JIRA:

“Status: Done.”

You give a shout out to Brianna and ⌘-SHIFT-Q.

Rocket Lab Telemetry From Video

2021-05-15T00:00:00+00:00

I’m both excited and saddened to share with you my experience pulling rocket telemetry data from a recorded video live stream of a rocket launch. What I’m sharing with you is going to feature very hacky brute force data munging, security worst practices, and, the data engineer’s favorite: Google Sheets.

But before I share, I want to give my condolences and pay my respects to Rocket Lab and their engineering team. Peter Beck’s empathy and humility is demonstrated in all the work that they do. Despite today’s anomoly, Rocket Lab is nevertheless one of the most ambitious, successful, and consistent areospace organizations, and thanks to their participation within the wider community, we can all learn from them.

On one of our toughest days, our team operated with professionalism and worked swiftly to ensure the anomaly was managed safely. Our team is resilient, and our top priority remains to safely and reliably return to flight for our customers. We will learn from this, and we’ll be back on the pad again. —Rocket Lab Experiences Anomaly During Launch

The Anomaly

Early today, Rocket Lab launched their 20th Electron rocket, Running out of Toes from their Launch Complex 1 Mahia Peninsula in New Zealand.

Unfortunately, the mission ended in failure right around T+00:02:30 shortly after main engine cutoff and second stage separation. In the live stream, the second stage appears to ignite and then quickly cutoff. Rocket Lab released a statement confirming that the anomaly had lead to the failure of the mission, but not yet releasing any preliminary data that might point to the cause of the failure, of course, pending the investigation.

Days like to day are a magnet for armchair engineers.

Youtube Video Stream

This is a single frame from the live stream right at T-00:00:00. On the right hand side, you can see that they helpfully show some telemetry: flight time, speed, and altitude, along with the video feed itself, and some stages or “gates” on the left hand side.

When wanting to figure out what happened, besides the video, this was all the data we in the peanut gallery had.

So, I did what any sane person would do and started going frame by frame to record the telemetry by hand in a spreadsheet. After I realized that the area of interest was about three minutes and thirty second long, and the stream was at 30 frames/second, I was looking at transcribing thousands of frames! Given that the data wasn’t really updating every single frame, the number was likely to be a lot lower, but still… The band of DIY rocket scientists were waiting; this couldn’t take all day!!

I pulled the video down using youtube-dl just in case the video was taken down from Youtube. Rocket Lab isn’t known to do this, but SpaceX, for example, de-lists their videos and they can be difficult to find.

$ youtube-dl https://www.youtube.com/watch?v=Zw3sIUyfSfc

Google Video Intelligence API

Luckily, this is the exact kind of job that neural networks are suited for. In fact, the first algorithms you learn about are ones used for handwritten digit recognition. These weren’t even handwritten, so the job could end up being quite easy! One challenge was that we’re dealing with thousands of frames, not just a single image. My personal toolkit doesn’t support analyzing video, so I reached for Google Cloud Video Intelligence API

This service uses OCR (Optical Character Recognition) to detect and extract text from video. Perfect!

After configuring and deploying the resources in my Google Cloud account. I uploaded the video to Google Cloud Storage and called the REST API

// request.json
{
  "inputUri": "gs://thomascountz/rocket-lab-20-launch-anomaly.mp4",
  "features": ["TEXT_DETECTION"],
  "videoContext": {
    "textDetectionConfig": {
      "languageHints": ["en-US"]
    }
  }
}

curl -X POST \
  -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
  -H "Content-Type: application/json; charset=utf-8" \
  -d @request.json \
  https://videointelligence.googleapis.com/v1/videos:annotate

Data Cleanup/Presentation

What came back was exciting and a little unexpected. I guess I didn’t know what to expect (it’s not like I had read any documentation or anything).

Firstly, the API responds with a job name that then you can later use to query for the results and other metadata. I used that name to send another API request and I got back results almost instantly. The entire took maybe 2 second to complete and I got something back like this:

{
  "name": "projects/REDACTED/locations/us-west1/operations/REDACTED",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoProgress",
    "annotationProgress": [
      {
        "inputUri": "/thomascountz/rocket-lab-20-launch-anomaly.mp4",
        "progressPercent": 100,
        "startTime": "2021-05-15T15:39:55.442511Z",
        "updateTime": "2021-05-15T15:41:17.361940Z"
      }
    ]
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoResponse",
    "annotationResults": [
      {
        "inputUri": "/thomascountz/rocket-lab-20-launch-anomaly.mp4",
        "segment": {
          "startTimeOffset": "0s",
          "endTimeOffset": "218.100s"
        },
        "textAnnotations": [
          {
            "text": "2612",
            "segments": [
              {
                "segment": {
                  "startTimeOffset": "98.400s",
                  "endTimeOffset": "98.500s"
                },
                "confidence": 1,
                "frames": [
                  {
                    "rotatedBoundingBox": {
                      "vertices": [
                        {
                          "x": 0.7729167,
                          "y": 0.05277778
//... snip...

That’s a heavily truncated version, but the things I noticed were:

The result was so big, I was having trouble exploring it all at once.
The data was not in order by frame count the way a video is. This makes sense since OCR acts on a single image and Google Cloud most likely parallelized the work for each image before collecting it into a single result (fan-out/fan-in).
There wasn’t an obvious way to tell which frames contained the data I cared about (time, speed, altitude) and for each frame, it wasn’t entirely clear which bit of text mapped to which data point (altitude and speed are both numbers).

1. The results were big

Ha! 1116879 lines of unminified JSON never hurt anyone! Although my computer choked on opening the response in something like vim, I used jq to begin to explore the data.

After getting an understanding for the shape, I actually “trimmed” of the pieces of data I didn’t care about by mapping through each textAnnotation and only keeping the text found and the start and stop timestamps. (This ended up making my life more difficult… More on this later.)

jq '.response.annotationResults[0].textAnnotations[] |= { text: .text, start: .segments[0].segment.startTimeOffset, end: .segments[0].segment.endTimeOffset }' response.json > response_trim.json

This produced something a little more manageable:

{
  "name": "projects/REDACTED/locations/us-west1/operations/REDACTED",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoProgress",
    "annotationProgress": [
      {
        "inputUri": "/thomascountz/rocket-lab-20-launch-anomaly.mp4",
        "progressPercent": 100,
        "startTime": "2021-05-15T15:39:55.442511Z",
        "updateTime": "2021-05-15T15:41:17.361940Z"
      }
    ]
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoResponse",
    "annotationResults": [
      {
        "inputUri": "/rocket_lab_001/end-to-end-flight.mp4",
        "segment": {
          "startTimeOffset": "0s",
          "endTimeOffset": "218.100s"
        },
        "textAnnotations": [
          {
            "text": "2612",
            "start": "98.400s",
            "end": "98.500s"
          },
          {
            "text": "3467",
            "start": "110.300s",
            "end": "110.300s"
          },
          {
            "text": "1362",
            "start": "73s",
            "end": "73.100s"
          },
          {
            "text": "2392",
            "start": "94.800s",
            "end": "94.900s"
          },
          {
            "text": "2079",
            "start": "89.300s",
            "end": "89.300s"
          },
// ... snip ...

2. Data out of order

With the timestamps in place, the data being out of order was no longer an issue. I easily parsed the start value of each object into a float, and then sorted in ascending order.

Note that this was the time in seconds based on the video I uploaded to cloud storage which was only about three minutes of the hour+ Youtube stream.

3. Knowing what the text means for each timestamp

Now that things were sorted (pun intended), I had to then understand what each of the text values represented. For this, I moved to Ruby. (I told you that this was brute force data munging and worst practices. I hope you didn’t come here thinking I knew what I was doing).

In Ruby, I read in the trimmed file as a hash, removed the metadata, converted everything to floats, integers, and symbols, and grouped by the start timestamp.

Yes, there can be, and were, many annotations that started at the same time. Not all of the data on the video was updated in sync, especially because the flight time was only updated once per second, but the telemetry data was updated more often. (Thanks for the high fidelity Rocket Lab!).

I ended up with a hash that looked like this:

{
  # ...snip...
  205.6=>
   [{:text=>"7687", :start=>205.6, :end=>205.6},
    {:text=>"T+// OO:03:18", :start=>205.6, :end=>206.5}],
  205.7=>[{:text=>"7661", :start=>205.7, :end=>206.1}],
  206.3=>[{:text=>"7677", :start=>206.3, :end=>206.3}],
  206.4=>
   [{:text=>"110.0", :start=>206.4, :end=>207.8},
    {:text=>"7689", :start=>206.4, :end=>206.7}],
  206.6=>[{:text=>"T+// OO:03:19", :start=>206.6, :end=>207.5}],
  206.8=>[{:text=>"7676", :start=>206.8, :end=>206.8}],
  207.0=>[{:text=>"7656", :start=>207.0, :end=>207.0}],
  207.1=>[{:text=>"7647", :start=>207.1, :end=>207.2}],
  207.3=>[{:text=>"7649", :start=>207.3, :end=>207.3}],
  207.4=>[{:text=>"7657", :start=>207.4, :end=>207.4}],
  207.5=>[{:text=>"7669", :start=>207.5, :end=>207.5}]
  # ...snip...
}

Just taking a look at the :text values, we can see some integers, floats, and a string that starts with T+// ; this formatting happens to map to the speed, altitude, and flight time, respectively.

Remember earlier when I used jq to delete a bunch of data that I thought I didn’t need? It turns out that that data contained the bounding boxes for each annotation, that is for each piece of text found, the results give you four Cartesian coordinates that correspond to the x and y pixel locations of four corners of a square that describe where in the frame the text was found. In other words, instead of relying on each data point coincidentally having a different format, I could have used the bounding boxes to query for each data point in a given frame. (Worst. Practices. I literally warned you about this…).

I ran the hash through an each block that will make any programmer cry and ended up with four values for each frame: frame, which represents the frame in seconds, time, or flight time, speed and altitude.

result = []
data.each do |frame_data|
  f = frame_data[0]
  t = frame_data[1].find { |v| v[:text].include?("//") }
  t = t[:text] if t
  s = frame_data[1].find { |v| v[:text].to_i.to_s == v[:text] }
  s = s[:text] if s
  a = frame_data[1].find { |v| v[:text].include?(".") }
  a = a[:text] if a
  result << { frame: f, time: t, speed: s, altitude: a }
end

Looking at this feels embarrassing, but you know what? It worked!

Here’s the same slice of data from above:

[
  # ...snip...
    {:frame=>205.6, :time=>"T+// OO:03:18", :speed=>"7687", :altitude=>nil},
    {:frame=>205.7, :time=>nil, :speed=>"7661", :altitude=>nil},
    {:frame=>206.3, :time=>nil, :speed=>"7677", :altitude=>nil},
    {:frame=>206.4, :time=>nil, :speed=>"7689", :altitude=>"110.0"},
    {:frame=>206.6, :time=>"T+// OO:03:19", :speed=>nil, :altitude=>nil},
    {:frame=>206.8, :time=>nil, :speed=>"7676", :altitude=>nil},
    {:frame=>207.0, :time=>nil, :speed=>"7656", :altitude=>nil},
    {:frame=>207.1, :time=>nil, :speed=>"7647", :altitude=>nil},
    {:frame=>207.3, :time=>nil, :speed=>"7649", :altitude=>nil},
    {:frame=>207.4, :time=>nil, :speed=>"7657", :altitude=>nil},
    {:frame=>207.5, :time=>nil, :speed=>"7669", :altitude=>nil}
  # ...snip...
]

Now, all I had to do was clean up the T+// from the timestamp and ship this off to a CSV!

The Results

You can view the entire dataset on Google Sheets here.

This is kind of where my armchair engineering stopped. I mean, I don’t really know much about rockets, I just love data! (I also love rockets, but I just don’t know enough to make guesses as to what happened). I shared this data with my rocket-knowing friends to see if they could make anything of it.

What I do know is that things shouldn’t have ended around T+00:03:30 and that the green line (velocity) should have kept going up, but around T+00:02:30, right around main engine cut off and stage separation, it started going down and stayed down.

Conclusion

This was a fun project from a bad day. It really illustrates to me the power of motivation. This isn’t too dissimilar to my day job, but at work, I can’t just spin up cloud resources and the scale of the data is too large to try to manipulate in memory. That said, it was really fun to try to dive head first into a problem and come out on the other side having learned more about Google Cloud, jq, rocket trajectories, and basic data manipulation.

If I were to approach this problem again, I would keep the rotatedBoundingBox data and use it to identify the meaning of the data. SpaceX’s Falcon 9 streams, for example, contain two readouts each of speed and altitude, one for the first and second stages. It would be impossible to distinguish between the two on format alone.

Secondly, I’d get the data in SQL database right away (or at least store it somewhere where I can use a SQL-like API). Maybe I should up my data game, but my brain likes to think in SQL for problems like these; having to think procedurally in a loop was unintuitive for me.

Thanks for reading! Let me know what you think or if you have any questions. Would you be interested in having raw telemetry data formatted and presented after a launch stream? I’m certainly impressed by the quality of the data coming through! See if you can spot MAX-Q in the velocity graph!

Hexo - Schematic Capture, Mechanical Design, & PCB Layout

2021-05-13T00:00:00+00:00

Pictured above is the schematic for Hexo’s circuit. Like I mentioned earlier, reading and interpreting a schematic is a creative process akin to jazz… although in this case it is highly precise. Not only is this a graphical representation meant to aid human designers, each component and connection is backed by a computer model which can do things like check for output pin conflicts and missed connections, in a process called electrical rules checking. I use a free open source tool called KiCAD EDA (EDA: electrical design automation) for this work.

Schematic capture is about the process of encoding a circuit into an EDA. KiCAD has a very large library of components, but also has tools for building new components, cataloging datasheets (technical document and specification produced by component manufacturers), and exporting files to use in SPICE simulations.

After schematic capture, I moved onto mechanical design. All I knew was that I wanted Hexo to be worn as a pendant on a necklace. In electronics projects, mechanical design is concerned primarily with the shape of the PCB as it’s often mounted without an enclosure or case. For Hexo, the PCB itself would be exposed as the final form factor and I was inspired by game pieces from one of my husband and my (mine?) favorite games, Eclipse, pictured below. Eclipse is a sci-fi based game and besides the size and shape of its pieces, I was inspired by the futuristic and mysterious vibe of the game’s design.

The mechanical design was constrained primarily by the size of the battery I planned to use (or rather the size of the mechanical clip that holds the battery) and the user experience of pressing a small button and reading the numbers. The important constraints limited how small Hexo could be because I wanted Hexo to be as small as possible (because I thought it would look cooler).

Other than that, the form factor was left to my aesthetic sense and what was feasibly manufacturable (these constraints were helpfully available on the manufacturer’s website).

I went with a 30mm x 30mm square that had three corners cut away at 45º angles because… I thought it would look cooler (sensing a theme?) and because the final surface area was within the tolerances of the manufacturer’s cheapest options. (Definitely cooler).

Finally, the PCB Layout, or the process by which you plan where to mount each component on the final board, was all that was left.

During prototyping, we used the THT (through-hole technology) components that had leads that allow them to “plug-into” a breadboard. For PCBs, we can also use SMT (surface mount technology) components that, instead of having leads that go into a hole, they have pins that are soldered to pads.

These pads are where the electrical connection between the component pins and the rest of the circuit is made. Taken together, a group of pads make up a component’s footprint or the overall shape and dimensions of the electrical connections between the component and the PCB.

In the image above, you can see the various component footprints on the bare Hexo PCBs before they were assembled.

The battery clip datasheet, pictured above, shows the only footprint I had to design by hand in CAD for Hexo (the other components used standard footprints that were pre-designed in KiCAD). The footprint is a relatively simple three square design, but it was a lot of fun to get practice at taking specs and turning them into something that would leave the page.

You can see the battery clip’s footprint as the three green squares in the layout diagram below. Note that the diagram is showing both the front and the back of Hexo at the same time and that the battery clip was mounted on the back.

The final large technical constraint, after the board shape and the component footprints, is the management of electrical traces, or the copper “wires” that run (unseen) through the board and connect everything together.

Traces are the things that connect the LED to the battery and the battery to the switch, for example. The way this is done has to do with the manufacturing process, which is step 4, so I can’t go into it here, but I will leave you this video link if you’re interested.

There’s an art to PCB layout and there are tools to help make it easier. Like a lot of design, there’s a mix of technical and aesthetic attitudes that lead to a PCB being designed one way or another. When you begin the PCB layout process, you have, what is literally called, a rats nest of traces and components that need to be untangled, taking into account electrical requirements, interference, manufacturability, etc.

Now that Hexo was designed, and (nearly) every decision that could be made, was made, it was time to send the board to the manufacturer. After which, you’ll realize that you made a tiny mistake and hope that either the manufacturer will notice and fix it, or that it was tiny enough to have no impact.

Hexo - Prototyping, Design, Simulation

2021-05-13T00:00:00+00:00

Pictured above is a printed circuit board or PCB. Dubbed Hexo, a play on the prefix hexa-, meaning having six, it’s an electrically random “six-sided” die. I like to call it “electrically random,” because that sounds cool, and because the LED which lights up at the end is an output of a function whose inputs are voltages and time. When you push the button, Hexo cycles through its six LED, slower and slower, until eventually leaving just one LED on.

I built Hexo because I wanted to learn how I should go about bringing something into existence… something inanimate, that is. What I’m left with is a kind of systematic process that I hope will allow me to build bigger and more ambitious projects in the future!

This is what I’ve discovered so far; electronics projects like this can be broken down into the following steps:

Requirements Gathering
Prototyping, Analysis, & Simulation
Schematic Capture, Mechanical Design, & PCB Layout
Manufacture & Parts Ordering
Assembly
Validation, Testing, & Integration

In this post, I want to share with you my experience with prototyping, design, and simulation, the steps centered around implementing functional design. So grab a cup of tea or coffee and let’s geek out!

Prototyping, Analysis, & Simulation

I didn’t design Hexo’s primary circuitry; the design came from 555 Timer Circuits’ Roulette schematic and the many derivatives thereof. With a few very minor modifications, I was able to turn their 9-volt, 10 LED design into a 3-volt, 6 LED design with a similar effect. To read more about that circuit and how it works, you can read my writeup on Hackaday here.

With that said, in order for me to understand how to modify the circuit and to measure its performance, I had to build prototypes, do some pen-and-paper analysis, and (although perhaps a bit overkill), employ the use of circuit simulation tools. These things all happened in a happy nonlinear feedback loop during multiple iterations.

For prototyping, I use these things called breadboards (also known as solderless breadboards, it got its name from the old practice of using nails hammered into a cutting board to wire up temporary circuits), pictured below. A breadboard allows you to quickly “plug in” a bunch of through-hole _(or _THT, through-hole technology) components and test out designs.

Below, you can see a few THT components next to their equivalent SMT (surface mount technology) counterparts, which are often used on PCBs because of their small size. You can probably guess why it’s easier to prototype with through-hole components, though they aren’t only used for prototyping!

Prototyping in this way allows you to take the circuit off of the page and place it in your hands in order to answer questions like: “does this even work?”, and “is it supposed work like that?”, and “ugh… why isn’t this working?”, usually followed by, “oh wow, how am I such an idiot?” (I put in the LED backwards… again.)

Having a small country’s worth of components sitting around is the only way I’ve seen electronics hobbyists get through prototyping. (That’s why a lot of us join a hackerspace, a physical place where geeks unite and stockpile their trove of parts and tools). Before I set off on my electronics journey, I invested quite a bit in building out a workbench and I have more than enough to share!

If prototyping is taking a circuit off of the page, then circuit analysis is… writing more stuff on the page? Analysis allows you to understand the mathematical theory behind the circuit. Equations derived from the Laws of Ohm and Kirchhoff allow us to understand idealized circuits, that is, how the design would function in a vacuum without interference, resistive wires, and perfect electron flow. Combined with higher-level patterns, one can begin to read circuit diagrams, called schematics, like lead sheets and fake books in the world of jazz or design patterns in the world of code writing.

Having not formally studied electrical engineering, this was one of the more exciting parts of this build. I love seeing theory on the page become measurable and observable in real life. This is the part where we use math to affect decisions about design. It turned out that even though I was switching from using a 9-volt battery to a 3-volt battery, I didn’t need to change any of the other components in the circuit.

Lastly, simulation. Using CAD (computer aided design) software allowed me to more easily and efficiently do circuit analysis; particularly with circuits that contained integrated circuits, or the little black things that look like microchips.

This is where I learned about SPICE: Simulation Program with Integrated Circuit Emphasis, and its variants like LTspice, pictured above.

SPICE uses, what I’ve only seen described as, an ASCII-based declarative language to define component models, netlists (how components are connected), and initial condition directives to calculate the electrical state of a given circuit. In Hexo’s case, I used a transient analysis, which allowed me to observe the circuit’s behavior over time. This was particularly useful given that the LEDs would blink, and “blinking” is a function of time.

Pictured above, the top half of the image shows the state of each of the ten LEDs, in that particular Hexo iteration, plotted over time. Taken as a whole, you can see how the rate of flashing (defined by the period and duty cycle of the square wave) begins to slow or stretch out over time as the LEDs blink and cycle slower and slower, before ending with only one remaining lit.

I’m really excited about SPICE. I’m not sure where it sits in the professional electronics industry, but there’s not a lot of documentation for us mere hobbyists. I hope that we, as a maker community, can work together to change that.

A more intuitive type of circuit simulation can be done using the Falstad Circuit Simulator.

In fact, if you want to play with Hexo, you can take a look at the simulation with Falstad’s free browser-based app here. Although Falstad’s models are less realistic in a few ways, the drag-and-drop interface, the current flow visualizations, and real-time interactivity helped me build up a more intuitive understanding of Hexo’s design.

From bench, to paper, to screen. Back and forth, I’ve iterated and studied Hexo’s design until I felt confident that the circuit would work after I tweaked it a bit. I imagine a similar, if not more rigorous, process would exist for building an original design as well.

Now that we’ve experimented with prototypes and understood the theory, it’s time to design the physical thing!

Communication Superhighways

2021-03-13T00:00:00+00:00

Good communication in an emergency is vital: What is happening? Who is leading? Where are we meeting? Et al. Just like firefighters, who use highways instead of local roads to get to a burning building more quickly, our teams should leverage communication highways to respond efficiently in the face of an emergency.

This communication infrastructure isn’t just logistical: you may have a dedicated chat channel, video conference URL, or a paging system already. Designing communication superhighways is also about investing in our workplace environment and ultimately in each other.

Cognition and emotion are tightly intertwined, which means that [we] must design with both in mind. (Norman, 2013).

How we feel about our work with one another will greatly influence our ability to work together when things are difficult—if I don’t feel comfortable or confident communicating about our work when there isn’t an emergency, how do you think we’ll get along when there is one?

In practice, this looks like investing in maintaining a healthy workplace environment, team collaboration, transparency, accountability, and responsible incentives. Going into each of these are beyond the scope of this essay—instead, I’ll leave you with this: ethically, we should all be working towards keeping our team dynamic positive and healthy, but capitalistic incentives being what they are, we could all do well to remember that those team outings really do help your bottom line.

Norman, D. A. (2013). The design of everyday things. MIT Press.

Crafting an Invitation to Think

2021-03-08T00:00:00+00:00

Foreshadowing: signaling an outcome of chapter eight in the middle of chapter one, i.e. the original spoiler. Although, akin to the mean-spirited handiwork of a Reddit troll, welded gracefully, foreshadowing helps an author build tension, introduce indirection, and drive home a butterfly-inducing denouement.

In non-fiction, we are afforded a similar device: we can build a compelling narrative or stack ever increasing layers of complexity onto a topic to help elevate our reader to a plane of inevitability. At this point, we want our readers to say, “Ah! Yes! This is so obvious!”

If we over-articulate the point, however, our audience disengages; like continually begging a friend to show up to your (virtual) party after they’ve already said yes. We want to posit an idea just enough to engage a reader’s curiosity and compel them to have thoughts of their own—the aim is to craft an invitation to think.

Repeated Signals Wear Grooves

2021-03-01T00:00:00+00:00

In The Design of Everyday Things, Don Norman shares an anecdote about “signifiers,” which are the design elements that communicate the purpose, structure, and operation of an object or system. The anecdote was about a wall onto which someone placed an empty bottle. Over time, more and more empty bottles showed up on this wall:

…as soon as one person discovers it can be used to dispose of empty drink containers, the discarded container becomes a signifier, telling others that it is permissible to discard their items there. (Norman, 2013).

This is a powerful message about the implicit, and often unconscious, signals we send one another on our teams.

For example, the boss may say “don’t feel like you have to work late,” but if they and other members of your team do, the implicit signal soon becomes louder than words.

This doesn’t always have to be negative, exhibiting healthy team customs, norms, and rituals also has the happy effect of instilling positive behaviors, too. If everyone is always on time to meetings, it can soon become a happy groove worn into the fabric of the team.

These grooves are worn in over time for better and for worse, however they aren’t a drop-in replacement for explicit team structures and boundaries. Signals help to reinforce healthy team habits and deter unhealthy ones, but they shouldn’t be relied upon to establish them.

Norman, D. A. (2013). The design of everyday things. MIT Press.

Arduino Tone for ESP32

2021-02-21T00:00:00+00:00

Arduino has a built-in tone() library which allows you to send a PWM frequency at 50% duty cycle to a specific pin in order to generate a tone on a piezoelectric buzzer with an optional duration.

tone(pin, frequency)
tone(pin, frequency, duration)

This functionality is famously unavailable in Espressif’s arduino-esp32 library and members of the community have found various work-arounds such as using the native LED Control functions to generate PWM signals.

However, looking more closely at arduino-esp32 library, not only has Espressif provided a clean API for generating tones, they’ve provided an interface for generating specific PWM frequencies for specific notes on the chromatic scale in different octaves.

double ledcWriteNote(uint8_t chan, note_t note, uint8_t octave){
  const uint16_t noteFrequencyBase[12] = {
  //   C        C#       D        Eb       E        F       F#        G       G#        A       Bb        B
      4186,    4435,    4699,    4978,    5274,    5588,    5920,    6272,    6645,    7040,    7459,    7902
  };

  if(octave > 8 || note >= NOTE_MAX){
      return 0;
  }
  double noteFreq =  (double)noteFrequencyBase[note] / (double)(1 << (8-octave));
  return ledcWriteTone(chan, noteFreq);
}

Although not directly compatible with Arduino’s tone(), the function provides a dedicated interface for producing named frequencies out of the box via the note_t type.

typedef enum {
    NOTE_C, NOTE_Cs, NOTE_D, NOTE_Eb, NOTE_E, NOTE_F, NOTE_Fs, NOTE_G, NOTE_Gs, NOTE_A, NOTE_Bb, NOTE_B, NOTE_MAX
} note_t;

Example Usage

I’ll take an example from @lbernstone’s Tone32 library. Their library is a great solution for providing cross-compatibility between code written for an Arduino and code written for an ESP32.

void loop() {
  tone(BUZZER_PIN, NOTE_C4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_D4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_E4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_F4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_G4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_A4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
  tone(BUZZER_PIN, NOTE_B4, 500, BUZZER_CHANNEL);
  noTone(BUZZER_PIN, BUZZER_CHANNEL);
}

In this example, a major C scale is played while holding each note for 0.5 seconds. The BUZZER_CHANNEL argument is optional. In the case of an ESP32, there are 16 PWM channels which can generate independent waveforms which need to explicitly assigned to any PWM-capable pin.

Here is the equivalent scale programmed using ledcWriteNote().

void loop() {
  ledcAttachPin(BUZZER_PIN, BUZZER_CHANNEL);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_C, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_D, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_E, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_F, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_G, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_A, 4);
  delay(500);
  ledcWriteNote(BUZZER_CHANNEL, NOTE_B, 4);
  delay(500);
  ledcDetachPin(pin)
}

The tradeoff here is that this is not cross-compatible with an Arduino, however, it means not having to import an external library or having to define note frequencies yourself. Similar to Arduino’s tone(), each signal is produced at 50% duty cycle. Under the hood, both implementations are calling ledcWriteTone.

References

https://randomnerdtutorials.com/esp32-pwm-arduino-ide/
https://github.com/espressif/arduino-esp32/issues/1720
https://community.platformio.org/t/tone-not-working-on-espressif32-platform/7587

SQL Injection Overview

2020-10-24T00:00:00+00:00

An SQL Injection occurs when untrusted input is used directly in the construction of an SQL query. This attack is commonly executed by introducing a meta character (such as a comment) into a data plane in such a way that allows an attacker to add commands to the control plane. Essentially, when building a SQL query from user input, an attacker can insert SQL instructions that cause the application to behave in unintended ways.

SQL Injection flaws are introduced when software developers create dynamic database queries that include user supplied input.

—OWASP SQL Injection Prevention Cheat Sheet

Authentication Form Example

Let’s take a login form with that requires a user enter a username and password to log in.

In the happy-path case, a user enters in their details, presses “Login,” and a request is sent to a server where we take what the user enters, search the database for that user, and then log that user into our web application.

# SQL Injection Vulnerable Pseudocode
# -- snip --

let user = ORM.query(
  "SELECT * 
    FROM users 
    WHERE username = '#{params['username']}'
    AND password = '#{params['password']}';"
).execute()[0];

# -- snip -- 

log_in_user(user)

Using a tool like SQL Fiddle, you can see how a query like this will behave in PostgreSQL 9.6. Here’s is the schema and seed data for our example:

CREATE TABLE users(
   id        INT PRIMARY KEY   NOT NULL,
   username  VARCHAR           NOT NULL,
   password  VARCHAR           NOT NULL
);

INSERT INTO users (id, username, password) 
VALUES (1, 'sarah', 'good-password');

INSERT INTO users (id, username, password) 
VALUES (2, 'thomas', 'password123');

INSERT INTO users (id, username, password) 
VALUES (3, 'patrice', md5('palm-kumquat-futon-padden'));

After the user submits the username:

thomas

and password:

password123

the resulting query will look like this:

SELECT * 
FROM users 
WHERE username='thomas' AND password='password123';

and the following results will be returned:

| id | username |    password |
|----|----------|-------------|
|  2 |  tcountz | password123 |

As expected, our backend code then takes this result, and calls the log_in_user() function which authorizes the user to access certain parts of the application.

Authentication Attack

The vulnerability here will allow an attacker to log in as any user.

Because user input is interpolated directly into the SQL query, we can have the server execute arbitrary SQL statements. Let’s take a look at an example that would allow us to log into any user, given we have their username.

After the user submits the username:

patrice'; --

and a blank (or arbitrary) password, the resulting query looks like this

SELECT * 
FROM users 
WHERE username='patrice'; --' AND password='<anything>'

and the results:

| id | username |                         password |
|----|----------|----------------------------------|
|  3 |  patrice | 26e9053a783f364d949b4e400dd2f68c |

Now, our application, again, takes the 0th results (this time the user patrice) and logs them in.

What Happened?

Even though we hashed Patrice’s password, our query was vulnerable to interpreting the ', ;, and -- PostgreSQL meta characters.

Firstly, the single quote: ', ends the string (or VARCHAR) that we’re searching for; in this case the username patrice. Next, the semi-colon ; represents the end of the SQL statement. Finally, the -- represents a comment and tells PostgreSQL to ignore everything that comes after.

This effectively makes our query look like this:

SELECT * 
FROM users 
WHERE username='patrice';

Which, as we’ve seen, and as we expect, returns the patrice user and logs them in. Now our attacker has been authenticated and has access to Patrice’s account!

This combination: '; --, and others like it, show up often in SQL injection attacks and it works by prematurely ending a SQL statement.

Data Destruction Attack

In the example above, the 0th row of the results returned from the query will be passed into the log_in_user() function, but the scope of this attack vector isn’t limited to logging in.

As an example of how we can attack the server to execute any SQL, take this example where we destroy the users table.

If we enter a username of:

'; DROP TABLE users; --

and a blank (or arbitrary) password, the resulting query looks like this

SELECT * 
FROM users 
WHERE username=''; DROP TABLE users; --' AND password='<anything>'

The user-facing effect of this query might not tell us exactly what has happened, but a developer might see something like this show up in the logs:

ERROR: relation "users" does not exist

What Happened?

Similar to the first attack, we’ve cut the original query short and this time, we’ve injected our own query to drop the users table.

Prevention

Parameterized Queries

The use of prepared statements with variable binding (aka parameterized queries) is how all developers should first be taught how to write database queries.

Parameterized queries force the developer to first define all the SQL code, and then pass in each parameter to the query later. This coding style allows the database to distinguish between code and data, regardless of what user input is supplied.

— OWASP SQL Injection Prevention Cheat Sheet

Our code was vulnerable because we use string interpolation to build an SQL statement directly from user input. Instead, we should “parameterize” our query by using whatever tools our language gives us to separate the data plane (input) from the control plane (SQL). This is the idea of using variable binding (placing user input into a type of variable) with prepared statements (the rest of the SQL that we don’t want the user to be able to alter).

The way to code this varies depending on the language you’re working with, so check out the OWASP SQL Injection Prevention Cheat Sheet section on parameterized queries.

Other Defenses

Another defense against SQL injection are stored procedures, which are predefined SQL statements stored in the data table. These procedures can have parameters and can effectively be similar to constructs from different languages. Read more here.

To read more about allow-listing or escaping user input, see the rest of the OWASP SQL Injection Prevention Cheat Sheet.

Next Steps

Research SQL injection prevention for your ORM/database
- e.g. https://guides.rubyonrails.org/security.html#sql-injection
Aim to separate queries and data (parameterized queries)
- e.g. https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Query_Parameterization_Cheat_Sheet.md
Consider SQL Injection when reviewing code

Code Review

Aim to validate user input by testing type, length, format, and range.
Avoid building SQL statements directly from user input.
Implement multiple layers of validation.
Avoid concatenating user input that is not validated; this is the primary point of entry for script injection.
You should review all code that calls execute(), exe(), and any SQL calls or commands that can call out outside resources or the command line.

Example Code Snippets

Ruby/ActiveRecord

# SQL Injection Vulnerable Ruby/ActiveRecord
# -- snip --

user = User.where(
    "username = #{params[:username]} AND " \
    "password = #{params[:password]}"
).first

# -- snip --

# SQL Injection Safe Ruby/ActiveRecord
# -- snip --

user = User.where(
    username: params[:username], password: params[:password]
).first

# -- snip --

Elixir/Ecto

# SQL Injection Vulnerable Elixir/Ecto
# -- snip --

query = """
  SELECT *
  FROM users
  WHERE username = \'#{params["username"]}\'
  AND password = \'#{params["pasword"]}\';
"""

user = Ecto.Adapters.SQL.query!(
    MyApp.Repo, query, []
)[:rows][0]

# -- snip --

# SQL Injection Safe Elixir/Ecto
# -- snip --

query = """
  SELECT * 
  FROM users 
  WHERE username = $1 
  AND password = $2;
"""

user = Ecto.Adapters.SQL.query!(
    MyApp.Repo, query, params["username"], params["password"]
)[:rows][0]

# -- snip --

Java

// SQL Injection Vulnerable Java
// -- snip --

String username = request.getParameter("username");
String password = request.getParameter("password");

String query = "SELECT * FROM users WHERE username = "
    + username + " AND password = " + password + ";";

Statement statement = connection.createStatement();
Object user = statement.executeQuery(query).getObject(0);

// -- snip --

// SQL Injection Safe Java
// -- snip --

String username = request.getParameter("username");
String password = request.getParameter("password");

String query = "SELECT * FROM users WHERE username = ? AND password = ?";
PreparedStatement pstmt = connection.prepareStatement(query);
pstmt.setString( 1, username);
pstmt.setString( 2, password);

Object user = pstmt.executeQuery().getObject(0);

// -- snip --

Resources

Dare to be Good Enough—You'll be Happier for it

2020-10-06T00:00:00+00:00

Originally published on 8th Light’s blog

As I sit down to write, I noticed a tug at the back of my neck—a tension forming as I begin editing my words obsessively. It’s a habit of writing characterized by my attempts at achieving the best possible outcome. “I know that I can write well,” I tell myself, “I’ve just got to strike the right tempo, tone, voice…”

I often have the same sensation when I’m writing code. The giants of our industry, whose shoulders I stand upon, tell me all sorts of things that should be true about my code—the way it reads, the way it’s coupled to other parts of a system, and the amount of complexity its allowed to have—it’s all qualified and quantified.

This tension, of balancing all of the small discrete decisions that crop up when implementing a software feature, causes me to try to painstakingly eke out the “correct” way to do things. It’s as if when faced with a choice, I scan through an endless tree of outcomes trying to game out what my best option would be. “Do I extract a shared setup for these tests, or do I duplicate it for each assertion?” “Under what circumstance should I choose the builder pattern over a factory method?” “Should this method be called is_ready or is_prepared?” The list goes on and on.

If this sounds familiar to you, you’re not alone. People who tackle decision-making in this way are called maximizers.

In order to determine their optimal decision outcome, maximizers feel compelled to examine each and every alternative available, which is often infeasible in reality due to the limitations in human cognition (Roets, et al., 2012).

In theory, maximizers make the best choices, but research shows that, in practice, maximizers aren’t ever sure that the choices they make are the best choices, and it’s this “not knowing” that leads to analysis paralysis, and ultimately, regret.

Satisficers, on the other hand, are happy with whatever “good enough” choice they make:

Several options may fall within a satisficer‘s threshold for acceptability, providing greater flexibility and latitude in achieving a desired decision outcome. As soon as [they encounter] a good enough option, the satisficer can easily ignore the addition of new choices to the decision domain (Peng, 2013).

Satisficers are pragmatists at their core; the solution that works is the solution they commit to.

This way of decision-making is embedded in the fabric of agile methodologies, which gives us permission to make small incremental changes over time, rather then grasping for the perfect solution from the start. From this, we know that iterative design can help us build tighter feedback loops, deliver value to stakeholders sooner, and ultimately keep our development reactive and responsive to the needs of our customers.

Our projects’ bottom line isn’t the only beneficiary to thinking like a satisficer. Even though the research is still out over whether maximizers or satisficers achieve objectively better decision-making (Peng, 2013), what is clear is that satisficers are happier. In fact, even if a better option occurs to them later, satisficers are less likely to experience regret (Schwartz et al., 2002).

If you find yourself like me, trying to game out the best future outcomes based on limited knowledge and with bounded cognition, remember the ways of the satisficers. Make a good enough decision today, and remain flexible and forgiving in order to make another good enough decision tomorrow. Of course, we all have metaphorical hills on which we may choose to die, but choose your metaphorical battles wisely and dare to be good enough.

References

Roets, A., Schwartz, B., & Guan, Y. (2012). The tyranny of choice: a cross-cultural investigation of satisficing effects on well-being. Judgment and Decision Making.

Peng, Starry. “Maximizing and Satisficing in Decision-Making Dyads.” ScholarlyCommons, 2013, https://repository.upenn.edu/cgi/viewcontent.cgi?referer=&httpsredir=1&article=1101&context=wharton_research_scholars

Schwartz, B., Ward, A., Monterosso, J.,Lyubomirsky, S., White, K., & Lehman, D. R. (2002). Maximizing versus satisficing: happiness is a matter of choice. Journal of personality and social psychology.

Agile Alliance. “What is Incremental Development?”, 2019, https://www.agilealliance.org/glossary/incremental-development

Specify Bundler Version

2020-09-18T00:00:00+00:00

If you’re like me, you may have versions of both Bundler 1 and Bundler 2 installed on your system. This can make it difficult to manage different codebases.

Say one requires an older version, like 1.13.6. You can run the following command to install it,

gem install bundler -v 1.13.6

but when you install gems,

bundle install

you might end up seeing something like

BUNDLED WITH
   2.1.4

in your Gemfile.lock file… which is not the same as bundling with 1.13.6.

You can tell Bundler that you’d like to use a specific version of Bundler by specifying the exact version number before the command you which to run. Like so:

bundle _x.x.x_ install

To install gems for your project using Bundler 1.13.6, you can use this command to force Bundler to use the correct version:

bundle _1.13.6_ install

And you should see

BUNDLED WITH
   1.13.6

at the end of your Gemfile.lock file.

Horray!

Types of Notetaking

2020-08-30T00:00:00+00:00

Earlier this week, my pair and I struck upon something interesting, and I stopped to take notes.

Every time I see you taking notes, you’re using a new method!

— Pair

She’s not wrong. Over the past several years, I’ve tried many different ways of capturing my thoughts: yellow legal pads, Basecamp, Notion, Vimwiki, Org Mode, Tyke, index cards, Gitbook, blogging with Jekyll and Medium, Slack/Emails to self, Notability, and voice notes. I didn’t switch note taking styles because I liked to try new things, I switched because I hadn’t found a note-taking tool that fit all of my needs. I’ve researched and tried different note taking systems like Getting Things Done and Building a Second Brain, but as soon as a system or tool caused me to think more about note taking and less about what I was taking notes on, I struggled to keep focused and motivated.

I realized that I should drop all of that and let my intuition guide how I record information best. It turns out that trying to force everything I wanted to write down into one system caused me to feel frustrated and confused. For some things, Vimwiki was perfect! For others, it felt clunky and difficult. A pocketbook was awesome for a while, then suddenly I felt like I could never keep things straight.

Once I stopped fighting this battle, I found that there were three types of notes that I was taking and therefore there were three different tools/methods I could use to take them.

Actionable TODOs

In Situ Documentation

Long-form Reference

Actionable TODOs

Buzz. Buzz.

I check my phone. It’s a text from my husband:

If you stop by the grocery store, I just used the last few onions. Can you grab some?

TODOs and short-term tasks come at me from all directions and at any time. Old me would just try to remember everything, but I’ve realized just how much bandwidth and brain cycles that takes up, so I’ve taken to writing these things down. I write down things that are “actionable,” and are “obvious when they’re done,” like “take out trash,” or “fill out form DL-80.”

My criteria for a TODO tool:

Readily available — I need access to my TODO list at all times. I don’t know when I’ll have the opportunity to check something off my list and I don’t know where I’ll be when I need to add a new task.
Low friction — Capturing a TODO needs to be a quick activity so that I can continue on with my day. Having to also schedule a reminder, add a category, a priority value, and a description, before I could “save” a task limited my willingness to commit to writing down my TODOs
Non-ceremoneous — similar to being low friction, I don’t want any other features tied to my TODO notes besides creating a task and marking as task a finished. For example, one TODO tool generated a weekly chart displaying how many tasks I completed compared to last week. I started re-structuring my TODOs to effect these outcomes and meant that I was no longer focusing on the tasks themselves.

My solution: a passport-sized notebook and pen

I can carry this notebook and pen most places with me, and I’m un-apologetic about what I write in it. I mix work-related things with personal task, and I scribble quick “…can you write down this number…” notes, too. Even though it’s for “tasks,” I occasionally find myself taking temporal notes like, “finished reading Convenience Store Woman” and ideas like “write blog post about note taking.”

A pocketbook is an always-charged smartphone.

In Situ Documentation

Do you remember what we ended up deciding on Friday?

In a world of synchrounous-first communication, after an hours long meeting, the last thing I want to do it write down everything we just talked about. More than that, after an hours long meeting, the last last thing I want to do is feel like we forgot everything we talked about.

Sometimes the meetings I’m in are a bit chaotic and cover a lot of ground. (Often more like mob programming than a meeting). In these cases, I’ve found that rapid-fire note taking has allowed me to both 1) keep a reference of what we may have decided and 2) free up my brain to keep listening.

Contemporaneous Work Notes look like this for me:

Draft-zero — I think of these notes as casting a very wide net. I scribe almost continuously and more often than not, half of it is garbage. With that said, I need a tool that can help me create private unedited drafts that allow me to hone the 50% later.
Adaptable — I might be taking notes during a meeting, while pairing, during code review, etc. I need a system that adapts to however my brain feels like organizing information in the moment. Besides a date, these notes can end up looking very different: a doddle here, a line reference there, a TODO might sneak in—who knows?!
Disposible — These notes depreciate in value over time. Ideally, I reference these notes shortly after taking them in order to formalize something that may need to be shared. When I’m done, the in situ notes shouldn’t contain any important information that isn’t captured in a different form elsewhere.

My solution: yellow legal pad and mechanical pencil*

Yellow legal pads are beloved (and hated) by many for their accessibility and ease of use, see: The Illustrious History of the Yellow Legal Pad. I fell in love with them because they feel limitless. I found that when taking notes in beautiful leather-bound $40 notebook, I became a bit too precious about what I would write in it and began to edit my notes before I put pencil to paper! On the contratry, yellow legal pads are cheap, thin, and always open. I prefer a mechanical pencil because I feel as though I can write cleaner lines with a 0.5mm lead than I can with an equally fine pen. Also, I do erase things sometimes, especially if I’m trying to capture a concept in an illustration.

A yellow legal pad is a private portable whiteboard.

*In an effort to minimize my impact on the environment, I’ve moved to using Notability on the iPad with an Apple Pencil or folding yellow legal pad pages into four—using one section per day and both sides of the page.

Long-Form Reference

CUT TO: Thomas is, yet again, digging through the pages of Building Git by James Coglan to remember how Ruby’s open3 module exposes child processes to IO streams.

I love diving deep into different topics, often topics that I may not directly visit or use again in my day-to-day work. Take machine learning for example. When I first studied it, I learned quite a lot! But, just a few months later, when I revisited data science, I felt as though I was back to square one. Not only can I have notes to come back to, I’ve noticed that my comprehension increases when I’m able to write new information down in my own words.

My solution to avoid this type of thing is to take long-form reference notes:

Reference-able — As you might be able to guess, these notes have to be notes that I can come back to and understand. That means that these aren’t draft-zero notes, they’re well crafted and continuously editable as I learn and discover new things. They’re written with the reader in mind, even on the first pass.
Structured — These notes should be both hierarchical and linkable. If I taking notes on material that is related to notes I’ve taken in the past, I want to connect them somehow. This helps with referencing things later, but also helps as I’m editing.
Pleasurable — It’s important to me that these notes be enjoyable to write and to read. This means the ergonomics have to be comfortable and familiar and help me to focus on the content, rather than the styling.

My solution: Gitbook/personal wiki

This is a new venture for me, but you can find the beginnings of this journey at research.thomascountz.com. Gitbook has switch from self-hosted to a SAAS offering in the past few years, but still offer free services for individuals. I enjoy Gitbook because the online editor is intuitive and comfortable, and all of my notes are backed by git/Github, which means exportability. Also, I like that these notes are public. They’re not quite a blog post, but being able to link directly to specific notes is helpful when sharing ideas.

I’ve also taken to recording personal project documentation in wiki format. Even when I’m working alone, I prefer to write asynchronous documentation as though I’m together with a team. These project docs help me link thoughts and ideas togethers that would otherwise decay over time as other priorities come into focus.

A personal wiki is the brain’s external hard drive.

Your Notes, Your Way!

It’s with this combination of a small notebook, yellow legal pad, and personal wiki that I declutter my brain and keep focused. What’s your preferred note-taking method/tool? Do you have one that fits all your needs or are you like me: different tools for different situations? I’d love to hear your thoughts!

FastBook Chapter 4 Questions & Notes

2020-05-09T00:00:00+00:00

Fastai, known for it’s MOOCs, is working on a book, Fastbook to go along with their new MOOC starting July 2020. In my eagerness, I’ve been going through the draft of the book (linked above, though they may remove it after publication) and have been coding alongside on Kaggle. At the end of each chapter of the book is a list of questions for the reader/students to answer. I’ve found these questions to be rigorous and useful to deepen my understanding.

This blog post might not be useful to anyone besides myself, but I want to keep my answers to these questions as a reference somewhere other than a Kaggle notebook.

How is a greyscale image represented on a computer? How about a color image?

A greysale image is represented by a matrix/rank 2 tensor/grid of pixels with a value between 0 and 255. Color images are a rank 3 tensor, height and width along two dimensions and third dimension representing values between 0 and 255 for redness, greenness, and blueness.

How are the files and folders in the MNIST_SAMPLE dataset structured? Why?

There are two directories: /train and /valid, which each contain two directories: /3 and /7. Inside those directories are images of the respective digit. The train/ directory contains the majority of images to be used to train a model likewise the /valid directory contains images to validate a model. There’s also a labels.csv file which likely contains a mapping between file names and digit label.

Explain how the “pixel similarity” approach to classifying digits works.

The idea behind this approach was to look at all of the images for a given digit and compute the average value for each pixel across all images. Then, by comparing an unlabeled digit to that “average” digit, we could determine how similar or dissimilar the unknown image was to the known average.

What is a list comprehension? Create one now that selects odd numbers from a list and doubles them.

It’s a way of mapping over an enumerable object in Python

[i * 2 for i in range(20) if i % 2 == 0]

What is a “rank 3 tensor”?

A rank 3 tensor is a tensor with three dimensions, such as a color photo.

What is the difference between tensor rank and shape? How do you get the rank from the shape?

A tensor’s rank is equivalent to the number of dimensions a tensor has. It’s shape is how many values exist in each dimension. Because the shape tells us the number of values per dimension, we can determine the number of values returned to determine the rank.

What are RMSE and L1 norm?

RMSE, or root mean square error is the average difference of the prediction and the label squared and then 2nd rooted?

def mse(y, yhat): ((y - yhat)**2).mean().sqrt()

L1 norm is the average of the absolute value of the difference between y and yhat

def l1(y, yhat): (y - yhat).abs().mean()

How can you apply a calculation on thousands of numbers at once, many thousands of times faster than a Python loop?

Broadcasting.

Create a 3x3 tensor or array containing the numbers from 1 to 9. Double it. Select the bottom right 4 numbers.

(torch.reshape(torch.arrange(1, 10), (3, 3)) * 2)[-2:,-2:]

What is broadcasting?

Broadcasting is traditionally used to support computation between two unequal tensor with the side effect of not needing to copy any data. This side effect can be taken advantage of for applying calculation to two tensors even if they are equal in rank.

This happens implicitly in Pytorch when operating of two vectors, for example:

Non-broadcasting: slow

>>> a = [1, 1, 1]
>>> b = [1, 1, 1]
>>> c = [x + y for x, y in zip(a, b)]
[2, 2, 2]

Broadcasting: fast

>>> a = tensor([1, 1, 1])
>>> b = tensor([1, 1, 1])
>>> a + b
tensor([2, 2, 2])

Are metrics generally calculated using the training set, or the validation set? Why?

A metric is a function that measures quality of the model’s predictions using the validation set, and will be printed at the end of each epoch. —Fastbook, Ch. 4

We use a metric to judge our network’s ability to accurately predict outputs for data is hasn’t seen before.

What is SGD?

Gradient Descent is the process by which we update the weights/parameters/coefficients of our model in order to minimize the loss function. By taking the gradient/derivative of our loss function, we can determine how a change in our parameters would result in a change to the output. We can use that gradient to update our weights in proportion to the learning rate. Stochastic just means measure loss and update our weights in in “batches.” Some definitions say that “Stochastic” means we calculate and update for the entire training set.

Why does SGD use mini batches?

Updating parameters after every training example takes a long time and can mean sporadic jumps in parameters after each example as the network tries to optimize for each individual example. Waiting until after going through the entire dataset can be impossible for large datasets that don’t fit into memory. It’s inefficient and doesn’t necessarily provide better results. Mini-batches solve both of these problems. We reduce the variance between each parameter update which can smooth out convergence, and we don’t run into memory issues waiting for the entire training to complete.

What are the 7 steps in SGD for machine learning?

Initialize weights, Predict, calculate loss, determine gradient, update parameters, repeat and stop.

How do we initialize the weights in a model?

Randomly. There are other methods, but randomly is a good starting point.

What is “loss”?

Loss is the function that tells us how well or how poorly our model predicted an output for a example.

Why can’t we always use a high learning rate?

The learning rate is what we use gently adjust the parameters. A high learning rate can cause our loss to jump around sporadically as we attempt to minimize it.

What is a “gradient”?

The gradient of a function, denoted as follows, is the vector of partial derivatives with respect to all of the independent variables, aka, the parameters.

The derivative, it’s a function that describes the slope of another function at a given point. It tells us how “quickly” a function changes at a certain input.

To calculate the partial derivative of a single parameter, you hold all other parameters constant. After computing all of the partial derivatives, they’re collected into a vector call the gradient.

Do you need to know how to calculate gradients yourself?

Nope. Thanks Pytorch!

Why can’t we use accuracy as a loss function?

Accuracy is for humans to consume. It tells us how well a model is at prediction examples that it has never seen before (in the validate set) overall. Accuracy isn’t necessarily a function from which we can calculate a gradient/derivative in order update our weights. In the MNIST model in this chapter, a small nudge in a parameter won’t necessarily effect accuracy, unless that small nudge changes a prediction from a 0 to a 1 or vice versa.

Draw the sigmoid function. What is special about its shape?

It looks like an S on it side with horizontal asymptotes a y=0 and y=-1. The smooth curve makes it special because it helps us to gently calculate derivatives of our loss.

What is the difference between loss and metric?

A we use a metric to determine human-interpretable accuracy, the model uses loss to determine how to update weights.

What is the function to calculate new weights using a learning rate?

In this example: Stochastic Gradient Descent

What does the DataLoader class do?

DataLoader will return an (x, y) tuple for our model, divide the data into training and validation sets, and create mini batches

Write pseudo-code showing the basic steps taken each epoch for SGD.

prediction = model(x, params)
loss = loss(prediction, label)
loss.backward()
for p in params:
    p.grad.data += lr * p.grad.data
    p.grad.data = None

Create a function which, if passed two arguments [1,2,3,4] and ‘abcd’, returns [(1, ‘a’), (2, ‘b’), (3, ‘c’), (4, ‘d’)]. What is special about that output data structure?

def f(a, b): return list(zip(a, list(b)))

It returns a list of tuples.

What does view do in PyTorch?

From the docs:

PyTorch allows a tensor to be a View of an existing tensor. View tensor shares the same underlying data with its base tensor. Supporting View avoids explicit data copy, thus allows us to do fast and memory efficient reshaping, slicing and element-wise operations.

For example, to get a view of an existing tensor t, you can call t.view(…).

>>> t = torch.rand(4, 4); t
tensor([[0.2108, 0.4824, 0.4418, 0.9436],
        [0.9554, 0.5866, 0.7631, 0.2809],
        [0.2934, 0.7608, 0.7741, 0.6948],
        [0.0813, 0.5682, 0.8023, 0.3858]])
>>> b = t.view(2, 8); b
tensor([[0.2108, 0.4824, 0.4418, 0.9436, 0.9554, 0.5866, 0.7631, 0.2809],
        [0.2934, 0.7608, 0.7741, 0.6948, 0.0813, 0.5682, 0.8023, 0.3858]])
>>> t.storage().data_ptr() == b.storage().data_ptr()  # `t` and `b` share the same underlying data.
True
# Modifying view tensor changes base tensor as well.
>>> b[0][0] = 3.14
>>> t[0][0]
tensor(3.14)

What are the “bias” parameters in a neural network? Why do we need them?

Bias terms are additional constants attached to neurons and added to the weighted input before the activation function is applied. Bias terms help models represent patterns that do not necessarily pass through the origin. For example, if all your features were 0, would your output also be zero? Is it possible there is some base value upon which your features have an effect? Bias terms typically accompany weights and must also be learned by your model. —https://ml-cheatsheet.readthedocs.io/en/latest/nn_concepts.html?highlight=bias#bias

What does the @ operator do in python?

Matrix multiplication ( * is element-wise multiplication)

What does the backward method do?

loss.backward() computes dloss/dx for every parameter x which has requires_grad=True. These are accumulated into x.grad for every parameter x. In pseudo-code:

x.grad += dloss/dx

https://medium.com/@zhang_yang/how-pytorch-tensors-backward-accumulates-gradient-8d1bf675579b

Why do we have to zero the gradients?

See link above, pytorch will accumulate the gradient from all operations.

What information do we have to pass to Learner?

The DataLoader that contains the data, the model, the optimization function (e.g. SGD), loss functions (e.g. MSE), and any metrics to print

learn = Learner(dls,
                nn.Linear(28*28,1),
                opt_func=SGD,
                loss_func=mnist_loss,
                metrics=batch_accuracy)

Show python or pseudo-code for the basic steps of a training loop.

for _ in range(epoch):
    prediction = model(x, params)
    loss = loss(prediction, label)
    loss.backward()
    for p in params:
        p.grad.data += lr * p.grad.data
        p.grad.data = None

What is “ReLU”? Draw a plot of it for values from -2 to +2.

A ReLU, or rectified linear unit, replaces every negative number with 0

What is an “activation function”?

I understand an activation function as being a non-linear layer in a neural network. This has the effect of allowing or preventing a certain “neuron” from activating depending on a threshold. For a ReLU, only positive values in a previous layer would be passed forward to the next layer. Similarly, a unit step function only passes values greater than 0 to the subsequent layers, but all positive values are passed on as 1.

What’s the difference between F.relu and nn.ReLU?

F.relu is the function, nn.ReLU is the Pytorch module module.

The universal approximation theorem shows that any function can be approximated as closely as needed using just one nonlinearity. So why do we normally use more?

Efficiency and performance. With deeper models, we don’t need as many parameters. Smaller matrices with more layers > larger matrices with fewer layers.

MLOps for Web Devs - Fastai2, Flask, Docker, & Heroku

2020-04-30T00:00:00+00:00

You’ve built your fastai2 model and now you want to expose it via an API. There are tools like Render, Google App Engine, AWS Lambda, Amazon SageMaker, and Microsoft Azure Functions that you may use in different production scenarios, but today, we’ll use Docker and Heroku.

Try out my model, below!

Get the code on Github, here!

I use this stack because, as a web developer, I’m already familiar with most of it. Heroku has a generous free-tier container registry and runtime, Flask gives me control over the API with very little boilerplate, and, I’m not vendor locked—If I don’t like Heroku, I can deploy my Docker container anywhere.

With that said, there are some prerequisites for getting the most out of this article:

You’ve already built your fastai2 model (check out fastai for their free courses!)
You have an account and have deployed to Heroku (not necessarily with their container registry)
You are familiar with Docker
You are familiar with Flask/web API frameworks

Flask

To begin, we’ll build a Flask app that wraps our model’s inference in an API.

# app.py

import os
from flask import Flask, request, make_response
from werkzeug.utils import secure_filename
from fastai2.vision.all import *

UPLOAD_FOLDER = 'tmp'
ALLOWED_EXTENSIONS = {'png', 'jpg', 'jpeg', 'gif'}

app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
learner = load_learner('export.pkl')

def allowed_file(filename):
    return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS

@app.route('/ping')
def ping():
    return {'success': 'pong'}, 200

@app.route('/predict', methods=['POST'])
def predict():
    if 'image' not in request.files:
        return {'error': 'no image found.'}, 400

    file = request.files['image'] 
    if file.filename == '':
        return {'error': 'no image found.'}, 400

    if file and allowed_file(file.filename): 
        filename = secure_filename(file.filename)
        filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)
        file.save(filepath)
        prediction = learner.predict(filepath)
        return {'success': prediction[0]}, 200

    return {'error': 'something went wrong.'}, 500

if __name__ == '__main__':
    port = os.getenv('PORT',5000)
    app.run(debug=True, host='0.0.0.0', port=port) 

Let’s go through this, together.

First, we import some required modules. From os, we’ll be able to do things like get environment variables and use filepath utilities. From Flask, we’ll get what we need to use the framework. wekzeug.utils gives us a way to create a secure file name from a file uploaded by the user. Finally, we need fastai2, which I’ve used to build a model to tell if a picture contains a mourning dove, a sparrow, or a pigeon.

After defining some environment variables and Flask configs, we load our model into memory by way of fastai2’s load_learner() function, which takes in the file path of the .pkl export of our model.

learner = load_learner('export.pkl')

This part is interesting because normally, a fastai2 “pickle-ed” model file would end up being too large to use with Heroku and you’d encounter an error like this:

Compiled slug size: 520.5MB is too large (max is 500MB).

In fact my .pkl file alone is 78MB and I had to use git-lfs to manage it effectively with version control.

du -sh pigeon/* | sort -hr
 78M	../pigeon/export.pkl
1.4M	../pigeon/tmp
 48K	../pigeon/Pipfile.lock
4.0K	../pigeon/requirements.txt
4.0K	../pigeon/app.py
4.0K	../pigeon/__pycache__
4.0K	../pigeon/Pipfile
4.0K	../pigeon/Dockerfile

⚡ I didn’t know about Heroku slug size limitations before this. Heroku compresses and pre-packages your application in order to optimize its ability to scale and expand it when needed.

I thought this meant that it would be impossible to host my model on Heroku without offloading the .pkl file to external storage, until I read this from the Container Registry & Runtime (Docker Deploys) Heroku documentation:

While Docker images are not subject to size restrictions (unlike slugs), they are subject to the dyno boot time restriction.

⚡ So, if we Docker-ize our Flask app, it means that we don’t have to worry about how large our .pkl file is, as long as it binds to the assigned $PORT within 60 seconds!

Next in our Flask app, we define a helper method for determining which files are allowed to be uploaded by the user, and define a /ping route to check for app health.

Then comes the bulk of the API: the /predict route.

@app.route('/predict', methods=['POST'])
def predict():
    if 'image' not in request.files:
        return {'error': 'no image found.'}, 400

    file = request.files['image'] 
    if file.filename == '':
        return {'error': 'no image found.'}, 400

    if file and allowed_file(file.filename): 
        filename = secure_filename(file.filename)
        filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)
        file.save(filepath)
        prediction = learner.predict(filepath)
        return {'success': prediction[0]}, 200

    return {'error': 'something went wrong.'}, 500

Within the error handling that checks that a POST request with an image field had made it to our endpoint with some data, we start the inference process.

First, we create a new filename based on the name of the file that the user uploads. (The secure_filename() function is a security measure that I invite you to read more about). Next, we create a filepath by concatenating our new filename with the UPLOAD_FOLDER name, and then we save the file to our sever.

⚠️ In this case, Docker containers on Heroku behave the same way as slugs do: files are saved in an ephemeral filesystem. This means that files only live as long as the dyno process does—once it’s restarted or stopped, the files are destroyed. This can be good for pet projects like this, but a more robust solution might be to upload to external file storage.

After that, we call the learner.predict() function and pass in the filepath of our user’s uploaded image. Finally, we take the first element of the prediction, (which in this case is the label name), and return it as JSON. If you’ve already built your fastai2 model, the inference part should be familiar to you. If you’ve worked with the web before, JSON is probably second nature.

Lastly, if all of the if-checks have failed, we return a 500 Internal Server Error, with our favorite error message.

We should be able to run our Flask app locally by using the python CLI:

$ python app.py

Test it out by using curl to POST to localhost:5000 and send an image field:

$ curl -F 'image=@/Users/thomas/Pictures/sparrow.jpg' localhost:5000/predict
{
  "success": "sparrow"
}

Docker

If it wasn’t for Heroku’s slug size restrictions, this would have been almost the end. You’d heroku push this up and your new ML app would be up and running. However, after of our discussion earlier, we know that we want to use Docker to encompass our application’s runtime environment so as to avoid Heroku’s slug size restrictions.

Here’s a minimal Dockerfile with a little bit of a twist on how I’ve handled dependencies…

FROM python:3.6-slim-buster
RUN pip3 install pipenv
COPY Pipfile* /tmp/
RUN cd /tmp/ && pipenv lock --requirements > requirements.txt
RUN pip3 install -r /tmp/requirements.txt
COPY . /app/
WORKDIR /app/
ENTRYPOINT ["python"]
CMD ["app.py"]

Let’s take a look, together.

FROM python:3.6-slim-buster

I’m not a Docker image expert—as far as I’m concerned, if you don’t want to build your own image, you can just find the smallest one that works for you. Itamar Turner-Trauring has written (and continually updated) The best Docker base for your Python application that I think provides some tips on how better to choose an image.

RUN pip3 install pipenv
COPY Pipfile* /tmp/
RUN cd /tmp/ && pipenv lock --requirements > requirements.txt
RUN pip3 install -r /tmp/requirements.txt

This is a bit of a hack to allow me to define requirements/dependencies in just one place: a Pipfile that I use for managing my development environment with pipenv. Itamar Turner-Trauring has also written about this hack in their article Faster Docker builds with pipenv, poetry, or pip-tools and these lines are taken verbatim from their advice. Essentially, we install pipenv in order to extract a requirements.txt file from the already defined Pipfile, before using pip3 to install our dependencies.

COPY . /app/
WORKDIR /app/
ENTRYPOINT ["python"]
CMD ["app.py"]

These last lines are standard operations: copy all of our files to the container (this includes our .pkl file), change our working directory, then run our Flask app using the python cli.

⚠️ Defining an ENTRYPOINT and a CMD means that our ENTRYPOINT argument will always be used, but our CMD can be overwritten by commandline arguments. This isn’t necessary, and in this example, we could have defined only an ENTRYPOINT so that our container would always be run as an executable without the option of being overridden.

That’s all for our Dockerfile. We can now build and tag our image:

$ docker build --tag pigeon:latest .

And then run it locally in the background:

$ docker run --detach --publish 5000:5000 pigeon

Our Flask app defaulted to using port 5000, so without specifying a different PORT environment variable, we can bind, or publish, that default port to our local machine’s port 5000.

Now you should be able to send a POST request to localhost:5000 just as before:

$ curl -F 'image=@/Users/thomas/Pictures/sparrow.jpg' localhost:5000/predict
{
  "success": "sparrow"
}

Heroku

Now that we’ve captured our model into a Flask app and Docker container, we can take it and deploy it in many different places, but I like Heroku best—I’ve deployed many apps with it, it’s generously free, and it works well.

⚠️ Make sure you check with Heroku’s Free Dyno Hours documentation to understand what the limitations are. Most notably, free web dynos (which our Flask app is one) will “sleep” after 30 minutes of inactivity. This means that you may experience latency when your dyno cold starts.

Heroku’s Container Registry & Runtime (Docker Deploys) documentation covers the steps on how to deploy your Docker container to their registry and runtime.

First, make sure you’re logged in to the heroku CLI

$ heroku login

$ heroku container:login

Navigate to the app’s directory and create a Heroku app:

$ heroku create

Creating shielded-eyrie-48894... done, stack is heroku-18
https://shielded-eyrie-48894.herokuapp.com | https://git.heroku.com/shielded-eyrie-48894.git

Build the image and push to Container Registry:

$ heroku container:push web

Then release the image to your app:

$ heroku container:release web

Now open the app in your browser:

$ heroku open

If you created a Flask app similar to mine, you may not have defined an index page (/ route), in which case, calling heroku open will open your browser to a 404 Not Found page.

Instead, you can curl the /ping route:

$ curl https://shielded-eyrie-48894.herokuapp.com/ping
{
  "success": "pong"
}

And call your /predict endpoint, just as before:

$ curl -F 'image=@/Users/thomas/Pictures/sparrow.jpg' https://shielded-eyrie-48894.herokuapp.com/predict
{
  "success": "sparrow"
}

Conclusion

Hopefully this article answered the question: “what do I do after fitting my model?” The reason that I like this stack in particular is because it parallels what I’m familiar with as a web developer—the fact that the API happens to loads a .pkl file is trivial thanks to fastai2. Also, with a Dockerfile, I’m not locked into the proprietary solutions that MLOps is building towards.

Is this stack production-ready?

Not quite. Heroku’s free-tier isn’t recommended for production, but their paid options are! Likewise, many other container registries offer similar solutions. Like all architectural decisions, your mileage will vary depending on your needs. If you’re building a system to recommend blog posts on your site, this might be enough! If you’re trying to erect mission-critical systems that will embed within your decision-making apparatus, probably not.

Try it Out!

As mentioned above, my app is on a free-tier Heroku dyno, so you might need to wake it up before trying it:

$ curl https://shielded-eyrie-48894.herokuapp.com/ping
{
  "success": "pong"
}

Now, test it out with these images, or upload your own!

⚠️ This model is probably not going to infer all of your images very accurately. It was trained on a tiny dirty dataset, but it worked for illustrating the deployment pipeline in this article. Secondly, I miss labelled mourning doves as morning_dove, sorry!

$ curl -F 'image=@/Users/thomas/Pictures/sparrow.jpg' https://shielded-eyrie-48894.herokuapp.com/predict
{
  "success": "sparrow"
}

Resources

Precommitments

2020-02-13T00:00:00+00:00

Originally Published in The Pseudocode 001

When John O’Nolan founded Ghost in April 2013, he made the same “we-won’t-sell-out” pitch to customers that every other startup was making at the time:

“We’re building Ghost because we want online publishing to be better. We want to make decisions focused on that goal, not on building a startup that we can sell to Facebook for $1billion.” — Ghost, Kickstarter Campaign

But for Ghost, this was more than just some cheap corporate-speak to woo consumers. John and his team took drastic measures to ensure that they would remain committed to this promise: after the crowdfunding campaign became a success, Ghost was founded as a non-profit. That meant that Ghost’s founders were, and forever will be, legally locked-out of owning any part of their business. Even if Facebook in theory wanted to buy Ghost, it would now be impossible.

As it turned out, just two years later, that very theory was put to the test.

“In 2015, I was in San Francisco and we had some investors trying to convince us to do a series A and we were like, ‘We’re a non-profit, we can’t,’ and they were like, ‘We’ve got $10 million and we really think that you can go far,’ and we were like, ‘We’re a non-profit, we can’t.’” — The Indie Hackers Podcast Episode #139

John’s story is an extreme example of the power of precommitments. Precommiements help us make calm decisions today, in order to prevent us from making rash ones later.

At work, the same concept can be used to define boundaries between what our jobs ask of us and our own priorities.

A few of my personal boundaries related to work are:

I work up to 40 hours/week
I read/respond to email between 8am - 6pm U.S. Eastern Standard Time
During vacations, I will only engage in personal communication

When boundaries are not defined, it can be difficult to be principled about them. As simple as some boundaries may seem, it’s important to articulate them both as precommitments to ourselves and as expectations that others can have of us.

“Your resolve waivers, no matter how much you believe something. Having the safeguard of, ‘I’ve already made this decision and I’ve… locked myself into this decision…”’I’ve never been more grateful for [it].” — The Indie Hackers Podcast Episode #139

Boundaries are malleable and may change over time in good-faith negotiation with ourselves, our partners, or our workplaces. What’s important, before you make commitments to others, evaluate your priorities and make precommitments to yourself.

Why I Share the Agenda

2020-02-01T00:00:00+00:00

Originally Published in The Pseudocode 001

Meetings, meetings, meetings! A day chopped up into a series of fits and starts where nothing gets finished and we all end up fatigued. (Sound familiar?) Although I long for a world where everyone can work at their own pace, for many of us, days like these are unavoidable.

Having been in charge of meetings on days like these, here’s what I try to do to make them better for everyone: Share the meeting’s agenda with everyone before you ask for their time.

A meeting agenda isn’t just a schedule—when I send a meeting invite to someone, I make it super clear why I’d like them to be there and if I need them to prepare anything. By sharing the agenda with a colleague, I let them decide if their presence is actually needed. Time is money, after all, and I wouldn’t mindlessly ask my colleagues for their money.

In the agenda, I include my best guess at the timing for the meeting, and during the meeting, I stick to it. Of course, when the ball gets rolling, it can be difficult to tell everyone to switch gears. To combat this feeling, I use this exact phrasing:

Out of respect for everyone’s time, it’s now 2:10pm and we planned to move on to ___. Any objections to moving on?

I do this to perpetuate a team culture in which meetings are not allowed to hold everyone hostage. If I run a meeting past schedule, then I’ve done a poor job planning; this is my problem to solve, and I’ll have to solve it on my own time (not while everyone is stuck in a room together).

One last thing, by sharing the agenda, I’m forced to write one. Placing this mindful barrier around scheduling a meeting in the first place, can help avoid them altogether. By the time I’ve clearly stated what I need from someone, they simply send me an email, and voilà—no meeting required.

Third-Party Cookies

2020-01-13T00:00:00+00:00

As internet privacy continues to rise in our collective consciousnesses, cookies are caught in the cross-hairs. Cookies aren’t always bad, but they can be used to identify you, track your behavior online. Let’s take a look at the technical side of cookies and what makes them so useful for good and bad actors, alike.

Request-Response Cycle

When you visit a website on the internet, your browser sends a request to a server. Although there are many stops along the way, a request is eventually routed to a server that houses the files you wish to see in your browser window. This is called the request/response cycle.

We call this server the “first-party server,” because it is the place where the content that the user has requested is stored—this is the server where the user is expecting to get a webpage from.

First-Party Cookies

First-party servers sometimes return additional data besides the content that a user has requested, like cookies, for example. They do that by setting the Set-Cookie header in response to the request from your browser. After a cookie has been set in a browser, that browser will send back that cookie with every subsequent request to the server.

According to the Electronic Frontier Foundation, or EFF, browser cookies “…are a web technology that let websites recognize your browser. Cookies were originally designed to allow sites to offer online shopping carts, save preferences, or keep you logged on to a site.”

As an example, after you log into a website, like Facebook, with a username and password, a unique cookie is set in your browser. This unique string is used to identify your browser on every subsequent visit to the site, that way, you don’t need to keep on logging in with your username and password.

As another example, if you visit an online shopping site and place things in your cart and then leave, those items may still be there if you later return the site—this can happen without having to login or create an account.

This is because your browser, after receiving a cookie from a particular server, will send that same cookie back to the server on every next request, until that cookie expires or is deleted by the user.

These types of cookies are mostly benign and are not considered “trackers.” First-party severs can use cookies to monitor your behavior/save your preferences only when you’re interacting with that particular server. That isn’t to say that there aren’t security implications to using cookies for authentication, but in terms of privacy, the impacts can be mild.

Third-Party Cookies

Privacy becomes a concern when cookies are used to track you outside of a particular website and across the internet. Cookies often become trackers when your browser, unwittingly, interacts with third-party servers.

In this example, your browser makes a request to a first-party server, just like before, but then the first-party server causes your browser to make (sometimes hundreds of) additional requests to servers that may have nothing to do with the content that you’re looking for. When this happens, these third-party servers can set cookies just like first-party servers.

If you remember, your browser only sends back cookies in requests to the servers that originally set them, however if two different first-party servers, say server A and server B, cause your browser to send a request to the same third-party server (like Google Analytics), that third-party server now knows that you’ve visited both server A and server B. This third-party server is now tracking you across different websites and can build a profile about the kinds of sites that you visit.

Besides cookies, there are additional pieces of identifiable information that your browser sends to a server when making a request. This data, in combination, can build a strong profile about your behavior on the internet, and it’s this profile that advertisers use to target ads to you.

How Website Owners Can Prevent Users From Being Tracked

Often websites don’t know that they’re aiding advertisers by allowing their site visitors to be tracked, monitored, fingerprinted, and profiled. Adding a social network “share” button, a visitor counter/analytics tool, or even deciding to use certain fonts can cause a website’s users to become tracked or their existing profiles added to.

Website developers often use tools & resources built by companies which support major ad networks, like Google, Facebook, and Amazon, but slowly, more an more privacy-conscious alternatives are being developed, like Font Squirrel for fonts, and TinyFeather, for analytics.

TinyFeather provides the kinds of analytics that businesses use to improve their products & services without compromising their users’ & customers’ privacy. TinyFeather uses anonymous data—this is data that could never be tied back to an individual—to track a website’s performance, not a user’s behavior.

Mitigation For Users

There are a lot of browsers & browser tools designed to block third-party cookies. Most modern browsers will provide settings to turn off third-party cookies altogether. Because of the emphasis placed on browser cookies as privacy concern, third-party advertisers have begun to use other pieces of identifiable information in order to build profiles on it’s targets.

Range of Tolerance

2020-01-10T00:00:00+00:00

Originally Published in The Pseudocode 001

When it comes to working on a team, how do we agree on anything? An office policy, a tool choice, a coding style—not everyone can have everything go their way. Of course, some things matter more to us than others, and it’s in-between the things we want and the things we need that we find our range of tolerance .

In biology, range of tolerance is the range of environmental conditions that are survivable for a species. For example, humans, needing oxygen, can survive along a range of atmospheric oxygen concentrations. That range can be plotted on a graph as a bell curve.

Source: unknown.

Too much or too little oxygen, and we enter a zone of physiological stress. Beyond that we reach an intolerable limit where we can no longer survive, (although in some cases we can adapt !).

We can use this as a metaphor for our working environment; everything needs to be within our range of tolerance, though not everything will be optimal.

In our work environment, our range of tolerance for sits along a spectrum between personal preference and personal objection . As individuals, we can place each aspect of our work environment somewhere on this spectrum. For example, it’s optimal for me to have our daily sync meeting at 10 AM, acceptable to me to have it at 9 AM, but outside of my range of tolerance to have it before 8 AM.

We use this concept on one of my teams in the form of an “any objections” vote. When introducing a change (whether in team rituals, coding style, or meeting times), we ask if there are objections, rather than if everyone agrees. By posing this question, we’re asking if the proposal sits within our individual ranges of tolerance while acknowledging that it may not be optimal for everyone.

Considering the ranges of tolerance at work means that we allow some things to be against the personal preferences of some, while not against the personal preferences of others, as long as it’s tolerable to everyone. The next time that you feel yourself resisting your team’s ideas, consider whether you personally object, or if it’s somewhere within your range of tolerance.

End of Freshman Year

2019-12-23T00:00:00+00:00

As I enter my second year as a software consultant (my sophomore year, as no one will let me call it), I’ve gained a new perspective about being pushed, having to pull, personal principles, and being the adult in the room…

Being pushed was invaluable during my first few months of being a freshman consultant. My team and company helped “push” me towards challenges/opportunities and made sure to “push” support towards me. “Thomas, can you lead the client retro, next week?” Is an example of being pushed towards something new. “Do you want to talk through how to approach that implementation idea you brought up?” Is an example of support being pushed up underneath me.

Both happen without me asking! This isn’t always a good thing, however. What happens if I wait, but no one offers the kind of support I need? What if I get pushed towards something I don’t think I’m ready for? This is what happened to me. It lead to anxiety about my performance and a negative self-review…

Tackling the first issue is where I learned about “pulling”. Pulling is when you pull from (“ask,” is a better word) your team/organization to give you what you need.

The frustrating part was that I didn’t know what I needed until I wasn’t getting it, and once I realized this, I was afraid to ask.

It takes a lot of self-worth/self-confidence to feel like you can ask for some things. Between being a minority in tech (gay & POC) and having imposter syndrome, getting to a place where I felt comfortable asking took me a long time. On top of that, I felt very indignant for having to ask for some things at all!

“Why isn’t there just a clear policy about X??”

I haven’t quite solved this issue yet, but luckily, I work in a psychologically safe organization and often, I just had to reach out.

But anyway, “pulling” is that: asking for what you need. It’s a type of professional dance one does with their company. (I think people with white-collar backgrounds might already know this). Going into my sophomore year, I’m no Fred Astaire, but I think I’ve learned to Waltz.

The other issue with “pushing” is getting pushed towards something I’m not ready for. This is where personal principles come in. They’re related to “pulling” in that they require a certain amount of self-value/worth that can be hard to gain.

A few of mine related to work:

I work up to 40 hours/week
I read/respond to email between 8am - 6pm
I only have personal communication during vacation

If they’re not defined, you can’t be principled about them. They’re not used as demands, but as reminders to ourselves about life priorities that cannot be compromised for work. They’re agreements we make with ourselves about what’s important to us.

These working principles are internalized pre-commitment devices that we can use to defend our priorities by making pacts with ourselves before we make commitments to others. They are rigid, and cannot be flexed without good-faith negotiation & compromise.

This doesn’t mean that I will never work overtime, for e.g. This just means that I need to manage the expectations that others can have of me. I have to make pre-commitment agreements with my team so that they can understand what my priorities are vis-a-vis our collective work.

The power of principles is that others can always say no. And if/when they do, you must say no to those people. This is the weight that your principles carry. Spend time negotiating with yourself about what’s most important to you. Be kind to yourself.

So, when it comes to being pushed, there are ways that I can’t be pushed. If I’ve done my job well, I’ve communicated these effectively, with kindness, and understanding that my priorities aren’t the same as others.

One more thing about principles, I try to state them in the affirmative. When I put them negatively, I immediately feel as though I need to become defensive, but ultimately, they are positive! What I commit to is valuable!

Lastly: being the adult in the room… This one just occurred to me when I was recently asked to do something difficult. I immediately thought: “well, this sounds really difficult and the person asking certainly hasn’t set me up for success!!”

After some reflection, I realized that even though, in some contexts, that thought might be fair, what I was being asked to do didn’t encroach on my principles, I didn’t intrinsically disagree with it, and ultimately, I was just afraid of failing.

In terms of setting me up for success, they had! They explained to me, in detail, how difficult the thing was they were asking me to do, and they helped create an atmosphere of psychological safety by assuring me that failure to succeed wasn’t a reflection on me.

What I was feeling was my child brain recoiling in fear and projecting refusal to participate because I didn’t believe that I could be successful. “I’m not good enough to be able to do that!” Imposter syndrome. “I’m only a freshman, after all!”

After realizing this, I went back to them and said flat out:

“I think we can do this and do it well. I have good feelings about this, no matter the outcome. Let’s do it!”

It’s this exact confidence, (that is often said that cis-White men have inherently), and commitment that costs me very little, but has the psychological benefit of forcing me to start to believe that it’s true. Note: It wasn’t a question of “can I”, but “would I?”

I thought about the people I admire—they are the people who would have also said yes. Saying yes to difficult things feels very adult to me. I think of my parents and how many times they’ve done the hard thing. To me they didn’t flinch, but of course, everyone has doubts.

I humbly thank everyone who I’ve gotten to learn from this past year!

You are all rock stars to me! ❤️

Becoming Unprecious

2019-09-29T00:00:00+00:00

In a recent “Footnotes” episode of My Dad Wrote A Porno, Tony-, Emmy-, Oscar, and Pulitzer Prize-winning guest, Lin-Manuel Miranda, sat down to talk about all things Belinda Blinked. But, he also got to talking a bit about his process as a writer:

I try really hard not to be precious about where I write because I think that I am someone who could get very superstitious very quickly… and I don’t want to be like ‘oh if I don’t have my perfect pencil…’ and, I mean I wrote the last quarter of Hamilton in my in-laws’ laundry room…”

—Lin-Manuel Miranda

That resonated with me deeply. I haven’t become superstitious, thankfully, but I have become precious.

I have a special notebook and a special pen and I work hard on getting things “perfect,” before I even get started writing. I get bogged down and become precious about the “meta-work,” the mise en place, and I never get to actual writing part: I need a cup of coffee, nice lighting, the desk at the right height, plenty of notes and research, a podcast for background noise, and then, maybe I can start writing. But, if I’m a little bit hungry, or tired, or unconfident, I won’t even get started.

Why should I try to become unprecious like Lin-Manuel? Because I want to improve, I want to write more, and I want to be less afraid. Part of why I never begin is because I’m scared that I might write something awful or that it will feel like a waste of time. But, writing is important to me and it’s a skill I’d like to continue to hone through years and years of practice.

I want to experiment with this for a while: instead of trying to force the stars to align, I am going to just write.

Entry-level Compiler

2019-09-08T00:00:00+00:00

markie

A compiler’s job is to translate one language into another. We use them in computer programming to transform the high-level languages that humans can read and write into something that computers understand.

For example, our source language might be C, which we can write, and our target language could be assembly, which our computers can run. Without a compiler, (or an assembler in the case that our target language is assembly), we would have to work with computer instruction sequences that lack the expressiveness that we’re used to in modern-day software development.

I won’t pretend to speak with any authority on compilers, but what I would like to do is share the baby steps I’ve taken into the fray by introducing the markdown to HTML compiler that I’m currently working on in Ruby.

Of the most common compiler architectures that I’ve researched, at their core, they all seemed to have a few things in common: tokenizing, parsing, and target code emission.

Tokenizing is the act of scanning the source code character-by-character, and producing a list of tokens which contain metadata about the source code.

Parsing takes that list of tokens and creates a tree structure, specifically an abstract syntax tree. This tree represents the hierarchical structure of our source code and obfuscates any details about the source language’s syntax. It does this by following a set of rules know as the *grammar *of the language.

Finally, code emission turns the abstract syntax tree into the target language by walking the tree branch-by-branch, node-by-node.

Lexical Analysis

Also know as scanning or lexing, this first step in our compiler is to turn the characters of our markdown into tokens.

What’s a Token?

You can think of a token as a character, or group of characters, with metadata or context related to what those characters represent in our source language. “Character” in our case means the literal character strings that are used to make up the code that we write.

Let’s say we write some source code:

It’s very simple, it just contains the character a. If we were to tokenize our code, we might get something like:

Tokenizer.scan("a")
=> #<type: :text, value: "a">

In this example, we’ve captured some metadata about the “code” that we wrote and began to attribute some context , :text, to it.

Another example might be this source code:

_a_

If we tokenize this source code, we end up with:

Tokenizer.scan("_a_")
=> [
  #<type: :underscore, value: "_">
  #<type: :text, value: "a">
  #<type: :underscore, value: "_">
]

This time, we have more than just text; :underscore is a significant piece of syntactical information in our source language, and therefore we make sure to write a rule so that our tokenizer can capture it.

Why Tokenize?

Having a stream of tokens, where you once had a stream of characters, allows our next step, the parsing, to do it’s job more efficiently. Not only that, during the scanning process, we can start to look out for syntax issues, (like encountering a character that our language doesn’t have a definition for). Tokenizing is only one step under the umbrella of lexical analysis. This analysis can be more robust for real programming languages, but for markdown in particular, tokenizing is a pretty straightforward process:

markdown = "[Markie]([https://github.com/Thomascountz/markie](https://github.com/Thomascountz/markie)) isn't _the_ best, but it's fun!"

tokens = Tokenizer.scan(markdown)
=> [
 #<type: :open_square_bracket, value: "[">,
 #<type: :text, value: "Markie">,
 #<type: :close_square_bracket, value: "]">,
 #<type: :open_parenthesis, value: "(">,
 #<type: :text, value: "[https://github.com/Thomascountz/markie](https://github.com/Thomascountz/markie)">,
 #<type: :close_parenthesis, value: ")">,
 #<type: :text, value: " isn't ">,
 #<type: :underscore, value: "_">,
 #<type: :text, value: "the">,
 #<type: :underscore, value: "_">,
 #<type: :text, value: " best, but it's fun!">,
 #<type: :eof, value: "">
]

Real-world Example: Ruby & Ripper

Even though Ruby isn’t technically compiled, (it’s interpreted), a lot of the same compiling steps apply. Ruby ships with a tool called Ripper that allows us to peek into, and interact with, the interpretation process of the language itself. Let’s take a look at the lexical analysis of Ruby using Ripper.tokenize() and Ripper.lex()

require 'ripper'

source_code = <<CODE
def plus_two(x)
  x + 2
end
CODE

Ripper.tokenize(source_code)
=> ["def", " ", "plus_two", "(", "x", ")", "\n", "  ", "x", " ", "+", " ", "2", "\n", "end", "\n"]

In the example above, we see that Ripper.tokenize() returns an array of strings that represent the value of each token that it scanned. It was able to distinguish keywords like def, (, and end from methods like + and variables like x.

We can take an even deeper look with Ripper.lex()

require 'ripper'

source_code = <<CODE
def plus_two(x)
  x + 2
end
CODE

Ripper.lex(source_code)
=> [[[1, 0], :on_kw, "def", EXPR_FNAME],
 [[1, 3], :on_sp, " ", EXPR_FNAME],
 [[1, 4], :on_ident, "plus_two", EXPR_ENDFN],
 [[1, 12], :on_lparen, "(", EXPR_BEG|EXPR_LABEL],
 [[1, 13], :on_ident, "x", EXPR_ARG],
 [[1, 14], :on_rparen, ")", EXPR_ENDFN],
 [[1, 15], :on_ignored_nl, "\n", EXPR_BEG],
 [[2, 0], :on_sp, "  ", EXPR_BEG],
 [[2, 2], :on_ident, "x", EXPR_END|EXPR_LABEL],
 [[2, 3], :on_sp, " ", EXPR_END|EXPR_LABEL],
 [[2, 4], :on_op, "+", EXPR_BEG],
 [[2, 5], :on_sp, " ", EXPR_BEG],
 [[2, 6], :on_int, "2", EXPR_END],
 [[2, 7], :on_nl, "\n", EXPR_BEG],
 [[3, 0], :on_kw, "end", EXPR_END],
 [[3, 3], :on_nl, "\n", EXPR_BEG]]

In this example, Ripper.lex() returned an array of arrays containing some metadata about each token in the format of:

[[lineno, column], type, token, state]

Parsing

After completing lexical analysis, we end up with a list of tokens. These tokens are then used by the parser to create an abstract syntax tree.

What’s an Abstract Syntax Tree?

Also called an AST for short, it’s a tree data structure of branches and leaf nodes that encodes the structure of our source code sans any of the syntax, (that’s what makes it abstract.)

I learned more about ASTs from Vaidehi Joshi’s BaseCS article Leveling up One’s Parsing Game with ASTs, and highly recommend your read it for an in depth look at this data structure and how it’s used in parsers.

For parsing markdown specifically, markie builds an abstract syntax tree by translating certain groups of tokens into nodes on a tree. These nodes then define what those tokens represent in the context of a markup language.

Before we unpack that, let’s expand on one of our earlier examples:

source_code = "_a_"

tokens = Tokenizer.scan(source_code)
=> [
  #<type: :underscore, value: "_">
  #<type: :text, value: "a">
  #<type: :underscore, value: "_">
]

Parser.parse(tokens)
=> {
  "type": :body,
  "children": [
    {
      "type": :paragraph,
      "children": [
        {
          "type": :emphasis,
          "value": "a"
        }
      ]
    }
  ]
}

In this example, our AST has a root node of type :body, which has a single child node, :paragraph, which also has a single child node, :emphasis, with a value of a.

Notice that our AST doesn’t contain any information about :underscore? That’s the part of this that makes it abstract. Our parser has turned the sequence of tokens with types :underscore, :text, :underscore, into a node of type :emphasis. This is because, in this flavor of markdown, :underscore, :text, :underscore, is the same an emphasis tag () in HTML surrounding that same text.

The nodes :body and :paragraph are generated to aid in the code emission step, next. These represent the <body> and  tags from our target language, HTML.

Let’s Take a Peek Ahead to See What’s Going On

HTML is tree-link by design. For example, if we have a HTML page like this:

<body>
 <h1>Header</h1>
 <p>Paragraph Text<a href="link_url">link_text</a> more text.</p>
</body>

We could represent the elements in a tree like this:

.Body
├── Header
│   └── Text("Header")
└── Paragraph
    ├── Text("Paragraph Text")
    ├── Link("link_url")
    │   └── Text("link_text")
    └── Text(" more text.")

Ultimately, that tree is what our parser aims to build out of the list of tokens from our tokenizer. If our parser is able to do that we can see how our code emission step, where we take the AST and generate HTML, should be relatively straightforward.

How Do We Generate the AST?

We saw earlier how our parser takes the tokens :underscore, :text, :underscore, and turns them into a node of type :emphasis, these translation rules that our parser follows is called the grammar.

At a high level, grammars in programming languages are similar to grammars in natural languages; they’re the rules that define the syntax of the language.

(Again, Vaidehi Joshi has us covered with her article called Grammatically Rooting Oneself with Parse Trees where she talks about how grammar applies to generating parse trees, a close cousin of abstract syntax trees.)

Let’s continue our example to enlighten our understanding. We’ll notate the :emphasis grammar like this:

Emphasis := <UNDERSCORE> <TEXT> <UNDERSCORE>

Whilst looking through our list of tokens, if we encounter this sequence, we’ll know we have an Emphasis node.

Generally, in HTML,  sit inside of other tags, such as  or , as well as many others. To keep things simple, let’s just start with , therefore, it can be said that Emphasis nodes are child nodes of Paragraph nodes, so we can add that to our mapping as well:

Paragraph := Emphasis*
Emphasis  := <UNDERSCORE> <TEXT> <UNDERSCORE>

Here, we borrow the Kleene Star, from it’s implementation in regular expressions, which here means “zero or more of the previous,” so in our case, a Paragraph node’s children are made up of zero or more Emphasis nodes.

Let’s add some more things that paragraphs can be made up of:

Paragraph := Emphasis*
           | Bold*
           | Text*
Emphasis  := <UNDERSCORE> <TEXT> <UNDERSCORE>
Bold      := <UNDERSCORE> <UNDERSCORE> <TEXT> <UNDERSCORE> <UNDERSCORE>
Text      := <TEXT>

Here, we use

to indicate a logical OR, so now we have a Paragraph node that can be made up of zero or more Emphasis child nodes, or zero or more Bold child nodes,** or** zero or more Text child nodes. And now we know what sequence of tokens the Bold and Text nodes are made up from themselves.

However, this grammar isn’t quite correct. The way it’s written now, each Paragraph can have children made up of only one type of node. We need a way to represent AND/OR. i.e. a Paragraph can have children of zero or more Emphasis nodes, and/or zero or more Bold nodes, and/or zero or more Text nodes.

We can fix this by shimming in the concept of a Sentence, for example.

Paragraph := Sentence+ <NEWLINE> <NEWLINE>
Sentence  := Emphasis*
           | Bold*
           | Text*
Emphasis  := <UNDERSCORE> <TEXT> <UNDERSCORE>
Bold      := <UNDERSCORE> <UNDERSCORE> <TEXT> <UNDERSCORE> <UNDERSCORE>
Text      := <TEXT>

Here, we borrow the Kleene Plus, from it’s implementation in regular expressions, which here means “one or more of the previous.”

Now, Paragraph can have one or more Sentence which are made up of zero or more Emphasis, or Bold, or Text.

You may begin to see a pattern here. Each line is made up of an expression, followed by a definition. We started with Emphasis, which is a terminal in our grammar, meaning that it’s definition is based purely on tokens. Next, we added Paragraph, which is a non-terminal because it is defined in reference to other terminals *and/or *non-terminals. And we have the productions *which are the rules for turning *terminals into non-terminals, which for Paragraph is Emphasis* | Bold* | Text*.

What we just went through is a bastardization of the Backus-Naur form of defining context-free grammars. I’m not going to pretend to really know what those are, yet, but feel free to dig deeper!

Real-world Example: Ruby & RubyVM

Similar to before, even though Ruby is an interpreted language, it still goes though many of the same compilation steps, including the building of an abstract syntax tree. As of Ruby2.6, we can use RubyVM::AbstractSyntaxTree to interact with the virtual machine.

source_code = <<CODE
def plus_two(x)
  x + 2
end
CODE

RubyVM::AbstractSyntaxTree.parse(source_code)
=> (SCOPE@1:0-3:3
 tbl: []
 args: nil
 body:
   (DEFN@1:0-3:3
    mid: :add_two
    body:
      (SCOPE@1:0-3:3
       tbl: [:x]
       args:
         (ARGS@1:12-1:13
          pre_num: 1
          pre_init: nil
          opt: nil
          first_post: nil
          post_num: 0
          post_init: nil
          rest: nil
          kw: nil
          kwrest: nil
          block: nil)
       body:
         (OPCALL@2:2-2:7 (LVAR@2:2-2:3 :x) :+
            (ARRAY@2:6-2:7 (LIT@2:6-2:7 2) nil)))))

There is more data in RubyVM’s AST than our simple markdown example, but thanks to Ruby being so open, we can dig in and take a look around the RubyVM!

Target Code Emission

The last step for our markdown compiler is to generate HTML from the abstract syntax tree created by the parser. In our case, we traverse the tree top-down, left-to-right, and emit HTML fragments that are joined together in the end.

Continuing from our small markdown example from above:

source_code = "_a_"

tokens = Tokenizer.scan(source_code)
=> [
  #<type: :underscore, value: "_">
  #<type: :text, value: "a">
  #<type: :underscore, value: "_">
]

Parser.parse(tokens)
=> {
  "type": :body,
  "children": [
    {
      "type": :paragraph,
      "children": [
        {
          "type": :emphasis,
          "value": "a"
        }
      ]
    }
  ]
}

First, we start at the :body node, traverse it’s child, :paragraph, then traverse it’s child: emphasis. Since we’re at a leaf node, we can begin walking back up the tree, generating fragments along the way.

First, the :emphasis node generates:

<em>a</em>

Then the :paragraph node:

<p><em>a</em></p>

And finally the :body:

<body><p><em>a</em></p></body>

This traversal strategy is call *post-order *traversal, and would normally be written like this:

def post_order(node)
  if !node.nil?
    post_order(node.left_child)
    post_order(node.right_child)
    puts node.value
  end
end

However, this algorithm is for traversing binary trees, that is, trees where each node can have at most two children. For our abstract syntax tree, we can have many children, so instead, we can recursively map.

def post_order(node)
  if !node.nil?
    node.children.map do |child|
      post_order(child)
    end
    puts node.value
  end
end

We can do this because Ruby arrays are ordered, which means that we can depend on the order in which we placed elements into the array on remaining constant when we read from the array. We leverage this when building our list of tokens and our tree’s nodes by placing things in as we scan/parse them in order.

Conclusion

This article was to be written specifically about building a markdown compiler, but really it’s an overview of the theory behind the parser I’m currently working on called markie.

I originally tried to build a markdown-to-HTML gem in Ruby using only regex and String#gsub(), but that only got me so far.

Lucky for me, it turns out that real compiler architecture is battle tested and proven to be able to handle anything from turning markdown into HTML to turning C into Assembly. Instead of reinventing the wheel, I decided to stand on the shoulders of those giants who have come before me.

Ironically, markie itself is the reinvention of an old wheel; there are plenty of tools out there to do what I’m trying to get markie to do. However, the experience of learning about compilers and interpreters has been very exciting, and ultimately, markie is a jumping off point for many projects to come.

Thanks for reading! ❤️

Peel - Day 5 - Dynamic Columns

2019-06-07T00:00:00+00:00

Modelable

Yesterday, we left off with module, Modelable, that, when included, adds two class methods, .link_to(), and .columns().

module Modelable
  def self.included(base)
    base.extend(ClassMethods)
  end
    
  module ClassMethods
    def link_to(table_name)
      @table_name = table_name.to_s
    end
      
    def columns
      SQLite3::Database.new("books_app")
		.execute("PRAGMA table_info(#{@table_name})")
		.map { |column| column[1].to_sym }
    end
  end
end

We use this module like this:

class Book
  include Modelable
  link_to(:books)
end

Book.columns
#=> [:id, :title, :author, :isbn]

Notice, that if we don’t call the .link_to() method, we would get an error:

class Book
  include Modelable
end

Book.columns
#=> SQLite3::SQLException: near ")": syntax error

Because we haven’t defined the @table_name class instance variable, so the query is being interpolated with nil instead of a string.

Note: If nil were just an empty string, we would get back an empty list, rather than an exception. This is design choice to take into consideration. Perhaps if we were test driving this, this is the kind of behavior we would account for in our design.

Now that we have our list of column names, we want to create accessor methods for each of the columns. We can do this by creating instance variables for each column and then creating getter and setter methods that retrieve and set the instances variables, respectively.

Metaprogramming

class Book
  columns.each do |column|
    define_method("#{column}") { instance_variable_get("@#{column}")}
    define_method("#{column}=") { |value| instance_variable_set("@#{column}", value)}
  end
end

Ruby gives us access to metaprogramming, which I like to think of as just programming with meta-data. In this specific example, we use the #define_method() method, which does exactly like what is sounds like.

We pass into #define_method() the name of the method, and a block containing the body of the method. The two calls to define_method(), above, create a getter and a setter for each column. The body of the first retrieves and instance variable using #instance_variable_get(), and the second sets an instance variable using #instance_variable_set().

An important thing to note here is that #instance_variable_get() will return nil if that instance variable hasn’t been defined. So our getter method will either return a value that was set using the setter’s #instance_variable_set(), or it will return nil.

Note about nil: It’s important to realize the downfalls of using nil throughout a codebase in this way. nil represents both a state, (nothing has been set), as well as data, (the value of foo is nil, or null, as represented in the database). This conflation around nil is a topic I find really interesting, and has been tackled in functional languages with sum types; like Haskell’s Maybe and Elixir’s Option.

These calls to #define_method() are just plopped into the Book class, but we can just as easily move them to the Modelable::ClassMethods module and call them from our .link_to method:

module Modelable
  def self.included(base)
    base.extend(ClassMethods)
  end
    
  module ClassMethods
    def link_to(table_name)
      @table_name = table_name.to_s
      create_accessors
    end
  
    def create_accessors
      columns.each do |column|
        define_method("#{column}") { instance_variable_get("@#{column}")}
        define_method("#{column}=") { |value| instance_variable_set("@#{column}", value)}
      end
    end
      
    def columns
      SQLite3::Database.new("books_app")
		.execute("PRAGMA table_info(#{@table_name})")
		.map { |column| column[1].to_sym }
    end
  end
end

Now, whenever a client class calls .link_to(), or module will retrieve the names of the column for the given table, and automatically generate getters/setters for each of them:

class Book
  include Modelable
  link_to(:books)
end

book = Book.new
book.title
#=> nil
book.author
#=> nil

Find

After creating some infrastructure in the client class, we’ve gotten a good pattern here with our modules. When we want to add a class instance method, we add it to the Modelable::ClassMethods module, when we want to add an instance method, we place in Modelable module.

If we think about our interface for .find(), we know we want to be able to call it on the class, e.g., Book.find(1). And, we want that class instance method to return a new instance of Book, with all of the instances variables set to the values in the respective columns.

First, we’ll tackle the retrieval of information from the database:

def find(id)
  SQLite3::Database.new("books_app")
    .execute("SELECT * FROM #{table_name} WHERE id = ? LIMIT 1", id)
end

Note: we have a similar issue as before with the string interpolation…

Here we’re selecting all the rows from the books for all rows that have an id that match the id we pass into the method. SQLite3’s Database#execute() method allows us to pass in bound arguments as an array as the second argument. Here, it also allows us to pass in just one variable as a single parameter.

By default, SQLite3 returns the results in an array of arrays:

[[2, "Practical Object-Oriented Design in Ruby", "Metz, Sandi", "0311237841549"]]

We can use this, with our columns array to create a key-value pair, but SQLite3 also gives us access to a Database#execute2() method, which is exactly the same as Database#execute(), except that is returns the column names in an array, as well as the resulting values from the query:

[
  ["id", "title", "author", "isbn"], 
  [2, "Practical Object-Oriented Design in Ruby", "Metz, Sandi", "0311237841549"]
]

This allows us to run one query, instead of two, to get the column names. Now we can zip the two arrays, and convert it into a hash:

def find(id)
  result = SQLite3::Database.new("books_app")
    .execute2("SELECT * FROM #{table_name} WHERE id = ? LIMIT 1", id)
  result[0].zip(result[1]).to_h
end

Book.find(2)
#=> {"id"=>2, "title"=>"Practical Object-Oriented Design in Ruby", "author"=>"Metz, Sandi", "isbn"=>"0311237841549"}

The last thing we’ll want to take in account is if our query returns an empty result. If that is the case, we’ll return an empty hash:

def find(id)
  result = SQLite3::Database.new("books_app")
    .execute2("SELECT * FROM #{table_name} WHERE id = ? LIMIT 1", id)
  
  if result.empty?
    {}
  else
    result[0].zip(result[1]).to_h
  end
end

Book.find(1337)
#=> {}

Like before, we can write this function in the Modelable::ClassMethods module, an it will be available to our client class, Book.

Construction

Now that we have our hash, we’re just a stone throw’s away from having a new instance of Book. If we tried to pass our hash into Book.new, we’d get the following error:

Book.new(Book.find(2))
ArgumentError: wrong number of arguments (given 1, expected 0)

That’s because we haven’t defined a constructor for Book that accepts a hash.

We could start out with something like this:

class Book
  def initialize(**args)
    @id = args.fetch(:id, nil)
    @title = args.fetch(:title, nil)
    @author = args.fetch(:author, nil)
    @isbn = args.fetch(:isbn, nil)
  end
end

Now, if we pass in our hash, we get a new instance with the correct values!

Book.new(Book.find(2))
#<Book:0x007f8b8fafe6b0 @author="Metz, Sandi", @id=2, @isbn="0311237841549", @title="Practical Object-Oriented Design in Ruby">

However, we want to abstract that our generalized .find() method can return an instance of any class.

Instead of hardcoding instance variables, we can use the same #instance_variable_set() method we used earlier. We’ll iterate over the hash, and for each key/column we’ll create an instance variable and set it to the value/row data:

def initialize(**args)
  args.each do |attribute, value|
    instance_variable_set("@#{attribute.to_s}, value)
  end
end

Notice the @ symbol, without it, our instance variables wouldn’t have the conventional @ in front of it.

Now, we can initialize our Book with a hash, and it will assign instance variables for each of the key-value pairs!

Book.new(Book.find(2))
#<Book:0x007f8b8fafe6b0 @author="Metz, Sandi", @id=2, @isbn="0311237841549", @title="Practical Object-Oriented Design in Ruby">

We can also initialize our Book manually, like before:

Book.new(title: "POODR")
#<Book:0x0078e21fafe6b0 @title="POODR">

However, we can also initialize any instance variables we pass in:

Book.new(foo: "bar")
#<Book:0x0078e21fafe6b0 @foo="bar">

To solve this, we can limit the instance variables based on the columns in the database, if we pass in a key that doesn’t match a column, we will raise an ArgumentError:

def initialize(**args)
  args.each do |attribute, value|
    if self.class.columns.include?(attribute)
      instance_variable_set("@#{attribute.to_s}", value)
    else
      raise ArgumentError, "Unknown keyword: #{attribute}"
    end
  end
end

Tying it All Together

The last step is to move this constructor into the Modelable module. We do not want to move it into the Modelable::ClassMethods module, even though we’re defining the .new class method…

In ruby, when we define the instance method, #initialize(), ruby will automagically call it after we call the class method, .new.

The final step is to add a call to .new with the result of the call .find().

Our entire module looks like this:

module Modelable
  def initialize(**args)
    args.each do |attribute, value|
      if self.class.columns.include?(attribute)
        instance_variable_set("@#{attribute.to_s}", value)
      else
        raise ArgumentError, "Unknown keyword: #{attribute}"
      end
    end
  end

  def self.included(base)
    base.extend(ClassMethods)
  end
    
  module ClassMethods
    def link_to(table_name)
      @table_name = table_name.to_s
      create_accessors
    end
  
    def create_accessors
      columns.each do |column|
        define_method("#{column}") { instance_variable_get("@#{column}")}
        define_method("#{column}=") { |value| instance_variable_set("@#{column}", value)}
      end
    end
      
    def find(id)
      result = SQLite3::Database.new("books_app")
      .execute2("SELECT * FROM #{table_name} WHERE id = ? LIMIT 1", id)
  
      if result.empty?
        {}
      else
        result_hash = result[0].zip(result[1]).to_h
        new(result_hash)
      end
    end
      
    def columns
      SQLite3::Database.new("books_app")
		.execute("PRAGMA table_info(#{@table_name})")
		.map { |column| column[1].to_sym }
    end
  end
end

Grande Finale

We can now create Active Record objects with our module:

class Book
  include Modelable
  link_to(:books)
end

book = Book.find(1)
#=> {"id"=>1, "title"=>"Practical Object-Oriented Design in Ruby", "author"=>"Metz, Sandi", "isbn"=>"0311237841549"}

book.title
#=> "Practical Object-Oriented Design in Ruby"

Next Steps

There are quite a number of directions to go in with our new Modelable module, here are few key ones that interest me:

Abstracting the Database

We’re relying directly on the SQLite3 gem, which couples our implementation directly to SQLite. Even further, our implementation hardcodes the database book_app. Wrapping this library, abstracting a Repository object, and/or using a Configuration object, are all possible next steps. Moving towards database agnosticism is an even better move towards Rails’ ActiveRecord
Tests

We didn’t write any tests for Modelable, and that’s because we couldn’t see where we were going before we got there. Now is a good time to write some tests and drive this out again, or at least writing some integration tests that describe the current behavior.
Extend to Other Active Record Methods

We’ve implemented .find(), and have established a good abstraction patter in our module. It should be fairly straight forward to implement other common methods like #save(), .all, #update(), #destroy, etc. Soon, more patterns are sure to surface and lead to further abstractions of the module. This, of course, could really benefit from TDD!
Remove String Interpolation from SQL Queries

See Day_3
Replace nil with Null Objects?

Above, we talked about the downfalls of using nil to take on many meanings throughout the code base. Ruby is notorious for this, but often times the Null Object Pattern can step in and provide our nils with more context. Perhaps this could be usefully employed to solve our nil problem.

Peel - Day 4 - Mixins or Inheritance?

2019-06-06T00:00:00+00:00

Adding Functionality to Client Classes

The first step in tackling this problem is to add functionality to a model class. If we have a class Book, we want to add a class method Book.find , and some instance methods book#title, book#author, and book#isbn, without needed to explicitly write them, like we did in the original spike.

A Note on Mix-Ins & Inheritance

When you want to create a model with ActiveRecord, you have to inherit from the Base module:

class Book < ActiveRecord::Base; end

There are endless debates about composition v. inheritance, and, from my perspective, they all mostly lean towards composition, but what about using mix-ins versus using string inheritance?

In Ruby, inheriting and including a module both add a new entity in the method look-up path:

Inheritance

class Foo; end
class Bar < Foo; end
Bar.ancestors
#=> [Bar, Foo, Object, Kernel, BasicObject]

Module Inclusion

module Foo; end
class Bar
 include Foo
end
Bar.ancestors
#=> [Bar, Foo, Object, Kernel, BasicObject]

Which is what we want. When we think of the task at hand, we want to insert an object that defines .find() for a object we want to make an Active Record object.

For our intents and purposes, either solution will do, and perhaps the same could be said for the folks working on ActiveRecord. There are a few considerations to be made when deciding however; here’s a quote from Well Grounded Rubyist, 2nd Ed.

Modules don’t have instances. It follows that entities or things are generally best modeled in classes, and characteristics or properties of entities or things are best encapsulated in modules… class names tend to be nouns, whereas module names are often adjectives…

A class can have only one superclass, but it can mix in as many modules as it wants. If you’re using inheritance, give priority to creating a sensible superclass/subclass relationship. Don’t use up a class’s one and only superclass relationship to endow the class with what might turn out to be just one of several sets of characteristics.

With that said, I want to think of making client classes “model-able,” rather than making client classes “a model.” This pulls us towards using a module and including it, rather than using inheritance, a la ActiveRecord.

A Note about Include & Extend

There are a few different ways to add behavior from a module into our classes. For our purposes, we’ll focus on include and extend.

As mentioned above, include will add an entity in the method look-up path. So for our book#title, book#author, and book#isbn, this sounds like the kind of functionality we want. The users of our ORM shouldn’t have to define these methods, they should be provided to them. If the method is missing from there class, we can define them in our module and place it next in line.

extend on the other hand, defines a singleton class and places the method definitions from our module there. This is the same as if the users of our ORM defined class methods in their class. This is a good candidate for Book.find, which, again, we’ll define and place in singleton class of the client’s class.

Often, you will want to use a module to import instance methods on a class, but at the same time to define class methods. Normally, you would have to use two different modules, one with include to import instance methods, and another one with extend to define class methods.

— Léonard Hetsch

The idiomatic way of doing this like so:

module Foo
  def self.included(base)
    base.extend(ClassMethods)
  end
    
  module ClassMethods
    def bar
      "It works!"
    end
  end
end

class MyClass
  include Foo
end

The hook .included() gets called whenever a module is included into a class, and it’s passed the class that included it. The above code adds the method .bar to class MyClass’s singleton class, #MyClass:

MyClass.bar
#=> "It works!"

Bits & Bobs

Now we have all the bits & bobs to get started. We know we want to define Book.find, book#title, book#author, and book#isbn, but we’ll also want to define a constructor Book.new, which Ruby has us do by defining the instance method book#initialize.

The workflow will go like this: we’ll define a method in our client class, e.g. in Book, and then abstract those method out into modules.

Onward to the Database

First, well want a way to read our database columns in order to know what accessor methods we need to build. In SQLite, we can use PRAGMA

The PRAGMA statement is an SQL extension specific to SQLite and used to modify the operation of the SQLite library or to query the SQLite library for internal (non-table) data.

This works with the sql3 gem like this:

SQLite3::Database.new("books_app")
  .execute("PRAGMA table_info(books)")
#=> [[0, "id", "INTEGER", 0, nil, 1], [1, "title", "VARCHAR(255)", 0, nil, 0], [2, "author", "VARCHAR(50)", 0, nil, 0], [3, "isbn", "VARCHAR(13)", 0, nil, 0]]

Each array has the following values:

["cid", "name", "type", "notnull", "dflt_value", "pk"]

Because we want the “name” of each column, we can extract them like this:

SQLite3::Database.new("books_app")
  .execute("PRAGMA table_info(books)")
  .map { |column| column[1].to_sym }
#=> [:id, :title, :author, :isbn]

Notice the call to .to_sym which turns each of the strings into Ruby symbols. This makes them a bit easy to work with later on.

Notice in our use of the sql3 gem, there are two pieces of information that tightly couple this to our implementation: the database name, books_app, and the table name books.

Note: we’re going to ignore the hardcoded database name, for now, and only address the table name.

Our solution is going to be to define an instance variable with the table name, and use that to build our query.

class Book
  @table_name = "books"
  class << self
    def columns
      SQLite3::Database.new("books_app")
        .execute("PRAGMA table_info(#{@table_name})")
        .map { |column| column[1].to_sym }
    end
  end
end

Book.columns is a class method, because the columns will be the same for all instances of class Book. We could pass the table name into .columns, but likewise, the table name will be the same for all instances. However, we want to extract this into a module, and the table name will vary depending on the class that includes it. There are at least two ways to solve this problem.

ActiveRecord solves this by implicitly linking classes and tables based on the name of the class, i.e., a class of Book would be linked to a table books, and a class User would be linked to a table, users. Similar to the Broken Record project, I believe there’s value in explicitness, so instead of inferring a table name, we’ll pass it into our module and store it there as an instance variable.

module Modelable
  def self.included(base)
    base.extend(ClassMethods)
  end
    
  module ClassMethods
    def link_to(table_name)
      @table_name = table_name.to_s
    end
    ...
  end
end

This allows us to do the following from our Book class:

class Book
  include Modelable
  link_to(:books)
  ...
end

There is still an instance variable called @table_name, but we’ve assigned it using a method. That instance variable is available both to our Book class, as well as in the Modelable module, so we can now move the #columns class method into our module.

module Modelable
  def self.included(base)
    base.extend(ClassMethods)
  end
    
  module ClassMethods
    def link_to(table_name)
      @table_name = table_name.to_s
    end
      
    def columns
      SQLite3::Database.new("books_app")
        .execute("PRAGMA table_info(#{@table_name})")
        .map { |column| column[1].to_sym }
    end
  end
end

Note: Here were see string interpolation in an SQL query again, and we discussed how that produces a security vulnerability. Unfortunately, SQLite doesn’t allow you to bind variables in PRAGMA queries. We could find the column names another way, using a query that does allow us to use bound parameters, or we could even sanitize @table_name ourselves. For now, we’ll make a note of it and place it in the icebox where stories go to die.

Peel - Day 3 - SQL Injection Vulnerability

2019-06-05T00:00:00+00:00

SQL Injection Vulnerability

class Book
...
        
  def update(title: nil, author: nil, isbn: nil)
    title = title || @title
    author = author || @author
    isbn = isbn || @isbn
    
-    SQLite3::Database.new("books.db").execute <<-SQL
-      UPDATE books  
-      SET title = '#{title}', author = '#{author}', isbn = '#{isbn}'
-      WHERE id = #{@id};
-SQL
+    SQLite3::Database.new("books.db").execute(
+      "UPDATE books  
+      SET title = ?, author = ?, isbn = ?
+      WHERE id = ?;",
+      [title, author, isbn, @id]
+    )
    Book.new(id: @id, title: title, author: author, isbn: isbn)
  end

...
end

Small update of the Book#update() method. The original version of this method used string interpolation in order to set the values for title, author, isbn, and id.

I did this because, the query is just a string and I needed that string to be “filled out” at runtime, dynamically. However, this has serious security implications. This is a prime example of how easy it is to introduce an SQL injection vulnerability.

A SQL injection attack consists of insertion or “injection” of a SQL query via the input data from the client to the application.

The Attack

Our innocent code gets a book from the database, using the Book#find() method:

book = Book.find(1)
#=> #<Book:0x007fc3c00f9160 @author="Metz, Sandi", @id=1, @isbn="0115501237044", @title="Practical Object-Oriented Design in Ruby">

When we want to update our book, we call book.update(), with the parameters we want to update. In this case, we update the title:

book.update(title: "POODR")
#=> #<Book:0x007fc34ee1160 @author="Metz, Sandi", @id=1, @isbn="0115501237044", @title="POODR">

This is the expected behavior.

Next, to visualize what an attack looks like, lets remind ourselves of the method signature and SQL query:

def update(title: nil, author: nil, isbn: nil)
    ...
	"UPDATE books  
	SET title = '#{title}', author = '#{author}', isbn = '#{isbn}'
	WHERE id = #{@id};"
	...
end

Let’s consider what would happen to our query if we called the .update() method like this:

book.update(title: "PWNED' WHERE id = 2/*")

If we called our method like that, our query would look like this:

UPDATE books 
SET title = 'PWNED' WHERE id = 2/*, author = 'Metz, Sandi', isbn = '0115501237044' 
WHERE id = 1;

Notice that SQLite will consider everything after /* to be a comment, so if we strip that away, we’re left with:

UPDATE books
SET title = 'PWNED'
WHERE id = 2;

What happened? Even though we were working with an Active Record object for a book row where id = 1, our interpolated string allowed us to effectively change the original query in order to update a different row in our database, WHERE id = 2 in our case.

We can see the effect of our SQL injection by querying for the book with and id = 2

Book.find(2)
#=> #<Book:0x007fc3c20dd120 @author="Martin, Robert C.", @id=2, @isbn="0187123641198", @title="PWNED">

This is not the intended use of our client code, but once it’s out there, we can’t protect against this kind of attack. Imagine if we were updating a user password instead of a book title!

The Fix

The fix is to use parameterized queries.

The use of prepared statements with variable binding (aka parameterized queries) is how all developers should first be taught how to write database queries. They are simple to write, and easier to understand than dynamic queries. Parameterized queries force the developer to first define all the SQL code, and then pass in each parameter to the query later. This coding style allows the database to distinguish between code and data, regardless of what user input is supplied.

OWASP

The SQLite3 gem makes using these types of statements, really easy. We replace the interpolated code with a ?, and provide a list of arguments that will be used to fill in those blanks, later:

def update(title: nil, author: nil, isbn: nil)
  ...
  SQLite3::Database.new("books.db").execute(
    "UPDATE books SET title = ?, author = ?, isbn = ? WHERE id = ?",
      [title, author, isbn, @id]
  )
  ...
end

In our case, it’s as simple as passing in a second argument to #execute(). Let’s looks at that function definition:

def execute(sql, bind_vars = [], *args, &block)

You can read more about Binding Variables in SQLite, but essentially, the abstraction in the SQLite3 gem does exactly what you imagine: replace each, ?, with the corresponding value in the bind_var array, based on order.

If we try to do the same SQL injection now, we end up with this:

book.update(title: "PWNED' WHERE id = 2/*")
#<Book:0x007fc3c22ff638 @author="Metz, Sandi", @id=1, @isbn="0115501237044", @title="PWNED' WHERE id = 2/*">

Instead of effecting any other rows in our database, the prepared statement escaped any characters which could otherwise be interpreted as SQL syntax.

Peel - Day 2 - Could it be This Easy?

2019-06-04T00:00:00+00:00

Picking Up

It seems silly that it has taken me so long to realize that the goal isn’t to create an Active Record object, it’s to create an interface for dynamically adding Active Record behavior to a class.

I’m not sure I worded that well, so let me try to break it down, for my own sake.

In our class, Book, we shouldn’t define a class-level method called .find(). Instead, this should be dynamically added to the ancestry chain, so that when we call Book.find(), and it doesn’t exist inside Book, Ruby knows to look up the hierarchy to find where that method is defined.

Likewise for the instance methods, like #update(). Instead of “hardcoding” that method inside of the Book class, we’ll want to establish an interface that will allow Ruby to delegate to where that method is defined.

With that said, the challenge of defining these methods, (.find(), #update(), etc.) continues. How do we define them in an abstract way so that the their behavior is adjusted during runtime? What interface do we make available to classes who wish to add this behavior?

Behaviors

Construct an instance of the Active Record from an SQL result set row.

We want to define a class…

class Book
end

Where we can call a method like .find(1)…

Book.find(1)

Which will in turn make a call to SQLite3 and execute a SELECT query…

SQLite3::Database.new("books_app")
	.execute("SELECT * FROM books WHERE id = 1;")

And return to us a Book instance that contains the data from the result set row, as attributes on the object.

# => #<Book:0x007fa903965e98 @author="Metz, Sandi", @id=1, @isbn="9780321721334", @title="Practical Object Oriented Design in Ruby">

The bulk of the work in this flow lies in the SQLite3 execute method. This method has a lot of knowledge:

SQLite3::Database.new() means it depends on the SQLite3 library and knows how to get a new instance of a database object.
"books_app" is the name of the database.
execute() is the name of the method on the SQLite database instance that executes an SQL query.
"SELECT * FROM ... WHERE ..." is the SQL query string to be run.
books is the name of the database table.
id is that name of the column.
1 is the id we’re querying for.

Managing how all of that knowledge gets to our persistence layer is a vital part of this problem.

Another piece of the problem is how Book ends up with a .find() class method and how our persistence layer knows how to construct a new Book instance, once it’s retrieved that data from the database.

Peel - Day 1 - Spiking w/ MySQL

2019-06-03T00:00:00+00:00

Where to Begin

First, let’s quickly revisit our problem domain:

We want to build a simple CRUD app to manage a global list of books in our library. We should be able to record a book’s title, author, and ISBN number.

Now, let’s start with some initial, sometimes arbitrary, design decisions so that I can zero-in on the meat of the problem. First, like Gregory Brown’s Broken Record project, I think I’ll begin with an SQLite database. SQLite is the default Ruby on Rails development database, and its light footprint means I should be able to get up and running quickly. We’ll utilize the SQLite3 gem, which you can read up on here.

I’m also going to assume that the database table already exists. Later in the project, I hope to take a look at managing schema migrations, but for now, the creation and alteration of database tables/columns will happen outside of the active record object. This is a good example of how the ActiveRecord implementation from Ruby on Rails goes above and beyond the basic definition of the Active Record pattern.

The database is build like this:

CREATE TABLE IF NOT EXISTS books (
  id 	 INTEGER PRIMARY KEY,
  title  VARCHAR(255),
  author VARCHAR(50),
  isbn   VARCHAR(13)
); 

The line CREATE TABLE IF NOT EXISTS does exactly what it sounds like, it will either create the table based on the schema provided, or, if a books table exits, it will do nothing.

This is important to note because a change to this SQL statement will not update the books table. For example, if we wanted to add another column pages, to record the number of pages in our books, we couldn’t simply add another line to our statement and run it again.

/* This new statement will not update an existing table */
CREATE TABLE IF NOT EXISTS books (
  id 	 INTEGER PRIMARY KEY,
  title  VARCHAR(255),
  author VARCHAR(50),
  isbn   VARCHAR(13),
  pages  INTEGER
);

The next few lines create the columns of our table. id, title, author, and isbn, each hold the the data belonging to each book. The interesting one to note is id, which is declared as INTEGER PRIMARY KEY

If you declare a column of a table to be INTEGER PRIMARY KEY, then whenever you insert a NULL into that column of the table, the NULL is automatically converted into an integer which is one greater than the largest value of that column over all other rows in the table, or 1 if the table is empty.

-SQLite FAQ

SQLite3 Gem

In order to build an Active Record object, we need a database, and in order to use a database, we need an adapter. Fortunately for us, it is not the responsibility of an Active Record object to facility the infrastructure of communication between itself and the database. For our purposes, we’ll begin with the SQLite3 gem that abstracts the database away from our object.

Later we’ll see how we can wrap this gem in an object we own, so that we can move towards becoming database agnostic.

Spiking an Active Record Object

require 'sql3'

db = SQLite3::Database.new("books.db").execute <<-SQL
  CREATE TABLE IF NOT EXISTS books (
    id 	   INTEGER PRIMARY KEY,
    title  VARCHAR(255),
    author VARCHAR(50),
    isbn   VARCHAR(13)
  ); 
SQL

class Book
  class << self
    def create(title:, author:, isbn:)
      db = SQLite3::Database.new("books.db")
      db.execute(
        "INSERT INTO books (title, author, isbn) VALUES (?, ?, ?)",
        [title, author, isbn]
      )
      id = db.last_insert_row_id
      Book.new(id: id, title: title, author: author, isbn: isbn)
    end
    
    def find(id)
      result = SQLite3::Database.new("books.db").execute(
        "SELECT * FROM books WHERE id = ? LIMIT 1", id
      )
      result.empty? ? result : Book.new(id: result[0][0], 
                                        title: result[0][1], 
                                        author: result[0][2], 
                                        isbn: result[0][3]
                                        )
    end
  end

  def initialize(id: nil, title:, author:, isbn:)
    @id = id
    @title = title
    @author = author
    @isbn = isbn
  end

  def update(title: nil, author: nil, isbn: nil)
    title = title || @title
    author = author || @author
    isbn = isbn || @isbn

    SQLite3::Database.new("books.db").execute <<-SQL
      UPDATE books  
      SET title = '#{title}', author = '#{author}', isbn = '#{isbn}'
      WHERE id = #{@id};
SQL
    Book.new(id: @id, title: title, author: author, isbn: isbn)
  end

  def destroy
    SQLite3::Database.new("books.db").execute("DELETE FROM books WHERE id = ?", @id)
  end
end

Example Usage

Book.create(title: "Practical Object Oriented Design in Ruby", author: "Metz, Sandi", isbn: "9780321721334")
# => #<Book:0x007fa904193e80 @author="Metz, Sandi", @id=1, @isbn="9780321721334", @title="Practical Object Oriented Design in Ruby">

Book.create(title: "Patterns of Enterprise Apllication Architecture", author: "Fowler, Martin", isbn: "9780321127426")
# => #<Book:0x007fa9039290b0 @author="Fowler, Martin", @id=2, @isbn="9780321127426", @title="Patterns of Enterprise Apllication Architecture">

book = Book.find(1)
# => #<Book:0x007fa903965e98 @author="Metz, Sandi", @id=1, @isbn="9780321721334", @title="Practical Object Oriented Design in Ruby">

book.title
# => "Practical Object Oriented Design in Ruby"

new_book = book.update(title: "POODR")
# => #<Book:0x007fa90291f638 @author="Metz, Sandi", @id=1, @isbn="9780321721334", @title="POODR">

Book.find(2)
# => <Book:0x007fa9028a72c8 @author="Fowler, Martin", @id=2, @isbn="9780321127426", @title="Patterns of Enterprise Apllication Architecture">

Book.find(2).destroy
# => []

Book.find(2)
# => []

This is an Active Record object. It’s not pretty, but it illustrates the core features of the pattern.

From the client’s perspective, they’re only dealing with a POOR, plain old Ruby object. Without debating the pros and cons of obfuscating database interactions, I think our Book class does a pretty good job!

Here is a list that Martin Fowler describes as typical Active Record object behaviors and a mapping to what our object does:

Construct an instance of the Active Record from a SQL result set row (e.g. Book.find())
Static finder methods to wrap commonly used SQL queries and return Active Record objects (e.g. Book.find() & Book.create())
Update the database and insert into it the data in the Active Record (e.g. Book#update())
Get and set the fields (e.g. Book#id(), Book#title(), etc)

Two other behaviors are:

Implement some pieces of business logic
Construct a new instance for later insertion into the table

Which aren’t currently implemented, however, we could follow the pattern above and implement these features pretty quickly.

What I Learned

The spiked implementation highlights a few things for me.

The ActiveRecord implementation provided by Ruby on Rails does more than just implement the Active Record pattern.

ActiveRecord isn’t just an implementation of a pattern, it’s an abstraction of the pattern. ActiveRecord allows its clients to create Active Record objects, rather than being one. It uses object-orientation to produce interfaces which allow you or I to more easily create a Book class that interacts with a database. Using the Active Record pattern to build an ORM is not the same as using Active Record pattern to build an object.
Testing is tricky.

The clearest way of testing the spiked version of Book is to write end-to-end tests that actually touch the database. This isn’t ideal, so it’s worth taking a step back and considering #1, above. The goal is to build an ORM abstraction that other classes can use, not just build an Active Record class, so there’s some pieces missing in the current iteration.
It’s all about generalization.

The spike has left me with ugly code, but “working” code. I say, “working,” because I’ve only got manual tests to verify what “working” means. However, with working code in hand, now I can begin to see abstractions that I could only guess about before.

Next Steps

How can I make SQL queries dynamic? How can I generalize a class’ attributes, based on database columns? How can I remove hardcoded dependencies on the SQLite3 gem?

Tomorrow, it’s into the fire with these questions in hand. Thankfully, Ruby allows us to answer these questions in many different ways. Hopefully, tests can drive us to the answers!

Peel - Day 0 - Thee ActiveRecord Pattern

2019-06-02T00:00:00+00:00

The Project

The goal of this project is to implement a feature-light ORM in Ruby, modeled after the Active Record pattern and inspired by Gregory Brown’s Broken Record project.

“Pattern?” Up until this point, I knew about the ActiveRecord ORM that ships with Ruby on Rails, but I wasn’t aware that this was an implementation of the active record pattern…

Init Research

In Patterns of Enterprise Application Architecture, Martin Fowler introduces the Active Record pattern as:

…an object that wraps a row in a database table or view, encapsulates the database access, and adds domain logic on that data…

This sounds like how one might describe Ruby on Rails’ ActiveRecord, and that’s no mistake. In the Ruby on Rails guides’ Active Record Basics, they describe it like so:

It is an implementation of the Active Record pattern which itself is a description of an Object Relational Mapping system… In Active Record, objects carry both persistent data and behavior which operates on that data.

Let’s say I have a database table called users, which has three columns, id, firstname, and lastname. I could represent this table, and some data in it, like this:

| id | firstname | lastname |
|----|-----------|----------|
|  0 | Simon     | Parker   |
|  1 | Mary      | Souza    |
|  2 | Tristen   | Lu-Chen  |

If I want to get the data for Mary Souza, I might write an SQL query like this:

SELECT * FROM users WHERE id = 1;

And I could expect to get some data back like this:

| id | firstname | lastname |
|----|-----------|----------|
|  1 | Mary      | Souza    |

However, from my application, I might instead want to write something like this:

user = User.select_where(id: 1)

And get back and object like this:

#{User "id": 1, "firstname": "Mary", "lastname":"Souza"}

And supply behavior to that data, such as with a method called fullname:

user.fullname
#=> "Mary Souza"

This object is an Active Record object!

It 1) wraps data from a row in the database, 2) encapsulates the retrieval of that data with the .select_where method, and 3) adds domain-specific behavior, with the .fullname method.

Martin Fowler extends this descrption with the following:

The Active Record class typically has methods that do the following:

Construct an instance of the Active Record from a SQL result set row

Construct a new instance for later insertion into the table

Static finder methods to wrap commonly used SQL queries and return Active Record objects

Update the database and insert into it the data in the Active Record

Get and set the fields

Implement some pieces of business logic

This sounds pretty good, and more often than not, if you’ve used Ruby on Rails, you are familiar with the core feature-set of the active record implementation. However, throughout the apprenticeship, I’ve learned to always take a moment ask if this is the most cost-effective solution.

Side Note: What defines “cost” will always vary widely from project to project, feature to feature. Some general things to keep in mind are how easy a solution is to implement/maintain, how much time will the implementation take, does it directly solve our problem, is the expected return greater than the cost of delivery, and does it matter? The definition of cost must be defined for every feature, so for our purposes, I’ll define cost as “does it directly solve our problem,” and with that, we must first identify the problem!

For this project, the problem is, “Thomas needs a fun and exciting project to work on, he could benefit with some more time working with databases, and he loves working with Ruby, what should he build?” But, pushing that meta-problem aside, what problem does the active record pattern solve/when would we generally aim to implement it?

Active Record is a good choice for domain logic that isn’t too complex, such as creates, reads, updates, and deletes. Derivations and validations based on a single record work well in this structure… Their primary problem is that they work well only if the Active Record objects correspond directly to the database tables… Another argument against Active Record is the fact that it couples the object design to the database design.

Martin Fowler, Patterns of Enterprise Application Architecture

What’s The Problem

Let’s invent a little MVP scenario to get shake things loose and get us thinking creatively.

We want to build a simple CRUD app to manage a global list of books in our library. We should be able to record a book’s title, author, and ISBN number.

The Active Record pattern solves our scenario. Let’s also say that we’ve weighed all of the costs and the decision to implement Active Record has gotten the green light!

Where do we go from here?

Initial Thoughts on Implementing Active Record

Without doing any more research at this point, I’ll begin by reiterating what I understand as the three major features that need to be implemented in order to deliver an Active Record object:

Abstract database connections/queries
Encapsulate data into an “Active Record” object
Provide an interface to add domain-specific behavior

In the example presented in Patterns of Enterprise Application Architecture, Martin Fowler’s houses all of this within a single object, and I think I’ll begin with that approach.

If we can implement a single Active Record object that connects to the database, wraps data, and provides an interface to that data, then maybe we can iterate until we end up with a reusable abstracted interfaces for composing new domain-specific Active Record objects.

Weinberg's Second Law

2019-05-30T00:00:00+00:00

The Second Law of Consulting: No matter how it looks at first, it’s always a people problem.

— Weinberg, Gerald M. Secrets of Consulting: a Guide to Giving and Getting Advice Successfully. 1986.

Can I Take Notes?

2019-05-30T00:00:00+00:00

When I need to onboard onto a new client, or even just a new project within the same client, I want to absorb as much as I can, as quickly as I can.

This week, my mentor at 8th Light suggested that I experiment with asking to be included in as much as possible. If I have any trepidation about my role, I show up as the note-taker.

Combining this with having conversations in the Public Square is all in the effort to bring transparency and efficiency to the way our clients communicate.

The Burden of Self-Imposed Bureaucracy

2019-05-29T00:00:00+00:00

When I have an idea to build something, I usually get caught up in my own bureaucracy. I set up a board with a backlog of features, I make a Github repo with a super detailed README, I spend hours creating beautiful wireframes, I tell all of friends about how cool my idea is, so on and so forth. But, what I don’t end up doing, is getting any of the real stuff done.

The problem is, I usually only have a few hours to try to nail something down, so instead of tackling the hard problems, I nibble away at building artifacts for myself to ogle over. I shy away from sinking my teeth into the meat-and-bones of a project because I feel like I’ll never get to finish the whole thing.

But here’s the thing, working alone means that most of that stuff isn’t even necessary, at least not in the beginning.

Instead of all of the fussy work, we can spend our focus on taking what we might consider a Minimal Viable Product (MVP), and shrinking it down even further. The question to help do this might go something like this: “what is the thinnest slice of this project that I can get done that will help me 1) validate my idea, 2) leave me with something to work off of, and 3) (most importantly) keep me motivated?”

Justin Searls at Test Double codifies this into what he calls, the One Weekend Rule in his talk, The Selfish Programmer.

…If I’m not able to release a project by Sunday night, I don’t do it… the One Weekend Rule is a liberating constraint, because it’s forced me to shrink down my dreams into something small enough that I can get done—with the side effect of also fitting inside my brain…

Subscribing to The Public Square

2019-05-29T00:00:00+00:00

While on a client, it became a normal practice to have lots of work-related conversations in private: direct chat messages, quick video calls, and email threads that didn’t include everyone, were commonplace. Given the size of the organization, it made complete sense; a constant stream of public messages would mean that we would all become helplessly buried under notification rubble in no time.

What we found, however, was that we were being overly cautious about not wanting to “bother” each other. More and more often, we found ourselves out of the loop, misinformed, and spending lots of time getting each other back up to speed.

The solution we’re experimenting with is to organically introduce having conversations in the “Public Square.” For us, that means a dedicated team channel in Slack that everyone can check-in with: developers, product owners, and stakeholders have a single source for updates and communication.

The idea is to not be afraid to share what you know. A good place to start is to update to the channel when you learn something new. Chances are, if you’re hearing it for the first time, others might need to hear it, too.

Bobby - Designing an App Just For Me

2019-05-24T00:00:00+00:00

The tl;dr

Bobby is a mobile-first Rails app that uses MTA’s real-time bus tracking API to tell users when the bus will arrive at their stop. Instead of using GPS, WiFi, or Cell Towers to pinpoint a user’s location, users manually enter in the bus stop code printed at every MTA bus stop.

The Itch

The MTA had recently released an Official MTA App, but unfortunately, like most things MTA, it was slow and unreliable. In my pursuit for something better, I resorted to reading the paper bus schedules posted on bus stop pole boxes. Printed alongside the timetables, are instructions for how to use MTA’s real-time bus tracking service via SMS. You text the bus stop code to 551-123, and it quickly texts you back real-time data!
Not only was this service the best way that I’ve found to get real-time bus data, I found the idea of these little secret codes, fascinating.

The Product

I decided that a real-time bus tracking webapp was product that I wanted to see exist in this World.

I’d call it Bobby, and it would be my opportunity to experiment with the product ownership research I had been doing. Some of that research included listening to Justin Searls’ talk, The Selfish Programmer, which is about how to approach solo work differently than the way we approach our work, at work—which is often on teams. It caused me to reflect on how I often get in my own way when I have an idea for a new project. I wrote about this a bit in my blog post, The Burden of Self Imposed Bureaucracy.

I was also inspired by reading It Doesn’t Have to be Crazy at Work by DHH and Jason Fried at Basecamp. In their book, they present their ideas about how to run a better—calmer—business. Even though I wasn’t going into business, a lot of what they wrote about complimented product ownership and healthy working habits, in general.

The Design

Even though it’s a webapp, it was important that Bobby be mobile-first. This meant that the user experience was designed to be similar to that of other mobile-first/native applications.

For Bobby, this manifested as large buttons, big fonts, and, of course, spawning the correct mobile keyboard.

The Technology

As of version 0.0.2, Bobby is a Ruby on Rails app deployed on Heroku. When a user enters a bus stop code, Bobby sends a request to the MTA Bus Time JSON API. After parsing the response, Bobby displays the results in a human-readable format.

Later versions of Bobby utilize Stimulus for font-end validations and leverage browser caching to remember users’ recent and favorite bus stop codes.

Thoughts

As an experiment, Bobby is the first Innocent App I’ve built in an effort to refine how I approach bringing my ideas to life. Even though it’s not driven by the same motives as most real products, building Bobby gave me the opportunity to own something real—from design to development and from branding to user-testing.

The approach of using bus stop codes, rather than GPS, is intentional. Bus stop codes are an interesting analog solution to identifying someone’s location. Forcing a user to have a less-than convenient experience is part of the experiment of Bobby: how can we make technology simpler and put users back into their environment? The experience of finding bus stop codes on bus stop signs is fun for me; it’s a melding of old and new. Besides, what else would I do while waiting for a bus, check Twitter?

Innocent Applications

2019-05-23T00:00:00+00:00

Innocent applications don’t lead to successful startups, they don’t aim to go viral, or even replace something that already works well. Instead, they act as hotbeds for curiosity, discovery, and conversation around engineering, design, and product. Rather than being driven by profit, innocent applications are built to be useful, sustainable, and delightful for both the users and developers.

This concept aligns closely with Ind.ie’s Ethical Design Manifesto and it’s core tenets of Human Rights, Human Effort, and Human Experience.

Safer Monkey Patching in Ruby

2019-05-21T00:00:00+00:00

At 8th Light, my team and I are rigorously working on our most important client’s most important project: a command line Battleship gem, called battle_boats. \s

I recently sat down with my mentors and demo-ed version 0.0.4, where I added a bit of color in dev mode, and my mentors-as-stakeholders liked it so much, they’ve asked for more color! But how??

“It should be easy,” I thought. I already had some implementation ideas in mind thanks to this SO answer detailing how to open up the String class to add color methods that useANSI codes.

It goes something like this:

class String
  def blue
    "\e[34m#{self}\e[0m"
  end

  def red
    "\e[31m#{self}\e[0m"
  end

  def yellow
    "\e[33m#{self}\e[0m"
  end
end

So, we open up the String class and use the ANSI color codes to output our strings in color in ANSI-supported terminals.

This allows us to do this:

puts "Hello, World!".blue
#=> Hello, World! (in blue)

So What’s The Problem?

Being that this product is a Ruby gem, everything is namespaced under the module BattleBoats, so when I tried implementing this, it looked like this:

module BattleBoats
  class String
    # implementation
  end
end

I was surprised that this didn’t work, but I soon realized that in this instance, we’re not opening String, we’re defining BattleBoats::String. Not the same thing, and not what we want.

The fix is easy, right? Just don’t wrap it in the BattleBoats module. But that left me feeling antsy. There was something about polluting the String class in a global scope that didn’t sit right with me; it felt like an irresponsible use of metaprogramming.

Defining Pure Functions

I could always just define a new module and pass strings into its methods:

module BattleBoats
  module Colorize
    def blue(string)
      "\e[34m#{string}\e[0m"
    end
  end
end

This implementation just requires that we pass our string into our method to get the encoding we need to output in color:

include BattleBoats::Colorize
puts blue("Hello, World!")
#=> Hello, World! (in blue)

But this just seemed less fun. I have a chance to use monkey patching here, and you should always take advantage of every opportunity to use monkey patching!! /s

Using instance_eval

Another option I could think of was to define the color methods only on the instances of String that I need them. For example,

module BattleBoats
  def colorify(string)
    string.instance_eval do
      def blue(string)
        "\e[34m#{self}\e[0m"
      end
    end
  end
end

This implementation gives us the best of both worlds. 1) We get to call methods directly on the strings we want to colorize, and 2) we don’t effect any other String instances. It does have the downside of requiring an extra step, but no more than our previous approach:

include BattleBoats
puts colorify("Hello, World").blue
#=> Hello, World! (in blue)

This has the added advantage of allowing us to easily add new colors to the colorify‘d strings.

None of these implementations are bad, but I really really wished for a way to open up the String class only when I included the module where the actual methods were defined…

puts "Hello, World".blue
#=> NoMethodError: undefined method 'blue' for "Hello, World!":String

include Colorize
puts "Hello, World!".blue
#=> Hello, World! (in blue)

It turns out there is a metaprogramming technique to do exactly that!

Due to Ruby’s open classes you can redefine or add functionality to existing classes. This is called a “monkey patch”. Unfortunately the scope of such changes is global. All users of the monkey-patched class see the same changes. This can cause unintended side-effects or breakage of programs.

Refinements are designed to reduce the impact of monkey patching on other users of the monkey-patched class. Refinements provide a way to extend a class locally.

— Ruby Core Docs

It’s exactly what I was looking for! Scoped monkey patching!

module BattleBoats
  module Colorize
    refine String do
      def blue(string)
        "\e[34m#{string}\e[0m"
      end
    end
  end
end

We use it like this:

puts "Hello, World".blue
#=> NoMethodError: undefined method 'blue' for "Hello, World!":String

using Colorize
puts "Hello, World!".blue
#=> Hello, World! (in blue)

Where the keyword for introducing the refinement module is using rather than include.

Now I can keep my Colorize methods away from the String’s in the rest of the system and keep my conscious clear that I won’t introduce unexpected behavior in our very important client’s software!

Jokes aside, my exposure to metaprogramming and monkey patching has been fraught with warnings, so I’ve never really investigated the power it gives developers to write expressive code. Of course there are other ways of solving this problem, but given the opportunity to make safe use of this technique was just too exciting to pass up.

Essential & Relevant: A Unit Test Balancing Act

2019-02-19T00:00:00+00:00

Originally Published on 8th Light’s Blog

I have never been a fan of “DRYing,” out unit tests (i.e., abstracting duplicated test setup). I have always preferred to keep all of my test setup inside each individual test, and I opined about how this made my test suite more readable, isolated, and consistent; despite all of the duplication. I’ve never been good at articulating why I preferred to do things this way, but I felt that it was better than the alternative: a test suite full of setup methods that forced me to scan many lines of code to try to understand how the tests work.

Then, I read xUnit Test Patterns by Gerard Meszarso. In his book, he codified some of the most profound formulas for writing unit tests. Of them all, the most well-known is probably The Four-Phase Test. Later disseminated as a distilled variant, “Arrange, Act, Assert” (and its BDD variant “Given, When, Then”), the core of it remains the same: all unit tests, in all programming languages, can take the following form:

test do
  setup
  exercise
  verify
  teardown
end

In the setup step, we instantiate our system under test, or SUT, as well as the minimum number of dependencies it requires to ensure it is in the correct state:

user = User.new(first_name: "John", last_name: "Doe")

In the exercise step, we execute whatever behavior we want to verify, often a method on our subject, or a function we’re passing our subject into:

result = user.full_name()

In the verify step, we assert that the result of the exercise step matches our expectation:

assert(result == "John Doe")

Finally, in the teardown step, we restore our system to its pre-test state. This is usually taken care of by the language or framework we’re using to write our tests.

All together, our test ends up like so:

// Example 1
...
  describe("User#full_name") do
    it("returns the full name of the user") do
      user = User.new(first_name: "John", last_name: "Doe")
      result = user.full_name()
      assert(result == "John Doe")
    end
  end
...

It’s in the “setup” step where we want to establish only the essential & relevant information needed throughout the test. Example 1 demonstrates this: we’re verifying that a user’s full name is the concatenation of their first and last, therefore, including their first and last name explicitly within the test setup is both essential & relevant.

In Meszaro’s book, he writes about the testing anti-pattern, called the Obscure Test, which addresses the imbalance between what is essential and what is relevant to our test setup.

Non-Essential & Irrelevant

As an example of non-essential & irrelevant test setup, we could tweak our original assertion like this:

// Example 2
...
  describe("User#is_logged_in?") do
    it("returns false by default") do
      user = User.new(first_name: "John", last_name: "Doe")
      result = user.is_logged_in?()
      assertFalse(result)
    end
  end
...

Here, instead of testing user.full_name() as the concatenation of first_name and last_name, we’re testing that the user returned by User.new() responds to the is_logged_in?() message with false.

Is having a first_name and last_name relevant to is_logged_in?()? Probably not, but perhaps a user is only valid with a first_name and last_name, which is what makes that setup essential to the test. In this case, the only essential & relevant setup we need explicitly in our test is a valid user who is not logged in.

Having this irrelevant setup makes for an Obscure Test of the Irrelevant Information variety.

…Irrelevant Test can also occur because we make visible all the data the test needs to execute rather than focusing on the data the test needs to be understood. When writing tests, the path of least resistance is to use whatever methods are available (on the SUT and other objects) and to fill in all the parameters with values whether or not they are relevant to the test.

-xUnit Test Patterns

We fix this by extracting a setup function/factory method:

// Example 3
...
  describe("User#is_logged_in?") do
    it("returns false by default") do
      user = valid_user()  // setup function
      result = user.is_logged_in?()
      assertFalse(result)
    end
  end
...

The relevant information is here by way of the method name, and the essential setup is on the other side of the valid_user() method.

Essential But Irrelevant

Assuming there are a lot tests with similar setup, it’s common to pull duplicated setup code into a setup function like the example above. This is also the solution to writing tests which have a verbose setup, and it helps us to ensure that we don’t include any essential but irrelevant information in our tests:

// Example 4
...
  describe("User#full_name") do
    it("returns the full name of the user") do
      user: User.new(
          first_name"" "John"
          last_name: "Doe"
          street_address: "1000 Broadway Ave"
          city: "New York"
          state: "New York"
          zip_code: "11111"
          phone_number: "555555555"
          )
      result = user.full_name()
      assert(result == "John Doe")
    end
  end
...

In this case, it may be essential to instantiate a valid user with a first_name, last_name, street_address, etc., but some of it is irrelevant to our assertion!

Like in Example 1, we’re asserting against user.full_name(), and we established that including the first_name and last_name in the setup was in fact relevant to our test. However, if we used the valid_user() setup function from Example 2 here, our setup would not contain all of the relevant information:

// Example 5
...
  describe("User#full_name") do
    it("returns the full name of the user") do
      user = valid_user() // setup function
      sult = user.full_name()
      assert(result == "John Doe")
    end
  end
...

This type of Obscure Test is called Mystery Guest.

When either the fixture setup and/or the result verification part of a test depends on information that is not visible within the test and the test reader finds it difficult to understand the behavior that is being verified without first having to find and inspect the external information, we have a Mystery Guest on our hands.

-xUnit Test Patterns

This is a case where there is essential & relevant information missing from the test. The solutions here are to 1) create an explicitly named setup function that returns the user we need, 2) create a setup function that returns a mutable user that we can update before our assertion, or 3) alter our setup function to accept parameters:

// Example 6
...
describe("User#full_name") do
  it("returns the full name of the user") do
    user = valid_user(first_name: "John", last_name: "Doe")  // new setup function
    sult = user.full_name()
    assert(result == "John Doe")
  end
end
...

This is called a Parameterized Creation Method and we use it to execute all of the essential but irrelevant steps for setting up our test. With it, we’re able to keep our test setup DRY by creating a reusable method that keeps essential information inline.

When judging when to DRY our unit tests, I’ve found it important to consider what is essential for our setup vs relevant to our test reader. There are thousands of pages more about what makes good unit tests, and I find this topic particularly nascent as the focus begins to shift from “why should we TDD” to “how do we TDD well.” Being able to articulate what is essential & relevant to a test is the key to finding the balance between people like me, who always opposed DRY unit tests, to people who prefer to keep things tidy. There are smells in both directions, but essential & relevant is the middle ground.

Coordinate Frames in 2D w/ Homogeneous Coordinates + Matplotlib

2018-11-18T00:00:00+00:00

# Setup
import numpy as np;
import matplotlib.pyplot as plt;

2D Coordinate Frames

All coordinate frames are relative. A coordinate frame gives us a frame of reference in the world, and we can describe other frames of reference relative to the one we’re referencing.

For example, if Mary is standing 3 meters from a road, and she watches a car drive past her at 20 km/h, from her frame of reference, we could describe the velocity of the car in just that way: the car is moving at 20 km/h.

And if instead, our frame of reference is from the driver, Sandra, inside the car, we could say that Mary’s velocity is 20 km/hr.

Now, let’s say that May is jogging along the road at 5 km/hr and Sandra drives past her at 25 km/hr, from Mary’s frame of reference, Sandra’s velocity is still 20 km/hr, it’s all relative.

Implicitly, Mary’s 5 km/hr jogging and Sandra’s 25 km/hr driving is relative to a stationary frame of reference, perhaps Bill, who is standing on a corner as both Mary and Sandra move past him.

In robotics, we can use the concepts of frames of reference to mathematically model the mechanics of our robot. By using a Cartesiean coordinate system and leveraging linear algebra to model a system of joints and links, we can calculate the static and kinematic models of our robot.

A robot is mechanically constructed by connecting a set of bodies, called links, to each other using various types of joints. Actuators, such as electric motors, deliver forces or torques that cause the robot’s links to move. Usually an endeffector, such as a gripper or hand for grasping and manipulating objects, is attached to a specific link. - Modern Robotics, Kevin M. Lynch and Frank C. Park, Cambridge University Press, 2017

Today, we’re going to look at the maths of coordinate frames: how to describe them, transform them, and create models from them.

World Coordinate Frame or `frame0`

We’ll begin by defining a world coordinate frame. As I mentioned, frames are always relative, so we’ll want something to center us. In our case, we’ll define a coordinate frame relative to matplotlib’s origin position.

#Boilerplate
plt.gca().set_aspect('equal')        # Set aspect ratio
plt.xlim(-0.5, 2)                    # Set x-axis range 
plt.ylim(-0.5, 2)                    # Set y-axis range

# A coordinate frame defined by its origin & unit vectors
origin = np.array([0, 0])
xhat = np.array([1, 0])
yhat = np.array([0, 1])

# Plotting 2 unit vectors
plt.arrow(*origin, *xhat, head_width=0.05, color='b')
plt.arrow(*origin, *yhat, head_width=0.05, color='b')

plt.show()

Above, we’re defining our coordinate frame by its unit vectors, x̂, called xhat, and ŷ, called yhat, as well as it’s origin point. We plot this using matplotlib.pyplot.arrow().

Well use this initial blue frame as our frame of reference going forward.

Rotation

Rotation is a linear transformation of the affine variety. Affine transformations are, generally speaking, tranfomations where:

Straight lines remain straight
The origin stays fixed

Rotation, in terms of a coordinate frame, is a circular movement about an axis such that the axes remain orthogonal to one another.

To calculate the rotation of of a vector derived from an angle, θ and in relation to a coordinate frame, we use a rotation matrix and perform matrix-vector mutiplication.

We can apply the rotation matrix on the vectors, x̂ and ŷ to described a new coordinate frame in relation to the original coordinate frame.

When we describe an operation that transforms one coordinate frame to another, we use sub- and superscript notation to explicitly state the relationship between two coordinate frames. A rotation that transforms our frame0 to a new frame, frame1 is notated like this: ⁰R₁, and can be read, “a rotation, in respect of frame0 to frame1.”

The rotation matrix from frame0 to frame1 would be:

Where x and y are the component parts of both x̂ and ŷ, meaning we’ll have to do two calculations, one for each unit vector.

An execllent resource to learn more about the properties of a 2D rotation matrix the 2D Geometry Course taught by Professor Peter Corke of QUT, an amazing free resource!

#Boilerplate
plt.gca().set_aspect('equal')        # Set aspect ratio
plt.xlim(-0.5, 2)                    # Set x-axis range 
plt.ylim(-0.5, 2)                    # Set y-axis range

# frame0 defined by its origin & unit vectors
origin = np.array([0, 0])
xhat = np.array([1, 0])
yhat = np.array([0, 1])

# Set theta0
theta0 = np.radians(30)

# Rotation matrix
rotation = np.array([
    [np.cos(theta0), -np.sin(theta0)], 
    [np.sin(theta0),  np.cos(theta0)]
])

# Solve for x̂' and ŷ', the unit vectors of frame1
xhat_prime = rotation.dot(xhat)
yhat_prime = rotation.dot(yhat)

# Plotting 2 unit vectors of frame0
plt.arrow(*origin, *xhat, head_width=0.05, color='b')
plt.arrow(*origin, *yhat, head_width=0.05, color='b')

# Plotting 2 unit vectors of frame1
plt.arrow(*origin, *xhat_prime, head_width=0.05, color='g')
plt.arrow(*origin, *yhat_prime, head_width=0.05, color='g')

print("theta0: ", "\n", np.degrees(theta0), "\n")
print("rotation matrix: ", "\n", rotation, "\n")
print("origin: ", "\n", origin, "\n")
print("xhat: ", "\n", xhat, "\n")
print("yhat: ", "\n", yhat, "\n")
print("origin_prime: ", "\n", origin, "\n")
print("xhat_prime: ", "\n", xhat_prime, "\n")
print("yhat_prime: ", "\n", yhat_prime, "\n")

plt.show()

theta0:  
 29.999999999999996 

rotation matrix:  
 [[ 0.8660254 -0.5      ]
 [ 0.5        0.8660254]] 

origin:  
 [0 0] 

xhat:  
 [1 0] 

yhat:  
 [0 1] 

origin_prime:  
 [0 0] 

xhat_prime:  
 [0.8660254 0.5      ] 

yhat_prime:  
 [-0.5        0.8660254] 

Translation

Linear translation, or displacement, can be achieved simply through vector addition:

t(v) = v + u

The translation t of the vector v is the addition of a translation vector, u.

It’s importatnt to note that this works beause in pure translations, the axes of the resulting frame, frame1, and the reference frame, frame0, remate paralell to one another.

It’s also important to note that this type of translation acts on the origin of the reference frame.

One way we could define the translation vector is by using the angle, θ, like we did to find the rotation matrix.

If we know the length of the translation vector, u, and we know the angle in reference to frame0, we can solve for the x and y components of the vector using the sine and cosine of θ.

O' = [length(u) * cos(θ), length(u) * sin(θ)]

The code below is a bit convuluted because there is implicit addition happening in the plt.arrow() function.

#Boilerplate
plt.gca().set_aspect('equal')        # Set aspect ratio
plt.xlim(-0.5, 2)                    # Set x-axis range 
plt.ylim(-0.5, 2)                    # Set y-axis range

# frame0 defined by its origin & unit vectors
origin = np.array([0, 0])
xhat = np.array([1, 0])
yhat = np.array([0, 1])

# Set theta0
theta0 = np.radians(30)

# Translation vector describes the new origin
translation = np.array([np.cos(theta0), np.sin(theta0)])
origin_prime = translation

# Plotting 2 unit vectors of frame0
plt.arrow(*origin, *xhat, head_width=0.05, color='b')
plt.arrow(*origin, *yhat, head_width=0.05, color='b')

# Plotting 2 unit vectors of frame1
# This is where the implicit addition happens
plt.arrow(*origin_prime, *xhat, head_width=0.05, color='g')
plt.arrow(*origin_prime, *yhat, head_width=0.05, color='g')

# Plotting translation vector
plt.arrow(*origin, *translation, head_width=0.05, color='y')

print("theta0: ", "\n", np.degrees(theta0), "\n")
print("translation vector: ", "\n", translation, "\n")
print("origin: ", "\n", origin, "\n")
print("xhat: ", "\n", xhat, "\n")
print("yhat: ", "\n", yhat, "\n")
print("origin_prime: ", "\n", origin_prime, "\n")
print("xhat_prime: ", "\n", xhat + translation, "\n")
print("yhat_prime: ", "\n", yhat + translation, "\n")

plt.show()

theta0:  
 29.999999999999996 

translation vector:  
 [0.8660254 0.5      ] 

origin:  
 [0 0] 

xhat:  
 [1 0] 

yhat:  
 [0 1] 

origin_prime:  
 [0.8660254 0.5      ] 

xhat_prime:  
 [1.8660254 0.5      ] 

yhat_prime:  
 [0.8660254 1.5      ] 

Homogenous Transformation Matrices

We have two transformations, a rotation and a translation. Together, they describe relative pose, or the offset and angle of rotation from one coordinate frame to another.

We can use the greek letter, ξ or a three-part tuple (x, y, θ), where x and y describe the translation vector, [x, y], and θ describes the angle of rotation.

Because pose is relative, we can use it to describe three things:

the configuration (position and orientation) of a rigid body
the reference frame in which a vector or frame is represented
the displacement of a vector or frame

Independenly, the two transformation equations we have so far are:

Rotation:

x̂' = [x̂·cos(θ) - ŷ·sin(θ)]
ŷ' = [x̂·sin(θ) + ŷ·cos(θ)]

Translation

O' = [length(u) * cos(θ), length(u) * sin(θ)]

If we want to describe both, we could do the following, apply a rotation, and then a transformation:

#Boilerplate
plt.gca().set_aspect('equal')        # Set aspect ratio
plt.xlim(-0.5, 2)                    # Set x-axis range 
plt.ylim(-0.5, 2)                    # Set y-axis range

# frame0 defined by its origin & unit vectors
origin = np.array([0, 0])
xhat = np.array([1, 0])
yhat = np.array([0, 1])

# Set theta0
theta0 = np.radians(30)

# Translation vector 
translation = np.array([np.cos(theta0), np.sin(theta0)])

# Rotation matrix
rotation = np.array([
    [np.cos(theta0), -np.sin(theta0)], 
    [np.sin(theta0),  np.cos(theta0)]
])

# Solve for O', x̂' and ŷ'
origin_prime = translation
xhat_prime = rotation.dot(xhat)
yhat_prime = rotation.dot(yhat)

# Plotting 2 unit vectors of frame0
plt.arrow(*origin, *xhat, head_width=0.05, color='b')
plt.arrow(*origin, *yhat, head_width=0.05, color='b')

# Plotting 2 unit vectors of frame1
plt.arrow(*origin_prime, *xhat_prime, head_width=0.05, color='g')
plt.arrow(*origin_prime, *yhat_prime, head_width=0.05, color='g')

# Plotting translation vector
plt.arrow(*origin, *translation, head_width=0.05, color='y')

print("theta0: ", "\n", np.degrees(theta0), "\n")
print("translation vector: ", "\n", translation, "\n")
print("origin: ", "\n", origin, "\n")
print("xhat: ", "\n", xhat, "\n")
print("yhat: ", "\n", yhat, "\n")
print("origin_prime: ", "\n", origin_prime, "\n")
print("xhat_prime: ", "\n", xhat + translation, "\n")
print("yhat_prime: ", "\n", yhat + translation, "\n")

plt.show()

theta0:  
 29.999999999999996 

translation vector:  
 [0.8660254 0.5      ] 

origin:  
 [0 0] 

xhat:  
 [1 0] 

yhat:  
 [0 1] 

origin_prime:  
 [0.8660254 0.5      ] 

xhat_prime:  
 [1.8660254 0.5      ] 

yhat_prime:  
 [0.8660254 1.5      ] 

With code, that wasn’t so bad!

Before we get to the homogenous part, image now that what we have is a robot arm with one revolute joint (i.e. a servo), at frame0, and an end effector, at frame1. This rotation and translation represents the rotational movement of the servo at frame0. As the joint rotates, the end effect’s rotation is constant with the angle, θ; it doesn’t rotate independently.

The translation vector, in yellow, represents the link between the revolute joint and the end effector. This mathematical model is what is represented by kineamatic, or joint, diagrams.

Of course, we don’t want our robot just to have one joint and one end effector, let’s try adding a second joint:

#Boilerplate
plt.gca().set_aspect('equal')        # Set aspect ratio
plt.xlim(-0.5, 3)                    # Set x-axis range 
plt.ylim(-0.5, 3)                    # Set y-axis range

# frame0 defined by its origin & unit vectors
origin_0 = np.array([0, 0])
xhat_0 = np.array([1, 0])
yhat_0 = np.array([0, 1])

# Set theta0
theta0 = np.radians(30)

# Set theta1
theta1 = np.radians(30)

# Translation vector from frame0 to frame1
translation0_1 = np.array([np.cos(theta0), np.sin(theta0)])

# Translation vector from frame1 to frame2
translation1_2 = np.array([np.cos(theta0 + theta1), np.sin(theta0 + theta1)])

# Rotation matrix from frame0 to frame1
rotation0_1 = np.array([
    [np.cos(theta0), -np.sin(theta0)], 
    [np.sin(theta0),  np.cos(theta0)]
])

# Rotation matrix from frame0 to frame1
rotation1_2 = np.array([
    [np.cos(theta0 + theta1), -np.sin(theta0 + theta1)], 
    [np.sin(theta0 + theta1),  np.cos(theta0 + theta1)]
])

# Solve for O', x̂' and ŷ' of frame1
origin_1 = translation0_1
xhat_1 = rotation0_1.dot(xhat)
yhat_1 = rotation0_1.dot(yhat)

# Solve for O', x̂' and ŷ' of frame2
origin_2 = translation0_1 + translation1_2
xhat_2 = rotation1_2.dot(xhat)
yhat_2 = rotation1_2.dot(yhat)

# Plotting 2 unit vectors of frame0
plt.arrow(*origin_0, *xhat_0, head_width=0.05, color='b')
plt.arrow(*origin_0, *yhat_0, head_width=0.05, color='b')

# Plotting 2 unit vectors of frame1
plt.arrow(*origin_1, *xhat_1, head_width=0.05, color='g')
plt.arrow(*origin_1, *yhat_1, head_width=0.05, color='g')

# Plotting 2 unit vectors of frame2
plt.arrow(*origin_2, *xhat_2, head_width=0.05, color='r')
plt.arrow(*origin_2, *yhat_2, head_width=0.05, color='r')

# Plotting translation0_1 vector
plt.arrow(*origin_0, *translation0_1, head_width=0.05, color='y')

# Plotting translation1_2 vector
plt.arrow(*origin_1, *translation1_2, head_width=0.05, color='y')

plt.show()

Pretty neat!

Notice that we have to add transformations and thetas together is several places. This is because of the relativity. We need a way to say that frame1 is in reference to frame0, and frame2 is in reference to frame1. We do that by duplicating the math to get from frame0 to frame1 so that we can get to frame2.

There is a more consise way: homogenous transformation matricies.

Let’s take a look, and then we’ll go back and see how they work.

def coordinate_frame_plot2(plt, transformation, color='b', debug=False):
  origin = transformation.dot(np.array([0, 0, 1]))[:2]
  xhat   = transformation.dot(np.array([1, 0, 1]))[:2]
  yhat   = transformation.dot(np.array([0, 1, 1]))[:2]
  
  plt.arrow(*origin, *(xhat - origin), head_width=0.05, color=color)
  plt.arrow(*origin, *(yhat - origin), head_width=0.05, color=color)
    
  if debug:
    print("transformation_matix: ", "\n", transformation, "\n")
    print("origin: ", "\n", origin, "\n")
    print("xhat: ", "\n", xhat, "\n")
    print("yhat: ", "\n", yhat, "\n")

#Boilerplate
plt.gca().set_aspect('equal')        # Set aspect ratio
plt.xlim(-0.5, 3)                    # Set x-axis range 
plt.ylim(-0.5, 3)                    # Set y-axis range

theta0 = 30
a0 = 1

theta1 = 30
a1 = 1

# Homogenous transformation from frame0 to frame0
# Identity transformation

h0 = np.array([
    [1, 0, 0],
    [0, 1, 0],
    [0, 0, 1]
])

# Homogeneous transformation from frame0 to frame1

d01 = np.array([
    [a1 * np.cos(np.radians(theta0))], 
    [a1 * np.sin(np.radians(theta0))]
])

r01 = np.array([
    [np.cos(np.radians(theta0)), -np.sin(np.radians(theta0))],
    [np.sin(np.radians(theta0)),  np.cos(np.radians(theta0))]
])

h01 = np.concatenate((np.concatenate((r01, d01), 1), np.array([[0, 0, 1]])), 0)

# Homogeneous transformation from frame1 to frame2

d12 = np.array([
    [a1 * np.cos(np.radians(theta1))], 
    [a1 * np.sin(np.radians(theta1))]
])

r12 = np.array([
    [np.cos(np.radians(theta1)), -np.sin(np.radians(theta1))],
    [np.sin(np.radians(theta1)),  np.cos(np.radians(theta1))]
])

h12 = np.concatenate((np.concatenate((r12, d12), 1), np.array([[0, 0, 1]])), 0)

# Plotting frames

coordinate_frame_plot2(plt, h0, color='b')
coordinate_frame_plot2(plt, h01, color='g')
coordinate_frame_plot2(plt, h01.dot(h12), color='r')

Ownership in Rust, Part 2

2018-07-11T00:00:00+00:00

Checkout Ownership in Rust, Part 1.

When we looked at ownership in Rust last time, we looked at how Rust uses scope to determine when a resource/data in memory should be *dropped *or freed.

We saw that for types that have a “copy trait,” (i.e. types whose data can be stored on the stack), the ownership model behaves similarly to other languages that may use a different paradigm, like garbage collection. But for types without this trait, we needed to be more conscious of the ownership rules.

Despite the design compromises that ownership may introduce, it makes up for it with flexibility, explicitness, and safety.

Ownership and Functions

fn main() {
  let string = "Hello, World!";
  println("{:p}", string); // 0x5652d704aa80
  foo(string);
}

fn foo(string: &str) {
  println("{:p}", string); // 0x5652d704aa80
}

fn main() {
  let string = String::from("Hello, World!");
  println("{:p}", string.as_ptr()); // 0x7efced01c010
  foo(string);
}

fn foo(string: String) {
  println("{:p}", string.as_ptr()); // 0x7efced01c010
}

In the first example, we’re first passing a string literal, (which stores its data on the stack), into a function, foo(). In the second example, we’re passing a String type, (which stores it data in the heap), into a different function, foo(). In both implementations of main() and foo(), we print the memory address of the variable in their respective scopes.

In the first example, we see similar behavior to when we copied a variable’s value and bound it to a new variable. This happens because string literals use the stack; the size needed to store their pointers are known at compile time, and thus, we can easily copy it’s value and pop it onto the stack.

This means that each of the functions, main() and foo(), own their own copy of the of the pointer stored in string. When foo()’s scope is over, foo() is responsible for dropping it’s own string, and when main()’s scope is over, it too is responsible for dropping the string that it owns.

In the second example, on the other hand, main() is moving ownership of string into foo(). This means that main() no longer has ownership of the string variable, i.e. the place in memory that it points to. If we tried to accessing string from inside main() after it has been moved, we would receive an error.

Instead of copying, which could be expensive, Rust instead makes foo() responsible for the data in the memory address, 0x7efced01c010, as indicated in the comments of the example. Now, only when foo() goes out of scope will Rust free the memory at that address, and thus invalidate any other variables that have a pointer to that same address. Again, we do this to avoid a double free error.

`Clone`, redux.

For the second example, if we did want to copy the value of string, so that both main() and foo() own their own copies, similar to when using the string literal on the stack, we could make a “deep copy”, by using the clone() method:

fn main() {
  let string = String::from("Hello, World!");
  println("{:p}", string.as_ptr()); // 0x7efced01c010
  foo(string.clone());
}

fn foo(string: String) {
  println("{:p}", string.as_ptr()); // 0x7f89f841c028
}

Here, as indicated by the comments, main() and foo() have ownership of their respective copies of string. Although this is a valid solution, it is not the most efficient, since Rust needs to step through its heap allocation process each time. And sometimes you actually do want both functions to interact with the same piece of data! (More on that later).

Giving Ownership

Just as ownership is taken by calling another function and passing in a variable, a function can be given ownership via a return from a different function:

fn main() {
  let string = foo();
  println("{:p}", string.as_ptr()); // 0x7fc98be1c010
}

fn foo() -> String {
  let string = String::from("Hello, World!");
  println("{:p}", string.as_ptr()); // 00x7fc98be1c010
  return string;
}

foo() now gives ownership to main() by returning string to where foo()was called. As expected, only when main()’s scope ends will Rust _free _0x7fc98be1c010.

Give & Take

If we follow this trend, it makes* *sense that we can both give ownership and then have that ownership returned to us by accepting and return the same String type in foo():

fn main() {
  let string = String::from("Hello, World!");
  println("{:p}", string.as_ptr()); // 0x7fc98be1c010
  let string = foo(string);
  println("{:p}", string.as_ptr()); // 0x7fc98be1c010
}

fn foo(string: String) -> String {
  println("{:p}", string.as_ptr()); // 00x7fc98be1c010
  return string;
}

But this seems like a lot of headache just to pass values in and out of functions. Luckily, this is a headache that the Rust maintainers have taken into account:

Taking ownership and then returning ownership with every function is a bit tedious. What if we want to let a function use a value but not take ownership? It’s quite annoying that anything we pass in also needs to be passed back if we want to use it again, in addition to any data resulting from the body of the function that we might want to return as well. Luckily for us, Rust has a feature for this concept, called references.

— Rust Book

References & Borrowing

Ownership accommodates the sharing and passing of data, but, you’ve got to follow a few rules.

Borrowing looks like this:

main() gives foo() access to string, but, (as indicated by the label), main() is still the owner of string. This means that at the end of foo()’s scope, string will not be dropped from memory; main() is still responsible for string’s space in memory.

Here’s how we would write that interaction in Rust:

fn main() {
  let string = String::from("Hello, World!");
  println("{:p}", string.as_ptr()); // 0x7fc98be1c010
  foo(&:string);
  println("{:p}", string.as_ptr()); // 0x7fc98be1c010
}

fn foo(string: &String) {
  println("{:p}", string.as_ptr()); // 00x7fc98be1c010
}

Just like our drawing, we would say that main() passes a *reference *of string into foo(), and foo() excepts a String type reference. This is indicated by & symbol. After then end of foo()’s scope, execution returns to it’s caller, main(), and string is still valid. foo() doesn’t have to return ownership, because it was never given ownership, it only borrowed.

Ampersands indicate references, which allow the passing of values without giving up ownership! Rust knows that when we’re passing a reference, the ownership, and therefore the responsibility of deallocating that space in memory, still belongs to the original owner.

Rust allows us to create any number of references:

fn main() {
  let string = String::from("Hello, World!");
  println("{:p}", string.as_ptr()); // 0x7fc98be1c010
  foo(&:string);
  println("{:p}", string.as_ptr()); // 0x7fc98be1c010
}

fn foo(string: &String) {
  println("{:p}", string.as_ptr()); // 00x7fc98be1c010
  bar(&string);
}

fn bar(string: &String) {
  println("{:p}", string.as_ptr()); // 00x7fc98be1c010
  baz(&string);
}

fn baz(string: &String) {
  println("{:p}", string.as_ptr()); // 00x7fc98be1c010
}

No matter how many times we pass around a reference to string, ownership will return to it’s original owner. (In this case, ownership returns to the place where string was originally instantiated, but remember, we could have passed ownership and then created a reference).

Mutability

The last thing to mention is mutability. Rust is often written* *in a *functional* style, but the writers are very pragmatic and understand that modern languages aren’t always so black-and-white, thus Rust accommodates mutability.

fn main() {
  let mut string = String::from("Hello, World");
  println("{:p}", string.as_ptr()); // 00x7fc98be1c010
  string.push_str("!");
  println("{:p}", string.as_ptr()); // 00x7fc98be1d000
}

Rust allows us to use the mut keyword in order to make values mutable. Notice the change in memory address which indicates that string had to be reallocated in order to fit onto the heap.

Now that we have a mutable variable, we can make a mutable reference!

fn main() {
  let mut string = String::from("Hello, World");
  println("{:p}", string.as_ptr()); // 00x7fc98be1c010
  foo(&mut string);
  println("{:p}", string.as_ptr()); // 00x7fc98be1d000
}

fn foo(string: &mut String) {
  println("{:p}", string.as_ptr()); // 00x7fc98be1c010
  string.push_str("!");
}

The syntax here is a bit specific, but we see that first we need to declare a mutable variable let mut string. Then when we pass a mutable reference, using &mut. Finally, we use &mut in the function’s signature to explicitly state that our function accepts a mutable reference.

Now, we can still ensure that only main() is the responsibly for deallocating string, while also allowing other functions to mutate string!

Those familiar with memory management might think of how this can be dangerous if left unchecked. What happens if several functions hold a mutable reference and try to update the same memory location at the same time, asynchronously; like when using threads, for example? This leads to a data race condition.

A race condition occurs when two or more threads can access shared data and they try to change it at the same time. Because the thread scheduling algorithm can swap between threads at any time, you don’t know the order in which the threads will attempt to access the shared data. Therefore, the result of the change in data is dependent on the thread scheduling algorithm, i.e. both threads are “racing” to access/change the data.

— Lehane & Amit Joki via SO

This problem can be exacerbated when working with a low-level language, such as Rust. Rust allows us access to raw pointers, which may lend itself to a lot of unsafe scenarios.

This is the kind of thing ownership is set to protect against, and it does so by enforcing this rule: “at any given time, you can have either one mutable reference or any number of immutable references.”

The benefit of having this restriction is that Rust can prevent data races at compile time. A *data race *is similar to a race condition and happens when these three behaviors occur:

Two or more pointers access the same data at the same time.

At least one of the pointers is being used to write to the data.

There’s no mechanism being used to synchronize access to the data.

Data races cause undefined behavior and can be difficult to diagnose and fix when you’re trying to track them down at runtime; Rust prevents this problem from happening because it won’t even compile code with data races!

— Rust Book

Rust’s ownership rules come to the rescue again, which is emphasized as the core safety feature that Rust provides over other systems languages. This means that Ruby programmers, like myself, still don’t have to be intimately acquainted with the inner-working of memory management!

Dangling References

One last thing, when passing references, there is another condition what can cause bugs called dangling references.

Dangling references are pointers to data that has been deallocated, for example:

fn main() {
  let string = foo();
}

fn foo() -> &string {
  let string = String::from("Hello, World!");
  println("{}", string);
  &string
}

In this example, foo() returns a reference to string. However, once foo()’s scope ends, the memory for string is deallocated, which means the reference will point to a invalid place in memory!

Rust prevents this at compile time by throwing an error.

error[E0106]: missing lifetime specifier
  - --> src/main.rs:110:13
    |
5   | fn foo() -> &String {
    |              ^ expected lifetime parameter
    |
    = help: this function's return type contains a borrowed value, but there is no value for it to be borrowed from

Rustaceans can enjoy the benefits of the ownership model without understanding the protections it provides. However, being able to comprehend the problems that ownership solves only helps to write better code without fighting against the compiler.

There’s still a bit more left to uncover about Rust ownership, but with these two posts, hopefully you’re left with enough to get started working with this elegant solution to an otherwise unwieldy problem.

References

Ownership in Rust, Part 1

2018-07-09T00:00:00+00:00

Ownership in Rust, Part 2 →

As a Rubyist, all that I know about memory allocation is that it’s handled by some process called garbage collection and that it’s Aaron Patterson’s problem, not mine.

So, when I cracked open the Rust Book and saw that one of Rust’s defining features is its alternative to garbage collection, I became a bit worried.

Was the responsibility of dealing with memory management about to be heaped onto me?

Apparently, with other system programming languages, like C, dealing with memory allocation is a big deal, and can have significant consequences when done poorly.

With all of the other new things to learn, I felt things beginning to stack up.

Stack & Heap

No, its not a hipster clothing brand, the stack and heap are ways of managing memory at runtime.

First, we have the stack. The stack is considered fast because it stores and accesses data based on order. The last thing that was placed — pushed — onto* *the stack is the first thing removed — popped — from the stack. This is referred to as *LIFO, *Last In First Out, and means that we only need to keep track of where the top of the stack is when it comes time to free up memory.

The stack can also be fast because the amount of space needed from the stack is known at compile time. This means that we can allocate a fix-size portion of memory before we store anything into it.

For example, if you know that four people are coming to your dinner party, you can decide ahead of time where everyone will sit, how much food to prepare, and practice their names before they get there. This is super efficient!

Next, we have the alternative, the heap. When you don’t know exactly how many people are coming to your dinner party ahead of time, you can use the heap. Using the heap means finding extra chairs and giving out name tags as more and more people arrive to your dinner party.

When data of unknown-size needs to be stored during runtime, the computer searches for memory on the heap, marks it, and returns a pointer, which points back to that place in memory. This is called allocating. You can then push this pointer onto a stack, however, when you want to retrieve the actual data, you need to follow the pointer back to the heap.

As I keep digging into the stack & heap rabbit hole, it seems like managing data in the heap can be difficult. For example, you need to ensure that you allow the computer to reallocate a place in memory once you’re done using it. But if one part of your code frees a place in memory that another part of your code still has a pointer to, funky things can happen.

Keeping track of what parts of code are using what data on the heap, minimizing the amount of duplicate data on the heap, and cleaning up unused data on the heap so you don’t run out of space are all problems that ownership addresses.

Rust Book

Ownership & Scope

There are three rules about ownership in Rust:

Each value in Rust has a variable that’s called its *owner*.

There can only be one owner at a time.

When the owner goes out of scope, the value will be dropped.

The simplest illustration of this ownership magic is with variable scope:

fn main() {
  let hello = "Hello, World!";
  println("{}", hello);
} // variable `hello` is now invalid

Once the current function scope is over, denoted by the }, the variable hello goes out of scope, and is dropped.

“Well, duh!” That’s what I thought when I first read this. This is the same in most other programming languages. This is what I know as the behavior of a “locally-scoped variable.”

If this is all ownership does, I’m not sure what all the hubbub is about.

However, things get more interesting when we start passing around values and switching from using a string literal, which is stored on the stack, to using a String type, which is stored on the heap.

fn main() {
  let hello = "Hello, World!"; // string literal
  let hello1 = hello; // copy the value of `hello` and bind it to `hello1`
  println("{}", hello); // this works!
  
  let hello = String::from("Hello, World!"); // `String` type
  let hello1 = hello; // move the data of `hello` into `hello1`
  println("{}", hello); // error[E0382]: use of moved value: `hello`
}

We can see here, that when using a string literal, Rust is copying the value of hello into hello1, as we might expect. But when using a String type, Rust moves the value instead. Rust tells us that we attempted to retrieve a valued that has been moved by throwing the error:error[E0382]: use of moved value: 'hello’

It seems like when using a string literal, Rust will copy the value of that one variable into another variable, but when we use a String type, it moves the value instead.

In order to find out which types implement the copy trait “…you can check the documentation… but as a general rule, any group of simple scalar values can be Copy, and nothing that requires allocation or is some form of resource is Copy”

Why Not Copy Everything?

Updated June 13, 2018

For the related discussion which lead to this update, please visit this Rust language forum thread.

The string literal, "Hello, World!", is stored somewhere in read-only memory, (neither in the stack nor heap), and a pointer to that string is stored on the stack. Because it’s a string literal, it usually shows up as a reference, meaning that we use a pointer to a string stored in permanent memory, (see Ownership in Rust, Part 2 for more on references), and it’s guaranteed to be valid for the duration of the entire program, (it has a static lifetime).

Here, the pointers stored in hello and hello1 are using the stack. When we use the = operator, Rust pushes a new copy of the pointer stored in hello onto the stack, and binds it to hello1. At the end of the scope, Rust adds a call to [drop](https://doc.rust-lang.org/1.6.0/std/ops/trait.Drop.html) which pops the values from the stack in order to free up memory. These pointers can be stored and easily copied to the stack because their size is known at compile-time.

Over on the heap, the String type with value "Hello, World!" is bound to the variable hello , using the String::from method. However, unlike the string literal, there’s more data bound to hello than just a pointer, and the size of this data can change during runtime. Here, the = operator binds the data from hello to a new variable hello1, effectively *moving *the data from one variable to another. Poor hello is now invalid, as per ownership rule #2: “There can only be one owner at a time.”

But why do this? Why doesn’t Rust always just make a copy of the data and bind it to the new variable?

If we think back to the differences between the stack and heap, we remember that the size of data stored on the heap is not known at compile time, which means we need to run through some memory allocation steps during runtime. This can be expensive. Depending on how much data we’re storing, we could quickly run out of memory if we sit around making copies of data all day.

Besides that, the default behavior of Rust helps protect us from memory issues that we might run into in other languages.

Part of storing data on the heap, is store a pointer to that data on the stack. However, unlike using a pointer to locate read-only memory, like when using a string literal, the data at the end of the pointer that leads to the heap, can change. A pointer is part of the <<DATA>> that is bound to the hello variable that stores the String type. If we bind the same pointer data to two different variables, it might look something like this:

We have two variables, hello and hello1, which share ownership of the same value. This violates rule #2: “There can only be one owner at a time,” but let’s keep going.

At the end of the scope in which hello and hello1 are defined, we have to drop the memory in the heap, which frees it up to be used again elsewhere.

First, we call drop on the data stored at the end of the pointer bound to hello1, but what happens now when we call drop on hello, next?

This is called a double free error, which I think is best summarized in this Stack Overflow answer:

A double free in C, technically speaking, leads to undefined behavior. This means that the program can behave completely arbitrarily and all bets are off about what happens. That’s certainly a bad thing to have happen! In practice, double-freeing a block of memory will corrupt the state of the memory manager, which might cause existing blocks of memory to get corrupted or for future allocations to fail in bizarre ways (for example, the same memory getting handed out on two different successive calls of malloc).

Double frees can happen in all sorts of cases. A fairly common one is when multiple different objects all have pointers to one another and start getting cleaned up by calls to free. When this happens, if you aren’t careful, you might free the same pointer multiple times when cleaning up the objects. There are lots of other cases as well, though.

— templatetypedef

This is what Rust is trying to prevent! By invalidating hello, the compiler knows to only make a call to drop, (which calls free behind the scenes), on hello1.

This is all well and good, but there are instances when we do want to copy data that’s stored in the heap. Rust provides an easy way of doing that with clone().

fn main() {
  let hello = String::from("Hello, World!"); // `String` type
  let hello1 = hello.clone(); // clone data from `hello` into `hello1`
  println("{}", hello); // => "Hello, World!"
}

Keep in mind that calls to clone() can be expensive, which is why Rust prevents this “deep copying” by default.

Apparently, there’s a lot more about Rust ownership than covered here; there are concepts called borrowing, referencing, and slicing, too!

So far, it seems like learning about ownership is more to do with navigating Rust’s memory management solution than it is to learn about the problem it solves. But, instead of taking it as a quirk of the language, the Rust Book encourages you to learn about why the language writers were eager create a safer language.

Read Ownership in Rust, Part 2 →

References

Calculate the Decision Boundary of a Single Perceptron; Visualizing Linear Separability

2018-04-13T00:00:00+00:00

tl;dr Skip to the Summary.

In the appendix of 19-line Line-by-line Python Perceptron, I touched briefly on the idea of linear separability.

A perceptron is a classifier. You give it some inputs, and it spits out one of two possible outputs, or classes. Because it only outputs a 1 or a 0, we say that it focuses on binarily classified data.

A perceptron is more specifically a linear classification algorithm, because it uses a line to determine an input’s class. If we draw that line on a plot, we call that line a decision boundary.

I spent a lot of time wanting to plot this decision boundary so that I could visually, and algebraically, understand how a perceptron works. So today, we’ll look at the maths of taking a perceptron’s inputs, weights, and bias, and turning it into a line on a plot.

The first thing to consider is that a I’m only interested in plotting a decision boundary in a 2-D space, this means that our input vector must also be 2-dimensional, and each input in the vector can be represented as a point on a graph.

For example, the following training data can be plotted like the following:

x1 | x2 | label
----------------
1  |  1 |   0
2  |  2 |   0
4  |  4 |   1
5  |  5 |   1

Where x1 is the x and x2 is the y.

Once I’ve asked a perceptron to learn how to classify these labeled inputs, I get the following results:

weights:  [ 0.2, -0.1]
bias:  -0.29

And, when I ask it to classify an input that wasn’t in the training dataset, I get an intuitive result.

We can visually guess that the new input (5, 4) belongs in the same class as the other blue inputs, (though there are exceptions). We can also imagine the line that the perceptron might be drawing, but how can we plot that line?

Maths

Remember, the summation of that our perceptron uses to determine its output is the dot product of the inputs and weights vectors, plus the bias:

w · x + b

When our inputs and weights vectors of are of 2-dimensions, the long form of our dot product summation looks like this:

w1 * x1 + w2 * x2 + b

Since we’re consider x1 to be the x and x2 to be the y, we can rewrite it:

w1x + w2y + b

That now looks an awful lot like the standard equation of a line!

Ax + By - C = 0

Note: I’ve subtracted C from both sides to set the equation equal to 0.

We can now solve for two points on our graph: the x-intercept:

x = -(b - w2y) / w1if y == 0
x = -(b - w2 * 0) / w1x = -b / w1

And the y-intercept:

y = -(b - w1x) / w2if x == 0
y = -(b - w1 * 0) / w2y = -b / w2

With those two points, we can find the slope, m:

point_1 = (0, -b / w2)
point_2 = (-b / w1, 0)m = (y2 - y1) / (x2 - x1)m = (0 - -(b / w2)) / (-(b / w1) - 0)m = -(b / w2) / (b / w1)

Now, we have the two values we need to to construct our line in slope-intercept form:

slope = -(b / w2) / (b / w1)
y-intercept = -b / w2y = (-(b / w2) / (b / w1))x + (-b / w2)

Plugging in our numbers from the dataset above, we get the following:

y = (-(-0.29 / -0.1) / (-0.29 / 0.2))x + (-(-0.29) / -0.1)y = (-2.9 / -1.45)x + -2.9y = 2x - 2.9

Plotting the Line

Summary

For a perceptron with a 2-dimensional input vector, plug in your weights and bias into the standard form equation of a line:

w1x + w2y + b = 0

Solve for the x- and y-intercepts in order to find two points on the line:

x_intercept = (0, -b / w2)
y_intercept = (-b / w1, 0)

Solve for the slope:

m = -(b / w2) / (b / w1)

Fill in the slope-intercept form equation:

y = (-(b / w2) / (b / w1))x + (-b / w2)

Resources

19-line Line-by-line Python Perceptron

2018-04-05T00:00:00+00:00

"""
MIT License

Copyright (c) 2018 Thomas Countz

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""

import numpy as np

class Perceptron(object):

    def __init__(self, no_of_inputs, threshold=100, learning_rate=0.01):
        self.threshold = threshold
        self.learning_rate = learning_rate
        self.weights = np.zeros(no_of_inputs + 1)
           
    def predict(self, inputs):
        summation = np.dot(inputs, self.weights[1:]) + self.weights[0]
        if summation > 0:
          activation = 1
        else:
          activation = 0            
        return activation

    def train(self, training_inputs, labels):
        for _ in range(self.threshold):
            for inputs, label in zip(training_inputs, labels):
                prediction = self.predict(inputs)
                self.weights[1:] += self.learning_rate * (label - prediction) * inputs
                self.weights[0] += self.learning_rate * (label - prediction)

Get the code: here, “regression” type tests here.

So far, we’ve been doing a lot of learning, with not a lot of “machine.” Today, that changes, because we’re going to implement a perceptron in Python.

What makes this Python perceptron unique, is that we’re going to be as explicit as possible with our variable names and formulas, and we’ll go through it all, line-by-line, before we get clever, import a bunch of libraries, and refactor.

Before we begin, we’ll start with a little recap and summary.

Recap & Summary

In Perceptron in Neural Networks, we looked at what a perceptron was, and we discussed the formula that describes the process it uses to binarily classify inputs. We learned that the perceptron takes in an input vector, x, multiplies it by a corresponding weight vector w, and then adds it to a bias, b. It then uses an activation function, (the step function, in this case), to determine if our resulting summation is greater than 0, in order to to classify it as 1 or 0.

In Preceptron Implementing AND - Part 1, we looked at how we could use a perceptron to mimic the behavior of an AND logic gate. We walked through, and reasoned about, how to determine the values of the weight vector, w, and the bias, b, in order for our perceptron to accurately classify the inputs from the AND truth table.

In Preceptron Implementing AND - Part 2, we looked at the Perceptron Learning Rule. We learned that by using labeled data, we could have our perceptron predict an output, determine if it was correct or not, and then adjust the weights and bias accordingly. In the end, we ended up with two formulas to describe the perceptron:

f(x) = 1 if w · x + b > 0
       0 otherwise

w <- w + (y - f(x)) * x

In Summary, we now have in our arsenal a classification algorithm.

Classification is a subcategory of supervised learning where the goal is to predict the categorical class labels of new instances, based on past observations.

Sebastian Raschka, Vahid Mirjalili, Python Machine Learning — 2nd Ed.

Supervised learning, is a subcategory of Machine Learning, where learning data is labeled, meaning that for each of the examples used to train the perceptron, the output in known in advanced.

When considering what kinds of problems a perceptron is useful for, we can determine that it’s good for tasks where we want to predict if an input belongs in one of two categories, based on it’s features and the features of inputs that are known to belong to one of those two categories.

These tasks are called binary classification tasks. Real-world examples include email spam filtering, search result indexing, medical evaluations, financial predictions, and, well, almost anything that is “binarily classifiable.”

Today, we’ll be continuing with AND:

 A   B  | AND
--- --- |-----
 1   1  |  1
 1   0  |  0
 0   1  |  0
 0   0  |  0

The Code:

I would be remiss to say, “that’s it,” because it took me quite a bit of work to write these 19 lines (minus newlines), but when considering what these 19 lines can do, it’s kind of surprising that this is all it takes. Let’s walk through it.

Line-by-line

import numpy as np

If you’re like me, not familiar with the numpy module, the only important thing to know here is that we’re using it to evaluate our dot product w · x during our summation. numpy lets us create vectors, and gives us both linear algebra functions and python list-like methods to use with it. We access its functions by calling them on np.

class Perceptron(object):

Here, we’re creating a new class Perceptron. This will, among other things, allow us to maintain state in order to use our perceptron after it has learned and assigned values to its weights.

def __init__(self, no_of_inputs, threshold=100, learning_rate=0.01):

In our constructor, we accept a few parameters that represent concepts that we looked at the end of Perceptron Implementing AND - Part 2.

The no_of_inputs is used to determine how many weights we need to learn.

The threshold, is the number of epochs we’ll allow our learning algorithm to iterate through before ending, and it’s defaulted to 100.

The learning_rate is used to determine the magnitude of change for our weights during each step through our training data, and is defaulted to 0.01.

The threshold and learning_rate variables can be played with to alter the efficiency of our perceptron learning rule, because of that, I’ve decided to make them optional parameters, so that they can be experimented with at runtime.

self.threshold = threshold
self.learning_rate = learning_rate

These two lines set the threshold and learning_rate arguments to instance variables.

self.weights = np.zeros(no_of_inputs + 1)

Here, we initialize our weight vector. np.zeros(n), will create a vector with an n-number of 0’s. Here, we use the no_of_inputs, (which again, is number of inputs in our input vector, x), plus 1.

Remember in Perceptron Implementing AND - Part 2, we move our bias into the weight vector, so that we didn’t have to deal with it independently of our other weights? This bias is the +1 to our weight vector, and is referred to as the bias weight.

def predict(self, inputs):

Now, we define our predict method. This is the method we first looked at, way back in Perceptron in Neural Networks. This method will house the f(x) = 1 if w · x + b > 0 : 0 otherwise algorithm.

The predict method takes one argument, inputs, which it expects to be an numpy array/vector of a dimension equal to the no_of_inputs parameter that the perceptron was initialized with on line 5.

summation = np.dot(inputs, self.weights[1:]) + self.weights[0]

This is where the numpy dot product function comes in, and it works exactly how you might expect. np.dot(a, b) == a · b. It’s important to remember that dot products only work if both vectors are of equal dimension. [1, 2, 3] · [1, 2, 3, 4] is invalid. Things get a bit tricky here because we’ve added an extra dimension to our self.weights vector to act as the bias.

There are two options here, either we can add a 1 to the beginning of our inputs vector, like we discussed in Perceptron Implementing AND - Part 2, or, we can take the dot product of the inputs and the self.weights vector with the the first value “removed”, and then add the first value of the self.weights vector to the dot product. Either way works, I just happened to think that this way was cleaner.

We then store the result in the variable, summation.

if summation > 0:
  activiation = 1
else:
  activation = 0
return activation

This is our step function. It kind of reads like pseudocode: if the summation from above is greater than 0, we store 1 in the variable activation, otherwise, activation = 0, then we return that value.

We don’t need the temporary variable activation, but for now, the goal is to be explicit.

def train(self, training_inputs, labels):

Next, we define the train method, which takes two arguments: training_inputs and labels.

training_inputs is expected to be a list made up of numpy vectors to be used as inputs by the predict method.

labels is expected to be a numpy array of expected output values for each of the corresponding inputs in the training_inputs list.

In essence, the input vector at training_inputs[n] has the expected output at labels[n], therefore len(training_inputs) == len(labels).

for _ in range(self.threshold):

This creates a loop wherein the following code block will be run a number of times equal to the threshold argument we passed into the Perceptron constructor. If one hasn’t been passed in, it’s defaulted to 100 epochs. Because we don’t care to use an iterator variable, convention has us set it to _.

for inputs, label in zip(training_inputs, labels):

There are three important steps happening in this line:

We zip training_inputs and labels together to create a new iterable object
We loop through the new object
While we iterate through, we store each elements in the training_inputs list into the inputs variable, and each of the elements in labels, in the variable label.

In the code block after this line, when we reference label, we get the *expected output *of the input vector stored in the inputs variable, and we do this once for every inputs/label pair.

prediction = self.predict(inputs)

Here, we pass the inputs vector into our previously defined predict method, and we store the result in the prediction variable.

self.weights[1:] += self.learning_rate * (label - prediction) * inputs

This is almost all of the learning rule implementation:

w <- w + α(y — f(x))x

We find the error, label — prediction, then we multiply it by our self.learning_rate, and by our inputs vector, we then add that result to the weight vector (with the bias weight removed), and store it back into self.weights[1:].

Remember that self.weights[0] is our bias weight, so we can’t add self.weights and inputs vectors directly, as they’re of different dimensions.

There were several options to take care of this, but I think the most explicit was is to mimic what we have done early, by only considering the vector created by “removing” the bias weight at self.weights[0].

We can’t just ignore the bias, so we deal with it next:

self.weights[0] += self.learning_rate * (label - prediction)

We update the bias in the same way as the other weights, except, we don’t multiply it by the inputs vector.

TA DA!

In just 19 lines of explicit code, we were able to implement a perceptron in Python!

Usage

Let’s put it to work and finally wrap up implementing AND

import numpy as np
from perceptron import Perceptron

First, we import numpy so that we can create our vectors, then we import our new perceptron.

training_inputs = []
training_inputs.append(np.array([1, 1]))
training_inputs.append(np.array([1, 0]))
training_inputs.append(np.array([0, 1]))
training_inputs.append(np.array([0, 0]))

Next, we generate our training data. These inputs are the A and B columns from the AND truth table stored in an array of numpy arrays, called training_inputs.

labels = np.array([1, 0, 0, 0])

Here, we store the expected outputs, or labels in the label variable, making sure that each label index lines up with the index of the input it’s meant to represent.

perceptron = Perceptron(2)

We instantiate a new perceptron, only passing in the argument 2 therefore allowing for the default threshold=100 and learning_rate=0.01. Note that such a large threshold and such a small learning rate probably isn’t needed, so feel free to play around to find what’s most efficient! What happens if learning_rate=10? What if threshold=2?

perceptron.train(training_inputs, labels)

Now we train the perceptron by calling perceptron.train and passing in our training_inputs and labels.

This should finish rather quickly. Even though there are 100 epochs, our training data is so small and numpy is very efficient!

inputs = np.array([1, 1])
perceptron.predict(inputs)
#=> 1

inputs = np.array([0, 1])
perceptron.predict(inputs)
#=> 0

That’s it! Now, we can start to use the perceptron as a logic AND!

It may seem a bit bizarre that we’ve trained our perceptron with four inputs and we only really need it to classify those four inputs. Is that all perceptrons are good for? No! Remember, perceptrons can be used to classify almost any number of binarily classifiable things, (though there are some major caveats, see below).

What would happen if you removed one of the training inputs? Removed two of them? Are you able to remove the [1, 1] training input? What other logic operators can you train the perceptron on? What happens if we add more inputs?

Test! Experiment! Play!

Conclusion

This concludes our AND implementation, so now is a good time to sum up everything we’ve learned.

Perceptrons were first published in 1957 by Frank Rosenblatt at the Cornell Aeronautical Laboratory. He proposed a rule that could automatically determine the weights **for each of the artificial neuron’s **input features, (one input vector example), by using supervised learning to determine a decision boundary, (see below), between two binary classes.

The perceptron classifies inputs by finding the dot product of an input feature vector and weight vector and passing that number into a step function, which will return 1 for numbers greater than 0, or 0 otherwise.

f(x) = 1 if w · x + b > 0
       0 otherwise

In order to the determine the weights, the Perceptron Learning Rule:

Predicts an output based on the current weights and inputs
Compares it to the expected output, or label
Update its weights, if the prediction **!= the **label
Iterate until the epoch threshold has been reached

To update the weights during each iteration, it:

Finds the error by subtracting the prediction from the label
Multiplies the error and the learning rate
Multiplies the result to the inputs
Adds the resulting vector to the **weight **vector

w <- w + α(y - f(x))x

Appendix and Further Exploration

There are a few concepts we haven’t touch on yet. Notably, the limitations of the perceptron.

The Perceptron Convergence Theorem is, from what I understand, a lot of math that proves that a perceptron, given enough time, will always be able to find a decision boundary between two linearly separable classes.

It is important to note that the convergence of the perceptron is only guaranteed if the two classes are linearly separable and the learning rate is sufficiently small. If the two classes can’t be separated by a linear decision boundary, we can set a maximum number of passes over the training dataset (epochs) and/or a threshold for the number of tolerated misclassifications — the perceptron would never stop updating the weights otherwise.

Sebastian Raschka, Vahid Mirjalili, Python Machine Learning — 2nd Ed.

Linearly separable means that there exists a linear hyperplane, (line), that can separate input vectors into their correct classes; one class’ vectors falling on one side of the hyperplane, and the other class’, on the other.

In terms of our binary operator AND, linear separability means that:

If…

We plot each of our *A* and *B* inputs, from our truth table, as points,*(A, B)*, on a 2-D plane…

Then..

We could draw a single line on that plane in such a way so that all of the *(A, B)*points on one side of the line are the *A* and *B* inputs that give us *1*, and all the points on the other side, give us *0*.

Here is ourAND and its truth table:

( A , B ) | AND
---  ---  |-----
( 0 , 0 ) |  0
( 0 , 1 ) |  0
( 1 , 0 ) |  0
( 1 , 1 ) |  1

We see that all of the pairs of inputs that return 0 are red and on one side of the line, and the input that gives us 1, is on the other side of the line.

This is a graphical representation of what our perceptron does! Our perceptron defines a line to draw in the sand, so to speak, that classifies our inputs binarily, depending on which side of the line they fall on! This line is call the decision boundary*, and when employing a single perceptron, we only get one.*

In other words, if there is no single line that can separate our training data into two classes, our perceptron will never find weights that can satisfy all of our data. It doesn’t take long to hit this limitation. Take a look the XOR Perceptron Problem.

Perceptrons have gotten us pretty far, but we’re not done with them yet. Now that we’ve gotten our hands on some code, we can begin digging deeper into using Python as a tool to further explore machine learning and neural networks.

Next, we’ll refactor our perceptron code, take a look at how we can use our model to classify more complex data, and look at how to use tools like matplotlib to visualize decision boundaries.

Resources

Perceptron Convergence Theorem
Python Machine Learning — 2nd Ed. by Sebastian Raschka & Vahid Mirjalili
Single-Layer Neural Networks and Gradient Descent
10.2: Neural Networks: Perceptron Part 1 — The Nature of Code
Appendix F — Introduction to NumPy from Introduction to Artificial Neural Networks and Deep Learning A Practical Guide with Applications in Python by Sebastian Raschka

Perceptron Implementing AND, Part 2

2018-03-28T00:00:00+00:00

In Perceptron Implementing AND, Part 1, we looked at implementing the behavior of AND with a perceptron. We set up our perceptron with its inputs and expected outputs, and we sort of reasoned about what our weights and bias should be, in order to get the output we wanted.

Even for just a single artificial neuron, with only two binary inputs, a single binary output, and a complete set of training data, that was still a considerable amount of work, which resulted in mimicking behavior of a simple logic statement.

If we consider that neural networks are made up of dozens, thousands, or even 160 billion neurons, we can see that manually tuning our weights and biases is simply out of the question.

When we talk about a neural network “learning,” or “training,” this tuning process is exactly what we’re talking about. Our networks can learn to assign its own weights and biases, based on a set of training data, where the inputs and expected outputs are known.

Our neuron does this by iterating through training data, and updating the weights and bias after each set of inputs. Once our neuron determines that it has found parameters that successfully classify all of the training data, the learning process is complete, and the weights and biases can then be used to classify data that wasn’t in the training set!

Seeing that our AND truth table is a type of training data, and we have our model of a network, (even if it’s only one neuron), we can implement this learning behavior, and that’s what we’ll take a look at today.

A Bias is the Weight of an Always-Active Input

Before we take a look at the learning process, we’re going to tweak our perceptron model, just a little bit:

It turns out that the algorithm that determines the weights for each input, is the same algorithm that determines the bias. Because of this, it’s more efficient to lump the concept of a bias in with the other inputs by adding an additional input with an activation of 1, and allow our algorithm to treat the weight of that new input, as the bias.

If you remember our perceptron formula, (pictured to the left), you’ll recall that we add the dot product of vectors w and x, to the bias, b, to get what is called the weighted sum. Expanded, it looks like this:

(w1 * x1) + (w2 * x2) + ... + (wn * xn) + b

Where wn is the weight of input xn.

We add the product of all n-numbered w’s and their n-numbered x’s together, and then we add that result to the bias, b.

In this equation, we can also represent b, by adding another input whose activation is always 1, and multiplying it by a weight equal to b.

(w1 * x1) + (w2 * x2) + ... + (wn * xn) + (b * 1)

You’ll even sometimes see b as (w0 * x0), where x0 = 1.

Our inputs x ... xn will never be changed by our learning algorithm, and therefore, neither will our new input,1. Our algorithm now can focus only on adjusting the weights, which now also include our bias.

We can think of this new perceptron algorithm like this:

f(x) = 1 if w · x > 0
       0 otherwise

As long as we explicitly add an input with an activation of 1, and a weight equal to b.

How Perceptrons Learn

These are the abstract steps that our perceptron will take in order to learn, i.e., converge on a set of values for its weights that will accurately classify all of our training input:

Initialize the weights, sometimes randomly, or more simply, establish them all to 0.
For each set of inputs in the set of training examples our perceptron will:
Predict an output
Compare it to the expected output
Update its weights, if the expected output != the actual output.
Move to next set of inputs.

There are a few new concepts we’ll want to define.

First, we’ll need to know how well our perceptron is doing.

Given that our neuron produces an output, and we know what we expect the output to be, we can define how well our neuron is doing like this:

expected_output - actual_output

Which we’ll notate like this:

e = y - f(x)

Where e, for error, is equal to our expected output y, minus the output of our perceptron function, f, given inputs x.

Given that y and f(x) are binary, y — f(x), will produce only one of three values, given any x:

y   f(x) | y - f(x)
---  ---  | --------
 1    1   |    0
 0    0   |    0
 1    0   |   -1
 0    1   |    1

Next, we’ll see how we can use this information to adjust our weights.

The goal: get y — f(x) closer to 0.

The intuition behind this learning algorithm, developed by Frank Rosenblatt, who you’ll remember as the creator as the perceptron, follows a simple rule:

If the neuron activates, when we want it not, suppress it.

If the neuron does not activate, when we want it so, excite it.

If the neuron performs as asked, do nothing.

—Frank Rosenblatt

We know how to check how well our neuron is doing, so now, we’ll need to establish how we can excite it, suppress it, or do nothing to it.

If the perceptron outputs 1, (f(x) = 1), when we wanted 0, (y = 0), we’ll want to adjust it by making w · x smaller, because in order to get f(x) = 0, w · x must be <= 0.

Likewise, if the perceptron outputs 0, (f(x) = 0), when we wanted 1, (y = 1), we’ll want to adjust it by making w · x larger, because in order to get f(x) = 1, w · x must be > 0.

And finally, if the perceptron outputs what we expected, f(x) == y, we’ll want to adjust nothing, because our perceptron has successfully classified our input!

Again, for each input, we go through these adjustments, and then blindly move to the next set of inputs.

Lastly, let’s look at how we’ll make these adjustments our weights.

We can’t change our x, but we can change our w, and here is the formula for doing so:

w <- w + x if y - f(x) ==  1
w <- w - x if y - f(x) == -1
w <- w     if y - f(x) ==  0

So, to make our w larger or smaller, we simply add or subtract x to it.

Because our y — f(x) produces 1, -1, and 0, we can simplify:

w <- w + (y - f(x)) * x

That’s it!

Let’s see how this works with the first iteration by training AND:

Training data:

 x1   x2  |  y
---- ---- |-----
 1    1   |  1
 1    0   |  0
 0    1   |  0
 0    0   |  0

Initialize the weights, sometimes randomly, or more simply, establish them all to 0.

w = [0, 0, 0]  #=> [bias, weight, weight]

For each set of inputs in the set of training examples our perceptron will:

x = [1, 1, 1]  #=> [bias_input, x1, x2]
y = 1

Predict an output

f(x) = 1 if w `· `x > 0
       0 otherwise

w `· x == (0 * 1) + (0 * 1) + (0 * 1) == 0
      `∴  f(x) = 0

Compare it to the expected output

e = y - f(x)  #=> 1

Update its weights, if the expected output != the actual output

w <- w + 1 * x  #=> [1, 1, 1]

Move to next set of inputs.

Now, I’ll speed through the rest of the iterations, so that you can see the algorithm in action.

w    = [1, 1, 1]
x    = [1, 1, 0]
y    = 0
f(x) = 1
e    = -1
w    <- w + -1x = [0, 0, 1]
---------------------------------
w    = [0, 0, 1]
x    = [1, 0, 1]
y    = 0
f(x) = 1
e    = -1
w    <- w + -1x = [-1, 0, 0]
---------------------------------
w    = [-1, 0, 0]
x    = [1, 0, 0]
y    = 0
f(x) = 0
e    = 0
w    <- w + 0x = [-1, 0, 0]
---------------------------------
w    = [-1, 0, 0]
x    = [1, 1, 1]
y    = 1
f(x) = 0
e    = 1
w    <- w + 1x = [0, 1, 1]
---------------------------------
w    = [0, 1, 1]
x    = [1, 1, 0]
y    = 0
f(x) = 1
e    = -1
w    <- w + -1x = [-1, 0, 1]
---------------------------------
w    = [-1, 0, 1]
x    = [1, 0, 1]
y    = 0
f(x) = 0
e    = 0
w    <- w + 0x = [-1, 0, 1]
---------------------------------
w    = [-1, 0, 1]
x    = [1, 0, 0]
y    = 0
f(x) = 0
e    = 0
w    <- w + 0x = [-1, 0, 1]
---------------------------------
w    = [-1, 0, 1]
x    = [1, 1, 1]
y    = 1
f(x) = 0
e    = 1
w    <- w + 1x = [0, 1, 2]
---------------------------------
w    = [0, 1, 2]
x    = [1, 1, 0]
y    = 0
f(x) = 1
e    = -1
w    <- w + -1x = [-1, 0, 2]
---------------------------------
w    = [-1, 0, 2]
x    = [1, 0, 1]
y    = 0
f(x) = 1
e    = -1
w    <- w + -1x = [-2, 0, 1]
---------------------------------
w    = [-2, 0, 1]
x    = [1, 0, 0]
y    = 0
f(x) = 0
e    = 0
w    <- w + 0x = [-2, 0, 1]
---------------------------------
w    = [-2, 0, 1]
x    = [1, 1, 1]
y    = 1
f(x) = 0
e    = 1
w    <- w + 1x = [-1, 1, 2]
---------------------------------
w    = [-1, 1, 2]
x    = [1, 1, 0]
y    = 0
f(x) = 0
e    = 0
w    <- w + 0x = [-1, 1, 2]
---------------------------------
w    = [-1, 1, 2]
x    = [1, 0, 1]
y    = 0
f(x) = 1
e    = -1
w    <- w + -1x = [-2, 1, 1]
---------------------------------
w    = [-2, 1, 1]
x    = [1, 0, 0]
y    = 0
f(x) = 0
e    = 0
w    <- w + 0x = [-2, 1, 1]
---------------------------------
w    = [-2, 1, 1]
x    = [1, 1, 1]
y    = 1
f(x) = 0
e    = 1
w    <- w + 1x = [-1, 2, 2]
---------------------------------
w    = [-1, 2, 2]
x    = [1, 1, 0]
y    = 0
f(x) = 1
e    = -1
w    <- w + -1x = [-2, 1, 2]
---------------------------------
w    = [-2, 1, 2]
x    = [1, 0, 1]
y    = 0
f(x) = 0
e    = 0                          # No Error!
w    <- w + 0x = [-2, 1, 2]
---------------------------------
w    = [-2, 1, 2]
x    = [1, 0, 0]
y    = 0
f(x) = 0
e    = 0                          # No Error!
w    <- w + 0x = [-2, 1, 2]
---------------------------------
w    = [-2, 1, 2]
x    = [1, 1, 1]
y    = 1
f(x) = 1
e    = 0                          # No Error!
w    <- w + 0x = [-2, 1, 2]
---------------------------------
w    = [-2, 1, 2]
x    = [1, 1, 0]
y    = 0
f(x) = 0
e    = 0                          # No Error!
w    <- w + 0x = [-2, 1, 2]
---------------------------------
w    = [-2, 1, 2]
x    = [1, 0, 1]
y    = 0
f(x) = 0
e    = 0                          # No Error!
w    <- w + 0x = [-2, 1, 2]

Wrap Up

f(x) = 1 if w · x + b > 0
       0 otherwise

w <- w + (y - f(x)) * x

Those are our two formulas to describe perceptrons and our learning rule.

We ended up finding different weights (*and pseudo-bias) *than last time, when we simply reasoned about a solution. This was solved algorithmically, and is something a computer could do!

There Are a Few Concepts Missing

Three concepts in particular that are missing from our training algorithm are epoch,* threshold*, and learning rate.

Epoch is the number of times we’ve iterated through the entire training set. So for the example above, during epoch = 5, we were able to establish weights to classify all of our inputs, but we continued iterating, to be sure that our weights were tried on all of our inputs.

Threshold is different than what we now call our bias. Threshold is the maximum number of epoch we will allow to pass while training. There is not built in stopping point of our algorithm. It will continue adding 0 to our weights, on and on, forever. Adding a threshold is one way of stopping our training loop.

Learning rate, symbolize by α, is the magnitude at which we increase or decrease our weights during each iteration of training:

w <- w + α(y - f(x))x

This allows us to smooth out the change in our weights, which can prevent overshooting and rebounding around weight settings that accurately classify our training set.

Resources

Perceptron Implementing AND, Part 1

2018-03-26T00:00:00+00:00

In Perceptron in Neural Networks, we got my feet wet learning about perceptrons. Inspired by Michael Nielsen’s Neural Networks and Deep Learning book, today, the goal is to expand on that knowledge by using the perceptron formula to mimic the behavior of a logical AND.

In this post, we’ll reason about the settings of our network that, in Perceptrons Implementing AND, Part 2, we’ll have the computer do itself.

As a programmer, I am familiar with logic operators like AND, OR, XOR. Well, as I’m learning about artificial neurons, it turns out that the math behind perceptrons, see more here, can be used to recreate the functionality of these binary operators!

As a refresher, let’s look at the logic table for AND:

 A   B  | AND
--- --- |-----
 0   0  |  0
 0   1  |  0
 1   0  |  0
 1   1  |  1

Let’s break it down.

To produce a logical AND, we want our function to output 1, only when both inputs, A, and B, are also 1. For every other case, our AND should output 0.

Let’s take a look at this using our perceptron model from last time, with a few updates:

The equation we ended up with looks like this:

And when we insert our inputs and outputs into our model, it looks like this:

Side note: This model of a perceptron is slightly different than the last one. Here, I’ve tried to model the weights and bias more clearly.

All we’ve done so far, is plug our logic table into our perceptron model. All of our perceptrons are returning 0, except for when both of our inputs are “activated,” i.e. when they are 1.

What is missing from our model, is the actual implementation detail; the weights and biases that would actually give us our desired output. Moreover, we have four different models to represent each state of our perceptron, when what we really want, is one!

So the question becomes how do we represent the behavior of a logical `AND`, i.e., what weights and biases should we input into our model to produce the desired output?

The first thing to note is that our weights should be the same for both inputs, A and B.

If we look back at our logic chart, we can begin to notice that the position of our input values does not affect our output.

 A   B  | AND
--- --- |-----
 0   0  |  0
 0   1  |  0
 1   0  |  0
 1   1  |  1

For any statement above, if you swap A and B, the AND logic still stands true.

The second thing to note is that our summation + bias, w · x + b, should be negative, except when both A and B are equal to 1.

If we take a look back at our perceptron formula, we can generalize that our neuron will return 1, whenever our input is positive, x > 0, and return 0, otherwise, i.e., when the input is negative or 0.

Now, let’s work our way backwards.

If our inputs are A = 1 and B = 1, we need a positive result from our summation, **w · x**; for any other inputs, we need a 0 or negative result:

1w + 1w + b  > 0
0w + 1w + b <= 0
1w + 0w + b <= 0
0w + 0w + b <= 0`

We know that:

x * 0 = 0
1x + 1x = 2x
1x = x

So we can simplify the above to:

2w + b > 0
w + b <= 0
b <= 0`

Now we know that:

b is 0 or negative
w + b is 0 or negative
2w + b is positive

We also know that:

b cannot be 0. If b = 0, then 2w > 0 and w <= 0, which cannot be true.
w must be positive. If w were negative, any 2w, would also be negative. If 2w were negative, adding another negative number, b, could never result in a positive number, so 2w + b > 0 could never be true.
If b is negative and w is positive , w — b = 0, so that w + b <= 0.

That’s it!

We now know that we can set b to any negative number and both w’s to its opposite, and we can reproduce the behavior of AND by using a perceptron!

For simplicity, let’s setb = 1, w1 = -1, and w2 = -1

Resources

Perceptrons in Neural Networks

2018-03-23T00:00:00+00:00

Perceptrons are a type of artificial neuron that predates the sigmoid neuron. It appears that they were invented in 1957 by Frank Rosenblatt at the Cornell Aeronautical Laboratory.

The initial difference between sigmoids and perceptrons, as I understand it, is that perceptrons deal with binary inputs and outputs exclusively.

Taken from Michael Nielsen’s Neural Networks and Deep Learning we can model a perceptron that has 3 inputs like this:

A perceptron can have any number of inputs, but this one has three binary inputs x¹, x², and x³, and produces a binary output, which is called its activation.

How can we take three binary inputs and produce one binary output? First, we assign each input a weight, loosely meaning the amount of influence the input has over the output.

In the picture above, weights are illustrated by black arrows. We’ll call each weight w. Each input, x above has an associated weight: x¹ has a weight w¹, x² a weight of w², and x³, a weight of w³.

To determine the perceptron’s activation, we take the weighted sum of each of the inputs and then determine if it is above or below a certain threshold, or *bias, *represented by b.

The formula for perceptron neurons can can be expressed like this:

Let’s break this down.

output is the output of our formula, which is called the activation of our perceptron.
Both if branches start with the same ∑ formula which takes each input, x, multiplies it by its weight, w, and then add them all together. This is the *weighted sum, *in our case, x¹w¹ + x²w² + x³w³, which can also be, (and usually is), represented using dot product notation.
If the weighted sum is less than or equal to our threshold, or bias, b, then our output will be 0
If the weighted sum is greater than our threshold, or bias, b, then our output will be 1

This formula is called a Heaviside Step function, and it can be graphed like this:

f(x) = { x <= b : 0 , x > b : 1 }

Were x is our weighted sum, *and b is our *bias, 0, in this case.

For any negative x, (input), our y, (output), is 0, and for any positive x, our y is 1.

I want to record this graph, as simple as it is, because it will help demonstrate the differences between perceptrons and sigmoids, later.

(EDIT 25.03.18)

It’s more common to represent the perceptron math like this:

The summation is represented using dot product notation.
The “threshold” is moved to the other side of the equality and labeled b for “bias.”
The summation and bias are added together and compared to to 0.

This new way of comparing to 0, offers us a new way of thinking about these artificial neurons. We can think of the bias, now, like a predictor of how easily our neuron will activate, or produce 1 as an output. A neuron with a large biases will indicate that it will “fire” more easily than the same neuron with a smaller bias.

Lastly, pseudocode might look something like this:

def perceptron(inputs, bias)

 weighted_sum = sum {
  for each input in inputs
    input.value * input.weight
  }

  if weighted_sum <= bias
    return 0
  if weighted_sum > bias
    return 1

end

Phew! That was a lot, but now we can add more detail to our perceptron model:

IRL Example

Inspired by the first pages of Michael’s book.

My boyfriend and I want to know whether or not we should make a pizza for dinner. I’m going to rely on our perceptron formula to make a decision. In order to determine if we should make pizza, we’re going to check if we have all of the ingredients, if I’m in the mood for pizza, and if he’s in the mood for pizza.

I really enjoy making pizza, but I hate shopping, so if we don’t have all the ingredients, I’ll only want to make pizza if I’m in the mood for it. If my boyfriend is hungry for pizza, I’ll only want pizza if I don’t have to go to the store, unless I’m also craving pizza.

If, we have all of the ingredients and my boyfriend is in the mood for pizza, but I’m not, we can break down our problem thusly:

Decision:
  Make pizza for dinner?

Inputs:
  x¹ = 1 # We have all of the ingredients.
  x² = 0 # I'm not in the mood for pizza.
  x³ = 1 # My boyfriend is in the mood for pizza.

Weights:
  w¹ = 3 # Having the ingredients makes me willing to have pizza.
  w² = 4 # If I want pizza, I really want pizza!
  w³ = 2 # Him wanting pizza is the least of my concerns.

Bias:
  4

Let’s map it using our illustration!

Each input represents a binary state of each scenario I’m considering, and each weight represents how important each yes-or-no answer is to my final decision. Let’s plug in the numbers.

(1*3) + (0*4) + (1*2) = 5 <= 4 #=> FALSE
(1*3) + (0*4) + (1*2) = 5 <= 4 #=> TRUE = 1

It looks like pizza it is!

Given our perceptron model, there are a few things we could do to affect our output.

If we didn’t have control over out binary inputs, (let’s say they were objective states of being 1 or 0), we could still adjust the weight we give each input, and the bias. For our little pizza question, this is a fun experiment, and could maybe be analogous to how we, as humans, actually solve problems, given objective inputs! We are constantly adjusting the pros-and-cons and priorities we give each input before making a decision.

If I’m not in the mood for pizza, could I still eat it? If yes, then maybe I can decrease the importance of that input. Or maybe, I hate pizza. Then, whether or not I’m in the mood for it should be weighted even higher when it comes to making the decision to have it for dinner or not!

When looking at vanilla neural networks, (multilayer perceptrons), this balancing act is exactly what we’re asking the computer to do. We “train” a network by giving it inputs and expected outputs, and then we ask it to adjust the weights and biases in order to get closer to the expected output, i.e., how can you adjust the weights and biases to get this input, to equal this output?

After we train our network, we then present it inputs it has never seen before. If it’s weights and biases have been calibrated well, it will hopefully begin outputting meaningful “decisions” that has been determined by patterns observed from the many many training examples we’ve presented it.

And this is just the tip of the iceberg.

This is my first journal entry of my dive into machine learning. I’ll list the resources that have gotten me this far, below. Feedback is greatly appreciated, if I’ve gotten something wrong, or taken a misstep, any guidance will be met with open arms!

Thanks for Reading <3

Resources

Testing Ruby’s `gets` & `puts`

2018-03-07T00:00:00+00:00

The Problem.

You want to TDD some behavior that interacts with the command line:

puts "Would you like to continue? [yN]"
answer = gets.chomp.downcase

But testing this idea is difficult; when your tests run these lines of code, they can cause your tests to hang or send unwanted output to your console.

A Solution.

Wrap specific puts-ing and gets-ing behavior in code that you own and control.
Inject duck-types of the objects on which we can call puts and gets . We call puts and gets implicitly on the objects stored in the $stdout and $stdin variables , respectively. For our tests StringIO is a handy duck-type of both of these objects!
Call puts and gets on the, now injected, objects.
Use StringIO’s instance methods to write assertions.

An Example Test.

RSpec.describe ConsoleInterface do

  describe '#ask_question' do
    it 'sends a prompt question to output' do
      output = StringIO.new
      console_interface = ConsoleInterface.new(output: output)

      console_interface.ask_question

      expect(output.string).to include("continue?")
    end
  end

  describe '#answer' do
    it 'returns a formatted string received from input' do
      input = StringIO.new("iNPut\n")
      console_interface = ConsoleInterface.new(input: input)

      expect(console_interface.answer).to eq("input")
    end
  end

end

An Example Implementation.

class ConsoleInterface

  def initialize(input: $stdin, output: $stdout)
    @input = input
    @output = output
  end

  def ask_question
    @output.puts "Would you like to continue? [yN]"
  end

  def answer
    @input.gets.chomp.downcase
  end

end

Why I Like This.

The power of Ruby is that if it quacks like a duck, it must be a duck! By combining this freedom with dependency injection, we can quickly grab control over our dependency on what we often take for granted: the command line.

Also, you might be excited to learn that you’ve just used a test double! They can often be seen as big scary controversial things that require you to pull in heavy libraries, but because of duck-typing, using a test-double can be as simple as injecting a built-in object that we have more control over.

Thomas Countz

Utilizing Docker for Bundling in a Specific Ruby Environment

A Thought on Practice

Københavns Luftkvalitet og Cykling

Introduction

Data Ingestion & Initial Aggregation

Route Querying & Data Joining

Next Steps

Sargable Queries & MUL Indexes; or Why My Query is Slow

Sargable & Non-sargable

TEXT & VARCHAR

How it looks with an index

Back on Ops

Rocket Lab Telemetry From Video

The Anomaly

Youtube Video Stream

Google Video Intelligence API

Data Cleanup/Presentation

1. The results were big

2. Data out of order

3. Knowing what the text means for each timestamp

The Results

Conclusion

Hexo - Schematic Capture, Mechanical Design, & PCB Layout

Hexo - Prototyping, Design, Simulation

Prototyping, Analysis, & Simulation

Communication Superhighways

Crafting an Invitation to Think

Repeated Signals Wear Grooves

Arduino Tone for ESP32

Example Usage

References

SQL Injection Overview

Authentication Form Example

Authentication Attack

What Happened?

Data Destruction Attack

What Happened?

Prevention

Parameterized Queries

Other Defenses

Next Steps

Code Review

Example Code Snippets

Ruby/ActiveRecord

Elixir/Ecto

Java

Resources

Dare to be Good Enough—You'll be Happier for it

References

Specify Bundler Version

Types of Notetaking

Contents

Actionable TODOs

My solution: a passport-sized notebook and pen

In Situ Documentation

My solution: yellow legal pad and mechanical pencil*

Long-Form Reference

My solution: Gitbook/personal wiki

Your Notes, Your Way!

FastBook Chapter 4 Questions & Notes

How is a greyscale image represented on a computer? How about a color image?

How are the files and folders in the MNIST_SAMPLE dataset structured? Why?

Explain how the “pixel similarity” approach to classifying digits works.

What is a list comprehension? Create one now that selects odd numbers from a list and doubles them.

What is a “rank 3 tensor”?

What is the difference between tensor rank and shape? How do you get the rank from the shape?

What are RMSE and L1 norm?

How can you apply a calculation on thousands of numbers at once, many thousands of times faster than a Python loop?

Create a 3x3 tensor or array containing the numbers from 1 to 9. Double it. Select the bottom right 4 numbers.

What is broadcasting?

Are metrics generally calculated using the training set, or the validation set? Why?

What is SGD?

Why does SGD use mini batches?

What are the 7 steps in SGD for machine learning?

How do we initialize the weights in a model?

What is “loss”?

Why can’t we always use a high learning rate?

What is a “gradient”?

Do you need to know how to calculate gradients yourself?

`TEXT` & `VARCHAR`