The most fundamental goal of anyone who studies DSA (data structures and algorithms) is to attain a repertoire of patterns. We often start with arbitrary examples, such as the kind of problems we learn from resources like Leetcode. But the ultimate goal is not just to solve arbitrary puzzles to pass interviews with Big Tech. The goal is pattern recognition, so that we can apply techniques that reliably provide efficient solutions when we need them in real life scenarios.
The type of memory we apply to pattern recognition, semantic memory, is applied both implicitly and subconsciously (see Bob Snyder’s Music and Memory, MIT Press). That reference provides a significant clue where I’m going with this: both our perception of music and recognition of patterns are engaging our subconscious.
When we set out to study algorithms and data structures, we tend to immediately set out to learn these patterns by way of memorization and logical understanding. When first confronted with a problem like a binary search, we think we are being coerced into memorizing something that doesn’t come naturally. But we do subconsciously invent efficient algorithms in every day life. When looking up a word beginning with “Z” in an actual printed dictionary, we’ll start at the back of the book. For words starting with letters somewhere in the middle, we might even find anyone may apply a sort of binary search by starting at the middle, figuring whether to go before or after, then going to the midpoint in that direction, and so on. When navigating a maze in a corn field, or trying to find our favorite t-shirt in a dresser, we find ourselves intuitively employing a breadth-first search algorithm.
The vast majority of us, however, cannot come up with the code equivalent of a graph traversal algorithm out of thin air. That’s why the pioneering engineers who discovered these commonly used algorithms are revered and recorded in history, and many have even been honored by having algorithms named after them. The best that most of us can do is observe a walkthrough of the thought process behind the solution, and with a combination of memorization and comprehension of the reasoning behind each of the steps, store it away in our minds, adding to our gallery of patterns to which new problems can be matched and recalled when needed.
In our day to day work, it is likely that we only frequently use a relatively small subset of these patterns. Used often enough, these templates are recalled with all the speed of intuition. We just hear the hint of a problem, and we reflexively picture the solution. And in the context of real-world work, that’s what we want. The goal of any good design is to leverage frameworks and best practices, one of the beneficial side effects of which is to reduce cognitive load and enable productivity.
This leaves a lot of potentially useful patterns sitting in the corner of our mind, gathering dust and rusting from disuse. We may never need to think about even simple structures like stacks and queues in a given environment, but that doesn’t mean that we wouldn’t benefit by finding ways to bring them out of the attic and into our lightning-fast subconscious. Trying to do this by way of tried-and-true studying and practice is an admirable and rewarding path, but it has its limitations. We can only guess which of these patterns will eventually lend themselves to a needed solution. How can we know which areas of study will be useful, or simply take up precious space and time?
My experiment in forthcoming articles/chapters is to create cognitive connections to engineering concepts through our perception of music. I imagine an engineer who is planning an application, and in hearing a Bach fugue realizes that a multi-threaded approach provides a new way forward. Someone else may hear the chaotic improvisations of a jazz saxophonist and come to understand that a document database can provide for storage of unpredictable schemas they had been spending too much time trying to conform to something too rigid.
When Douglas Hofstadter wrote Gödel, Escher, Bach, despite winning a Pulitzer price for the work, he was disappointed that the public interpreted the work as a study of connections between math, art, and music. He was trying to convey the “strange loop” of consciousness through the exploration of recursive patterns, which he clarified in his follow-up I Am a Strange Loop. One could hardly fault the reading public for this misinterpretation, the author’s intentions were heavily obfuscated by his artistic approach. Clarity of communication rests most heavily on the communicator. My effort here really is to help us create more readily recalled cognitive connections through music to the vast universe of software engineering concepts at our disposal, much like the aforementioned Coltrane’s Circle led to a discovery in physics.
Next, I’m going to show how I found queue and stack data structures of associative arrays in a Broadway musical.
Until next time…